Transcript Brief Historical Overview of the Budapest Initiative and Testing Activities 20-22 January 2010 Palais des Nations, United Nations Geneva, Switzerland.

Brief Historical Overview of
the Budapest Initiative
and Testing Activities
20-22 January 2010
Palais des Nations, United Nations
Geneva, Switzerland
Measuring health status
for international comparisons…
• What constitutes core health?
• What measures should be used?
• What standards for producing the data?
• Are the data internationally comparable?
• What methods should be used for comparisons?
Historical Overview:
Establishment of the TF and First Meeting
• May
2004: UNECE sponsored a Joint Meeting on the Measurement
of Health Status with WHO and Eurostat
• Broad
consensus on a) the need for multiple indicators to provide a
full statistical picture of population health, both for individual
country use and for international comparisons; and b) the immediate
need for an indicator of ‘health state’
• Recommendation
to set up a Task Force on the measurement of
heath state under the aegis of the UNECE
• The
first meeting was held in Budapest in November 2005, and
following the City Group convention, now known as the Budapest
Initiative (BI)
Historical Overview:
Purpose and Objectives
Purpose: To develop a new common instrument to measure health
state suitable for inclusion in national interview surveys which will
provide basic necessary information on population health.
Objectives:
To develop a question set to assess overall health state through
a number of domains of functioning;
To describe trends in health over time within a country, across
subgroups of a population, and across countries; and
To do so in the framework of official national statistical
systems.
•
•
•
Historical Overview:
Framework, Definitions, Domains
• Development of a conceptual framework defining the concepts
of health status and health state
• Paper: Health as a Multi-Dimensional Construct and Cross-Population
Comparability (2005)
• Adopted a set of criteria for selecting functional domains
• Paper: Criteria For and Selection of Domains for the Measurement of
Health Status (2005)
• Selection of domains
• Developed a question set to measure selected domains
• Paper: Conceptual and Logistic Issues in Item Construction and Proposal
Questions for Domains (2005)
What is Health State?
Health State measures functional ability as opposed to
other aspects of health:
• Determinants and risk factors
• Disease states
• Use of health care
• Environment barriers and facilitators
Functioning is measured in terms of ‘capacity’ not
‘performance’
• ‘Within the skin’
• Without the use of aids in a reasonable environment
Health Status and Health States
HEALTH STATUS
Health states
(capacities)
Determinants
of health
Genetic contribution
Lifestyle/behaviour
Domain 1
Physiological
Physiological
risk
factors
risk factors and
and risk
risk markers
markers
Diseases,
symptoms
and injuries
Domain 2
Death
.
.
Domain N1
Physical environment
.
.
Economic environment
Domain N2
Quality of life
Wellbeing
Social environment
Health State: an individual’s levels of functioning within
a set of health domains.
BI Focus and Guiding Principles
for the Question Set
• Measure the health states of individuals;
• Operationalize health state as functional ability;
• Focus on capacity rather than performance;
• Measure across a parsimonious number of health
domains, capturing the most variation in health;
• Meet high standards of validity and international
comparability; and
• Simple and clear, and easily translated into many
languages.
Criteria for the Selection of Domains
In order to identify the domains of health to be included,
a set of criteria were established:
• Relevance: face validity, breadth of domains, importance for
population health monitoring, draw on selected key ideas of the
ICF;
• Feasibility: reasonable for inclusion in health interview surveys,
consistency in meaning across multiple social contexts,
heterogeneity, parsimony of domains
• Measurement: statistical & structural independence, clear series
of levels within domains, within/near the skin, suitability for
preference measurement.
Final 7 Domains for BI Questions
• Vision
• Hearing
• Walking / Mobility
• Cognition
• Affect
• Pain
• Fatigue*
The original set of 10 domains included dexterity and social functioning/
relationships, both of which have been dropped. Anxiety was combined with
depression in the current domain of affect.
Criteria and Considerations
for Item Construction
• Number of questions per domain
• Ensuring uni-dimensionality
• Duration of the recall period
• Dealing with assistive devices and medications
• Item wording and response categories
• Positive vs. negative wording
• 4- vs. 5-category response sets
• Summary/preference health measures
Testing the Question Set
Testing the BI Questions
Three rounds of testing have been conducted as part of the
development of a BI question set.
Round 1:
•
•
•
•
•
•
November – December 2006
BI-only test (4 countries, 3 languages): Australia,
Canada, Italy, U.S.
Cognitive testing of all 7 domains
Results reviewed at January 2007 TF meeting
Separate studies and reports, inconsistent analysis
methods, some general findings
Outcome: BI-M1 question set, submitted to Eurostat
Testing the BI Questions
Round 2:
•
•
•
•
•
•
•
November 2007 – January 2008
BI-ESS collaborative test (7 countries, 6 languages):
Bulgaria, Germany, Portugal, Spain, Switzerland, U.K,
U.S.
Cognitive testing of 6 domains (absent Vision)
Improvements: Evidence-based methodology & systematic
comparative analysis of patterns
Analysis meeting in U.S. February 2008, all participants
Results documented in report
Outcome: Further changes for cognitive testing
recommended
WG-BI Similarities
Substantive overlap/Differing perspectives:
•
•
Question domains include the same basic and complex activities
Disability includes interaction with environment and civil rights perspective
Requirements for question sets:
•
•
Minimize burden on national data collection
Parsimony in the number of indicators and measures; domains are succinct,
clearly defined
Reasonable expectation of high quality responses from samples of the
general public, demonstrated validity of measures
International comparability, relevant at national and international level
Focus on aspects of health that are more likely to produce comparable data
Need for cross national cognitive and field testing
Questions must be simple, clear, easily translated into many languages
Amenable to multi-modes of collection
•
•
•
•
•
•
Testing the BI Questions
Round 3 (Cognitive Test):
•
•
•
•
•
•
January 2009 – March 2009
UNESCAP-WG-BI collaborative test (10 countries):
Cambodia, Canada, Fiji, Kazakhstan, Maldives, Mongolia,
South Africa, Sri Lanka, Philippines, U.S.
Cognitive testing of 11 domains (additions: Upper Body,
Communication, Learning, and Life Activities)
Improvements: On-line data management and analysis tool
Analysis meeting in U.S. May 2009
Outcome: Further changes for field test recommended
Testing the BI Questions
Round 3 (Field Test):
•
•
•
•
June 2009 – August 2009
WG-BI-UNESCAP collaborative test (5 countries):
Cambodia, Kazakhstan, Maldives, Sri Lanka, Philippines
Cognitive testing of 11 domains (additions: Upper Body,
Communication, Learning, and Life Activities)
Improvements: Approx 1,000 field interviews at each site
Testing Issues and
Methodological Developments
General Issues for Testing
• Cross-cultural comparability: Do the survey questions work
consistently across all countries and subgroups?
• Translation comparability: Do terms (both in the question and in
the response set) have the same meaning across countries?
• Validity: Do respondents interpret questions consistently
regardless of country, language, or demographic? Do respondents
use the same thought processes to answer questions?
If not, then, why are there differences? What about the
countries, languages or demographic subgroups generate
different response processes?
How can we “fix” or manage these differences through
question design?
•
•
Objectives for the Test Protocol
Purpose: To develop systematic comparable method
with joint analysis.
Evidence based
Joint and coordinated interviewing
Similar protocol
Similar sample
Understanding of differences (at a minimum)
Joint and coordinated analysis
With interview data
Evidence based (as opposed to opinion)
•
•
•
Lessons Learned
• Semi-structured cognitive interviews offers critical and
•
•
•
unique insight into cross-national question performance
Transparency is critical
Of data from interviews
Of the process for drawing conclusions
Data collection oversight
Better data management
•
•
Example BI chart
Because of a physical, mental or emotional problem, do you have difficulty concentrating,
remembering or making decisions?
ID
Immediate
words after
question
Response
Category
Description of
any response
Difficulty
Activity,
experience
or Situation
Discussed
took meds ?
If so, was
this
considered?
Way in which answer
was formed; pattern
of Calculation
S1
"None"
No difficulty
None
Forgetting an
umbrella
when the forecast is
for
rain.
no
She was thinking about her
ability to remember &
concentrate in general. Even
though she occasionally
forgets things it's not a
problem. clause ignored
S2
"Sometimes I
have trouble
remembering
things"
a little
difficulty
Easy to answer, but
this difficulty she
sees as normal
stuff & isn't
concerned about it;
i.e., this isn't a
disability
walking in to the
kitchen and
forgetting why
she's there
no
a little difficulty means that
sometimes she forgets things
- it's frequency of forgetting,
not significance of things
forgotten; 1st clause ignored
Q-Notes
• On-line data entry and analysis tool
• Allows for continuous oversight
• Facilitates quick but thorough analysis
• Designed around analysis principles
Next Steps Toward BI-M2
Activities in 2010
Next Steps for the BI Questions
The BI-M1 did not meet all of the objectives for the
question set (some domains absent, some room for
improvement).
Testing to date has been focused on developing BI-M2.
BI-M2 will be based on lessons learned from:
•
•
•
2009 UNESCAP cognitive and field tests (reviewed here)
2010 European cognitive tests (Round 4)
2010 experience with U.S. National Health Interview Survey
2010 Testing - Round 4
Comparative Cognitive Interviewing Project
Testing will return to the European region, just as in
Round 2.
• Spain (Spanish)
• Portugal (Portuguese)
• Italy (Italian)
• France (French)
• Germany (German)
• Switzerland (French, German, Italian)
• United States (English, Spanish)
Project Group Coalition
Testing will be conducted by:
• Members of the 1 comparative group (from Round 2)
• Methodologists from QEM workshop at NCHS
• US and Canada
st
Project Goals
• Continue methodological work begun in Round 2 testing.
• Narrative interviewing vs. structured probes
• Interpretations of vague response categories across cultures and
languages
• Use of software for data collection.
• Examine issues raised at QEM.
• Use of mixed-method (PROMIS-WG; cognitive and field tests)
• Continue examination of WG/BI extended set, specifically what
questions capture.
• Patterns of interpretation
• Patterns of calculation
• Both in-scope and out-of-scope interpretations
Domains for Inclusion in the Test
• Cognition
• Communication
• Affect
• Learning
• Pain
• Fatigue
• PROMIS subset