Transcript AHRQ Slide Template 2004

Building Wave Response Rates in a Longitudinal Survey:
Essential for Nonsampling Error Reduction
or Last In - First Out?
Steven B. Cohen
Fred Rohde and William Yu
Agency for Healthcare Research and Quality
Advancing
Excellence in
Health Care
Purpose of Discussion
 Need for essential longitudinal data on
health care coverage, use and expenditures
to inform health care policy and practice
 Description of the Medical Expenditure
Panel Survey (MEPS): purpose, longitudinal
design and analytical capacity
 Focus on field efforts to achieve target
response rates
Advancing
Excellence in
Health Care
Purpose of Discussion
 Evaluations of the quality of the MEPS
nonresponse adjustment strategies
 Determination of characteristics for cases
fielded at end of field period (EOF) &
conversion of temporary refusals (TNR).
 Examine ROI for inclusion of these cases.
Advancing
Excellence in
Health Care
Purpose of Discussion
 Examine impact on annual and longitudinal
response rates; completion of self
administered questionnaires (SAQ)
 Impact on key survey estimates of health
insurance coverage and expenditures
 Implications of alternative field strategies
Advancing
Excellence in
Health Care
Medical Expenditure Panel
Survey (MEPS)
Annual Survey of 15,000 households:
provides national estimates of health care use,
expenditures, insurance coverage, sources of
payment, access to care and health care quality
Permits studies of:




Distribution of expenditures and sources of payment
Role of demographics, family structure, insurance
Expenditures for specific conditions
Trends over time
Key Features of MEPS-HC
Advancing
Excellence in
Health Care
 Survey of U.S. civilian noninstitutionalized population
 Sub-sample of respondents to the National Health Interview
Survey (NHIS)
–
Linkage to NHIS
 Oversample of minorities and other target groups
 Panel Survey – new panel introduced each year
– Continuous data collection over 2 ½ year period
– 5 in-person interviews (CAPI)
– Data from 1st year of new panel combined with data from
2nd year of previous panel
MEPS Overlapping Panels
(Panels 8 and 9)
Advancing
Excellence in
Health Care
MEPS Household
Component
MEPS Panel 8 2003-2004
1/1/2003
NHIS
2002
Round 1
1/1/2004
Round 2
NHIS
2003
Round 3
Round 4
Round 1
Round 5
Round 2
Round 3
MEPS Panel 9
2004-2005
Round 4
Round 5
Advancing
Excellence in
Health Care
1996
1997
1998-2001
2002+
MEPS Household Component
Sample Design
Oversampling of policy relevant domains
Minorities (Blacks & Hispanics)
Minorities
Low income
Children with activity limitations
Adults with functional limitations
Predicted high expenditure cases
Elderly
Minorities
Minorities, Asians, Low Income
15,000 households; ~35,000 persons
Advancing
Excellence in
Health Care
Design Specifications
 Target Precision
 Linkage to NHIS
Specifications for
national and regional
estimates; policy
relevant subgroups
 Overall Design effect of
1.6
 200 PSU design (Max)
 Overall/round specific
survey response rate
requirements
 Multistage design
 Disproportional
sampling
 Longitudinal design
 Minimize survey cost
for fixed precision
MEPS, 1996–2006: Number of uninsured,
under age 65
Number in millions
Any time in year
62.0
62.2
59.1
60
44.5
44.2
58.5
61.7
First half of year
61.9
Full year
61.7
62.9
63.9
65.8
47.0
48.1
49.8
42.0
42.6
43.8
45.7
45.9
50.1
40
31.6
32.1
31.0
28.7
31.5
32.0
33.7
34.4
35.8
31.3
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
20
0
2006
Source: Center for Financing, Access, and Cost Trends, AHRQ, Household Component of the Medical Expenditure Panel
Survey, 1996–2005 Full-Year and 1996–2006 Point-in-Time Files
Advancing
Excellence in
Health Care
Trends in Concentration
Percentage of expenditures
1977
100
90
80
70
60
50
40
30
20
10
0
1987
1996
2005
97 97 97 97
70 70 69
55 56 56
38 39 38
27 28 28
65
50
33
23
Top 1% Top 2% Top 5% Top 10% Top 50%
Population ranked by expenditures
Source: National Medical Care Expenditure Survey, 1977; National Medical Expenditure Survey, 1987; Medical Expenditure Panel
Survey, 1996 and 2005.
Advancing
Excellence in
Health Care
MEPS Field Force
 Westat is data collection organization
 500 interviewers
 Sample allocated in ~ 200 PSUs, spread across all
50 states
 extensive training modules
 Information on socio-demographic characteristics of
households available based on linkage with NHIS
 Remuneration for survey participation
Advancing
Excellence in
Health Care
Tool Chest of Methods to Maximize
Survey Response
 Recruitment of experienced and bilingual interviewer
 10+ days training (including procedures for obtaining signed







consents)
Uses of MEPS data as reference materials for interviewers
Periodic retraining and special trainings (e.g. methods to
improve response rates)
Respondent remuneration
Advance mailings from co-sponsors of survey
Monthly planning calendar and MEPS DVD
Daily emails to interviewers regarding interviewing progress
Multiple contacts for refusal conversions
Advancing
Excellence in
Health Care
MEPS Target Response Rates by Round
and Overall
Response Rate
NHIS
90%*
Round 1
80%
Round 2
95%
Round 3
96%
Year 1 of Panel
66%
Round 4
97%
Round 5
98%
Pooled Response Rate-Two Panels
65%
*NHIS response rate among households designated for MEPS.
Note: Year 1 and the Overall response rate include the NHIS response rate.
Advancing
Excellence in
Health Care

Person Level (survey attrition)
Nonresponse Adjustment Covariates
Factors associated with survey attrition (after R1)
– Indicator for initial refusal to R1 interview
– Family size
– Age
– MSA, census region
– Marital status (family reference person)
– Race/ethnicity
– Education of reference person
– Employment status
– Health insurance status
– Total expenditures (in yr 1 for yr 2 adj.)
– # doctor visits (in yr 1)
– Self reported health status
Advancing
Excellence in
Health Care
Person Level Adjustments:
Annual Estimates
 Each MEPS panel weighted separately
 Nonresponse adjustment for complete
nonresponse and for survey attrition
 Final Poststratification adjustment –CPS
12/31:
age, race/ethnicity, sex, region, MSA
status, poverty status
Advancing
Excellence in
Health Care
Testing for Panel Effect
Advancing
Excellence in
Health Care
Capacity of MEPS to Produce Comparable
NHIS Estimates of Health Insurance
Coverage
Advancing
Excellence in
Health Care
Characteristics of Respondents Fielded at
End of First Round or Temporary Refusal
Initial Refusals:
Higher likelihood:
 MSA residence; Northeast region; white NonHispanic; elderly; excellent health; some high
school; family size 2+; Attrite in future waves
of data collection
End of Field Period Cases
Higher likelihood:
 Race: Asian or Black
 in excellent health
 Attrite in future waves of data collection
Advancing
Excellence in
Health Care
Testing for Reluctant Response Effect
on Coverage Estimates
-------------------------------------------------------
DF Wald F P-Value
------------------------------------------------------OVERALL MODEL 22
107.98 0.0000
JULY INTERVIEW
1
2.38
0.1244
TEMP. REFUSALS
1
0.92
0.3393
SEX
1
98.02 <0.0001
RACE/ETHNICITY
3
58.42 <0.0001
MARITAL STATUS
4
16.90 <0.0001
EDUCATION
4
10.94 <0.0001
POVERTY STATUS
4
43.97 <0.0001
MSA STATUS
1
4.34
0.0382
INDIVIDUAL INCOME 1
35.52 <0.0001
MEDICAL $
1
35.79 <0.0001
-------------------------------------------------------
-2 * Normalized Log-Likelihood Full Model: 13037.78
Pseudo R2: :0.194167
Mean Number of Contacts by Month
Advancing
Excellence in
Health Care
< Mar
Overall
Apr
May
Jun
Jul
O refusals
20
Mean Number of Contacts
Mar
1+ refusals
Panel 9
Panel 10
15
10
5
< Mar
Mar
Apr
May
Jun
Jul
< Mar
R1 Interview Month
Mar
Apr
May
Jun
Jul
Mean Number of Temporary
Refusals by Month
Advancing
Excellence in
Health Care
< Mar
Overall
Mar
Apr
May
Jun
Jul
O refusals
1+ refusals
8
Mean Number of Refusals
Panel 9
Panel 10
6
4
2
0
< Mar
Mar
Apr
May
Jun
Jul
< Mar
R1 Interview Month
Mar
Apr
May
Jun
Jul
Conditional Response Rates by Month
of Round 1 Response: Panel 9
Advancing
Excellence in
Health Care
< Mar
Overall
Mar
Apr
May
Jun
Jul
O refusals
1+ refusals
Response Rate
0.8
0.6
1st-year response rate
1st-year SAQ response rate
1st- and 2nd-year SAQ response rate
Longitudinal response rate
0.4
0.2
< Mar
Mar
Apr
May
Jun
Jul
< Mar
R1 Interview Month
Mar
Apr
May
Jun
Jul
Advancing
Excellence in
Health Care
Panel/temp.
refusal
Interview
date
Overall P9
Overall
P10
P9
< Jul
P10
P9
Jul
P10
0 P9
< Jul
0 P10
1+ P9
Jul
1+ P10
Exclusions
July
June refusal July
Mean R1
contacts
Mean R1
Temp.
refusals
full year
Cond.
response
Longitud
Cond.
response
18,250
6.17
0.30
89.90%
85.40%
18,170
6.67
0.35
88.80%
.
17,537
5.64
0.19
90.30%
85.90%
17,380
6.16
0.23
89.30%
.
713
19.21
2.94
79.20%
71.10%
790
17.78
2.94
76.70%
.
16,631
5.42
0.00
91.00%
86.80%
16,377
5.90
0.00
89.90%
.
272
17.26
7.71
77.20%
62.50%
328
16.55
7.08
73.20%
.
.041
.959
.062
.938
N persons
RR reduction
Impact on MSE of Mean Medical Expenditure Estimates
for alternative field strategies
Measure
Standard Field
Operation
50% sample
Difficult Cases
Mean Expenditures
33% sample
25% sample
Exclusion
Difficult Cases
Difficult
Cases
Difficult Cases
3479.01
3477.90
3478.24
3521.02
3543.21
Standard Error
91.03
92.02
92.93
94.17
94.94
Responding sample
31589
30769
30495
30359
29948
July cases+June temp
refusals
1641
821
547
411
RSE
0.026
0.026
0.027
0.027
0.027
Bias
0
0
0
0
64.20
MSE
8287.10
8467.37
8635.93
8868.86
13135.24
91.03
92.02
92.93
94.17
114.61
1
1.022
1.042
1.070
1.586
14.2
22
22
22
12.5
670
1,284
2,132
RMSE
Ratio of MSE to
Standard
Mean no. hours per
complete
Add’l sample to meet
precision goal
Impact on MSE of Proportion with Medical Expenses in
excess of $10,000 for alternative field strategies
Measure
Standard Field
Operation
50% sample
33% sample
25% sample
Exclusion
Difficult Cases
Difficult Cases
Difficult
Cases
Difficult Cases
Proportion
0.0756
0.0756
0.0757
0.077
0.0774
Standard Error
0.0021
0.0022
0.0022
0.0023
0.0022
Responding sample
31589
30769
30495
30359
29948
July cases+June
temp refusals
1641
821
547
411
RSE
0.028
0.029
0.029
0.030
0.028
Bias
0
0
0
0
0.002
MSE
0.00000441
0.00000484
0.00000484
0.00000529
8.08E-06
0.0021
0.0022
0.0022
0.0023
0.0028
1
1.098
1.098
1.200
1.832
RMSE
Ratio of MSE to
Standard
Advancing
Excellence in
Health Care
Summary
 Need for accurate and reliable national data on
health insurance coverage to inform policy and





practice
MEPS design features and analytical capacity
Statistical, methodological and operational design
features to adjust for nonresponse and attrition
Evaluation of estimation strategies to correct for
survey attrition
Examination of ROI for inclusion of difficult cases
Options identified for more efficient and effective
field strategies
Agency for Healthcare Research and Quality
Advancing Excellence in Health Care • www.ahrq.gov