Transcript What Kind of Care Are We Buying?

An Assessment of the Impact of Two Distinct
Survey Design Modifications on Health
Insurance Coverage Estimates in a National
Health Care Survey
Steven B. Cohen, Trena Ezzati-Rice, and Marc Zodet
International Total Survey Error Workshop 2010
Stowe, VT
June 13-16, 2010
Background and motivation for
research
 The Medical Expenditure Panel Survey
(MEPS) is a national resource to inform health
care policy
 MEPS is a key survey resource to monitor:
– Trends in estimates of the uninsured
– Population characteristics of the uninsured
– Duration of spells of uninsurance and long term
uninsured
– Financial consequences of being uninsured
– Relationship between uninsurance and health
status
 Thus need for timely, high quality MEPS data
MEPS survey background
 Annual survey since 1996; nationally representative
sample of households
 5 rounds of data collection covering 2 calendar years
 Used to estimate medical care utilization, access to care,
and health care expenses for the U.S. civilian
noninstitutionalized population
 Integrated survey design
–
Each annual sample is a subsample of responding households
(from prior year) from another large ongoing U.S. health survey,
the National Health Interview Survey (NHIS)
 Overlapping panel design
–
Data from 1st year of new panel combined with data from 2nd
year of previous panel
Illustration of how panels/rounds
comprise MEPS calendar year data
MEPS survey design modifications in
2007: Panel 12
 1. Re-engineered CAPI survey instrument: Windows-based
platform replaced the DOS-based system (questionnaire
remained virtually unchanged)
 2. New sample design resulting from the sample redesign of
the NHIS in 2006
– MEPS Panel 12 fielded in January 2007 was the 1st
Panel to be selected based on the new NHIS sample
– New NHIS sample design conceptually very similar to the
1995-2005 design: complex area probability sample
– Changes in 2006 NHIS design that affected 2007 MEPS
 Sampling PSUs and SSUs independent of those sampled
under the previous design
 Previously only HHs with Black and Hispanic persons were
oversampled - New – oversampling of Asian persons
Goals of this research
 Evaluate if any significant impact on MEPS
estimates of insurance coverage as a result of
the dual survey changes implemented in Panel
12 which began January 2007
 Evaluate the effectiveness of the MEPS
nonresponse adjustment strategies in reducing
potential bias
Two evaluation approaches
 1. Internal survey comparisons
– takes advantage of MEPS overlapping panel design
 compare the two individual panel specific estimates of health
insurance coverage for 2007 and other years
– conduct multivariate analysis to determine if panel is a
significant predictor of insurance status after controlling
for other independent variables
 2. External comparisons
– takes advantage of MEPS linkage to NHIS survey
– after MEPS household nonresponse and survey
attrition, can insurance estimates for the sampling frame
source (NHIS) be replicated using just the MEPS survey
respondents and the MEPS nonresponse adjusted
weights?
Sources of data
 2007 MEPS compared to 2004-2006
– Panel specific estimates within each calendar year
(using panel specific weights)
 2003-2006 NHIS data
Evaluation Approach 1:
Internal MEPS Comparison
Step 1: Panel specific estimates within years
Step 2: Multivariate analysis
Step 1: Evaluation of concordance of
panel specific health insurance coverage
estimates
 Focus is on 2007 coverage estimates
 Panel 12 – dual survey changes
 Panel 11 – original sample design and
original DOS-based survey instrument
MEPS annual estimation weights
 Each panel is weighted separately and then
combined
 Panel specific weight is a function of:
– NHIS base weight
– Poststratification to NHIS full sample
– Nonresponse adjustment for dwelling unit
nonresponse and survey attrition within year 1
and year 2
– Final raking adjustment to CPS population
control totals
Evaluating the joint effect of the
dual survey design changes in 2007
Comparison of panel specific national health
insurance coverage estimates for the
population under age 65
 Measures evaluated:
– 1) some private coverage during the year
– 2) public-only coverage during the year
– 3) full year uninsured
Estimates of health insurance coverage for the
civilian non-institutionalized population
<65 years of age by panel within year
Calendar
Coverage
Year
Measure
2007
2006
2005
2004
Year 2 in Panel
Year 1 in Panel
% (SE)
% (SE)
Any private
69.2 (0.83)
68.7 (1.04)
Public only
15.8 (0.61)
15.8 (0.73)
Uninsured
15.0 (0.53)
15.5 (0.68)
Any private
69.9 (0.81)
70.9 (0.78)
Public only
15.6 (0.56)
15.0 (0.56)
Uninsured
14.5 (0.51)
14.1 (0.48)
Any private
70.7 (0.83)
70.8 (0.72)
Public only
15.3 (0.62)
15.1 (0.53)
Uninsured
14.0 (0.48)
14.2 (0.47)
Any private
70.5 (0.85)
71.7 (0.80)
Public only
15.2 (0.60)
14.4 (0.57)
Uninsured
14.3 (0.49)
13.9 (0.51)
Source: Medical Expenditure Panel Survey Household Component (MEPS-HC), 2004-07
Step 2: Multivariate analysis
 Logistic regression analysis to test for a
panel effect controlling for other
explanatory variables related to health
insurance coverage
 Separate models by year: 2004-2007
Covariates included in the models









Panel classification
Sex
Race/ethnicity
Self-reported health status
Marital status
Education
Income
MSA, Census region
Total healthcare expenses
Final logistic regression model of the uninsured,
ages 18-64 years; testing for panel effect
2007
2006
2005
2004
Degrees of
Freedom
Wald F
P-value
Wald F
P-value
Wald F
P-value
Wald F
P-value
Overall Model
26
94.84
<0.001
120.92
<0.001
101.55
<0.001
84.21
<0.001
Model minus intercept
25
42.66
<0.001
66.92
<0.001
50.91
<0.001
50.28
<0.001
Panel classification
1
1.79
0.1814
0.06
0.8135
0.34
0.5627
0.03
0.8644
Sex
1
71.57
<0.001
66.09
<0.001
75.24
<0.001
69.68
<0.001
Race/ethnicity
3
53.83
<0.001
42.42
<0.001
56.03
<0.001
47.57
<0.001
Health status
4
2.15
0.0737
2.72
0.0302
1.18
0.3194
4.90
0.0008
Marital status
4
14.84
<0.001
13.02
<0.001
18.21
<0.001
26.26
<0.001
Highest year of education
4
12.31
<0.001
32.71
<0.001
15.37
<0.001
12.69
<0.001
Poverty status
4
69.96
<0.001
103.81
<0.001
71.14
<0.001
79.53
<0.001
Region
3
21.00
<0.001
9.01
<0.001
11.96
<0.001
17.58
<0.001
Total health care
expenditures
1
34.88
<0.001
68.44
<0.001
32.47
<0.001
35.68
<0.001
-2 x Normalized LogLikelihood Full Model:
Pseudo Model R-Square:
13,422.9
14,319.7
14,223.5
14,052.5
0.1550
0.1588
0.1516
0.1569
Source: Medical Expenditure Panel Survey Household Component (MEPS-HC), 2004-07
Note: In logistic regression analysis, Y=1 specifies full year uninsured, Y=0 specifies other
Final logistic regression model of the privately
insured, ages 18-64 years; testing for panel effect
2007
2006
2005
2004
Degrees of
Freedom
Wald F
P-value
Wald F
P-value
Wald F
P-value
Wald F
P-value
Overall Model
26
105.10
<0.001
140.17
<0.001
108.79
<0.001
100.36
<0.001
Model minus intercept
25
89.83
<0.001
130.70
<0.001
105.91
<0.001
92.48
<0.001
Panel classification
1
2.64
0.1048
2.39
0.1234
0.31
0.5763
0.51
0.4749
Sex
1
17.37
<0.001
10.73
0.0012
13.42
0.0003
28.81
<0.001
Race/ethnicity
3
41.34
<0.001
34.14
<0.001
49.22
<0.001
45.73
<0.001
Health status
4
20.36
<0.001
31.84
<0.001
32.96
<0.001
21.91
<0.001
Marital status
4
34.08
<0.001
28.26
<0.001
33.95
<0.001
37.65
<0.001
Highest year of education
4
41.56
<0.001
76.06
<0.001
64.07
<0.001
58.32
<0.001
Poverty status
4
245.13
<0.001
308.26
<0.001
261.46
<0.001
282.70
<0.001
Region
3
3.70
0.0119
3.99
0.0085
0.49
0.6906
2.09
0.1027
Total health care
expenditures
1
19.94
<0.001
57.94
<0.001
30.63
<0.001
34.51
<0.001
-2 x Normalized LogLikelihood Full Model:
14555.5
15,608.2
15,268.0
15,354.9
Pseudo Model R-Square:
0.2837
0.2854
0.2918
0.2883
Source: Medical Expenditure Panel Survey Household Component (MEPS-HC), 2004-07
Note: In logistic regression analysis, Y=1 specifies some private insurance in year, Y=0 specifies other
Evaluation Approach 2: External
Evaluation
 Takes advantage of the linkage of the MEPS to the NHIS
 National estimates of health insurance are derived from
NHIS (NHIS insurance variables) for two analytical samples:
– Total NHIS sample → NHIS weights
– Reduced NHIS sample obtained from matching with
MEPS full year respondents → MEPS weights as
adjusted for MEPS nonresponse
 Since this analysis examines only NHIS insurance data, the
estimates are not impacted by the MEPS CAPI redesign
– Thus, assessment of joint impact of the MEPS sample
redesign and effectiveness of MEPS nonresponse
adjustments.
MEPS-NHIS linked analysis
The following 4 NHIS measures of health insurance
coverage were examined:
 any coverage at the time of the interview (covered; not
covered; refused/not ascertained/DK)
 private coverage at the time of the interview
(mentioned; not mentioned; refused/not
ascertained/DK)
 Medicaid coverage at the time of the interview
(mentioned; not mentioned; refused/not
ascertained/DK)
 No health insurance coverage for more than one year
(yes, no; refused/not ascertained/DK)
NHIS any insurance coverage at time of interview
derived from MEPS sample and estimation weights
compared to NHIS sample and weights, age <65
NHIS Variable
Calendar Year
2007
NOTCOV (Health
insurance
coverage status)
%
SE
(0.69)
16.8
(0.29)
Covered
83.7*
(0.71)
82.0
(0.29)
1.1
(0.18)
1.2
(0.08)
Not covered
15.9
(0.53)
16.3
(0.24)
Covered
83.3
(0.56)
82.8
(0.24)
0.7
(0.12)
1.0
(0.05)
Not covered
16.5
(0.54)
16.2
(0.23)
Covered
82.7
(0.56)
82.8
(0.24)
0.8
(0.14)
1.0
(0.06)
Not covered
15.7
(0.60)
16.3
(0.26)
Covered
83.4
(0.61)
82.6
(0.27)
0.9
(0.14)
1.1
(0.06)
Refused/NA/DK
2004
SE
15.3*
Refused/NA/DK
2005
%
Using the full sample NHIS and
NHIS weight (prior year)
Not covered
Refused/NA/DK
2006
Using the MEPS Full Year panel
specific weight for calendar year
respondents
Refused/NA/DK
Source: Medical Expenditure Panel Survey Household Component (MEPS-HC), 2004-07; National Health Interview Survey (NHIS), 2004-07
*significant difference in MEPS derived NHIS estimate relative to the NHIS derived estimate at the .05 level
NHIS private insurance derived from MEPS
sample and estimation weights compared to
NHIS sample and weights, age <65
NHIS Variable
Calendar Year
2007
HIKINDA (Private
insurance)
2005
2004
%
Using the full sample NHIS and
NHIS weight (prior year)
SE
%
SE
Mentioned
68.9*
(1.00)
66.3
(0.44)
Not Mentioned
30.1*
(1.01)
32.6
(0.44)
1.1
(0.18)
1.2
(0.08)
Mentioned
69.2
(0.81)
68.3
(0.39)
Not Mentioned
30.0
(0.80)
30.7
(0.39)
Refused/NA/DK
0.7
(0.12)
1.0
(0.05)
Mentioned
69.4
(0.80)
68.6
(0.39)
Not Mentioned
29.8
(0.80)
30.4
(0.39)
Refused/NA/DK
0.8
(0.14)
1.0
(0.06)
Mentioned
70.4
(0.85)
69.2
(0.39)
Not Mentioned
28.7
(0.84)
29.7
(0.39)
Refused/NA/DK
0.9
(0.14)
1.1
(0.06)
Refused/NA/DK
2006
Using the MEPS Full Year panel
specific weight for calendar year
respondents
Source: Medical Expenditure Panel Survey Household Component (MEPS-HC), 2004-07; National Health Interview Survey (NHIS), 2004-07
*significant difference in MEPS derived NHIS estimate relative to the NHIS derived estimate at the .05 level
NHIS Medicaid coverage derived from MEPS
sample and estimation weights compared to NHIS
sample and weights, age <65
NHIS Variable
Using the MEPS Full Year panel
specific weight for calendar year
respondents
Using the full sample NHIS and
NHIS weight (prior year)
Calendar Year
HIKINDD
(Medicaid)
%
2007
Mentioned
8.5
(0.66)
9.2
(0.25)
Not Mentioned
90.4
(0.66)
89.7
(0.26)
Refused/NA/DK
1.1
(0.18)
1.2
(0.08)
Mentioned
8.2
(0.41)
8.8
(0.21)
Not Mentioned
91.0
(0.42)
90.3
(0.21)
Refused/NA/DK
0.7
(0.12)
1.0
(0.05)
Mentioned
7.9
(0.46)
8.5
(0.20)
Not Mentioned
91.3
(0.47)
90.5
(0.20)
Refused/NA/DK
0.8
(0.14)
1.0
(0.06)
Mentioned
7.8
(0.38)
8.6
(0.21)
Not Mentioned
91.4
(0.40)
90.3
(0.22)
Refused/NA/DK
0.9
(0.14)
1.1
(0.06)
2006
2005
2004
SE
%
SE
Source: Medical Expenditure Panel Survey Household Component (MEPS-HC), 2004-07; National Health Interview Survey (NHIS), 2004-07
*significant difference in MEPS derived NHIS estimate relative to the NHIS derived estimate at the .05 level
NHIS insurance coverage for >1 year derived from
MEPS sample and estimation weights compared
to NHIS sample and weights, age <65
NHIS Variable
Calendar Year
2007
2006
2005
2004
HILAST (No
health insurance
for more than
one year)
Using the MEPS Full Year panel
specific weight for calendar year
respondents
%
Using the full sample NHIS and
NHIS weight (prior year)
SE
%
SE
Mentioned
10.7
(0.61)
11.5
(0.24)
Not Mentioned
89.0
(0.61)
87.9
(0.25)
Refused/NA/DK
0.2
(0.05)
0.5
(0.04)
Mentioned
10.7
(0.43)
11.1
(0.20)
Not Mentioned
88.8
(0.45)
88.3
(0.21)
Refused/NA/DK
0.5
(0.12)
0.6
(0.04)
Mentioned
11.2
(0.43)
11.1
(0.18)
Not Mentioned
88.5
(0.44)
88.4
(0.19)
Refused/NA/DK
0.3
(0.06)
0.5
(0.04)
Mentioned
10.3
(0.45)
10.8
(0.21)
Not Mentioned
89.3
(0.46)
88.6
(0.21)
Refused/NA/DK
0.4
(0.09)
0.6
(0.05)
Source: Medical Expenditure Panel Survey Household Component (MEPS-HC), 2004-07; National Health Interview Survey (NHIS), 2004-07
*significant difference in MEPS derived NHIS estimate relative to the NHIS derived estimate at the .05 level
What did we learn?
 MEPS overlapping panel design facilitates the
evaluation of periodic survey design changes.
 The linkage of MEPS and NHIS also facilitates data
quality assessments.
 The internal comparison of MEPS insurance
estimates (<65) by panel for 2007 and for years
prior to the redesign did not show any significant
differences.
 The logistic regression analysis likewise did not
reveal any significant effect for Panel.
What did we learn (cont.)?
 Phase 2 of this study evaluated the joint impact of the new




2007 sample design and the survey’s post-survey
adjustments.
The MEPS-NHIS linked analysis only showed modest
differences in 2 of 4 NHIS insurance variables examined.
No evidence of nonresponse bias attributable to year 1
survey attrition.
Limitation of this study: We could not tease out the
independent effects of the two survey modifications.
Nevertheless, the dual survey changes in 2007 did not
appear to have a major impact on insurance coverage
estimates.
Future research
 Assess the 2008 MEPS panel specific estimates
 CAPI platform and sample design will be the same in
both individual overlapping panels.
 Continued evaluations of variables used to adjust for
nonresponse.
 Evaluate transitions in insurance coverage from NHIS to
MEPS, pre-and post-new design.
 Evaluate trends in transitions in coverage in MEPS from
year 1 to year 2, pre- and post new design.
 Review CAPI programming and edit specifications.
Discussion
 Comments on the internal and external
evaluation strategies used in this study?
 What other strategies have been (or can be)
used to evaluate survey data quality
following implementation of design or
methodological modifications?