Transcript AHRQ Slide Template 2004

Does Inclusion of Both Partial and Complete
Interviews from the Source Sampling Frame Have an
Effect on Nonresponse Error and Measurement Error
in a National Health Survey?
Trena M. Ezzati-Rice, Frederick Rohde, Steven B. Cohen
International Total Survey Error Workshop 2010
Stowe, VT
June 13-16, 2010
Motivation for research
 Large observed differences in
response rates between two types
of previously interviewed cases
included in the Medical
Expenditure Panel Survey (MEPS)
sample as a result of its integration
with another national health survey
MEPS survey background
 Annual survey since 1996; nationally representative
sample of households
 5 rounds of data collection covering 2 calendar years
 Used to estimate medical care utilization, access to
care, and health care expenses for the U.S. civilian
noninstitutionalized population
 Widely used to inform health care policy
 Integrated survey design
–
Each annual panel of households is a subsample of
responding households (from prior year) from another large
ongoing U.S. health survey, the National Health Interview
Survey (NHIS)
Integration of NHIS and MEPS

MEPS inherits the NHIS complex sample design with its
oversampling of selected minority populations

Reduces household screening costs for MEPS

Provides valuable sampling frame information to adjust for survey
nonresponse

Provides enhanced analytical potential (NHIS and MEPS linked
analyses)

MEPS response rate is conditioned on the NHIS response rate

MEPS inherits the composition of the NHIS interviewed sample
–
Completed household interviews
–
“Partially Completed” household interviews
Definition of an NHIS
“partially completed” interview

All questionnaire modules are not completed

NHIS questionnaire composed of 5 major sections

1. Household composition (demographic information)

2. Family core (basic health and sociodemographic data)

Family relationships, marital status

Health status, activity limitations, injury, poisoning

Access and utilization and citizenship

Health insurance

Educational attainment
STOP
Employment, earnings, or income data for partials NO Data
–
3. Sample adult qx. NO Data
–
4. Sample child qx. NO Data
–
5. Immunization section NO Data
MEPS Respondents
Interviews
NHIS complete
eligible for MEPS)
(NHIS Respondents
NHIS Sample Frame
Data Matrix: NHIS and MEPS
sample integration
MEPS Nonrespondents
MEPS Respondents
MEPS Nonrespondents
Background for this research
 Previously conducted research (Chiu et al, 2001)
indicated that “late”/difficult” NHIS interviews were
more likely to be partial completes.
 Partially completed NHIS cases in the MEPS sample
observed to:
 require more field effort
 have lower response rates compared to the NHIS complete
cases
Research questions

Do the NHIS partially completed cases as included in the MEPS
–
Have lower response propensity at Round 1 and subsequent rounds?
–
Adversely affect data collection burden?

Are there differences in the demographic and socioeconomic
characteristics of the MEPS respondents when stratified by their
NHIS interview status (partial versus complete)?

Do the NHIS partially completed cases as carried over to MEPS
affect survey data quality?
–
Do they have higher item nonresponse rates on MEPS variables?
–
Do MEPS survey estimates differ when stratified by partial vs. complete
NHIS interview status?
Source of Data
 MEPS
– Historical data for examining response rate trends
– Panels 11 & 12 for detailed analyses
– Paradata (survey burden assessment)
– Survey data (item nonresponse and weighted
survey estimates)
 NHIS
– Paradata (NHIS interview status and other data)
– Survey data (response propensity modeling)
Percentage of MEPS reporting units
with a previous NHIS partially
completed interview: MEPS Panels 3-14
Year MEPS
Panel Started
Panel
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
3
4
5
6
7
8
9
10
11
12
13
14
NHIS “partially
completed”
interview (%)
10
21
24
22
17
20
19
16
23
19
25
26
(%) unweighted response
MEPS Round 1 response rates
(reporting unit level) by NHIS interview
completion status: 2004-2009
100
90
80
70
60
50
40
30
20
10
0
Completes
Partials
2004
2005
2006
2007
Year
2008
2009
Percent refused MEPS interview (at the
RU level) by round and NHIS interview
status, MEPS Panel 12
Round
NHIS Partial
(%)
NHIS Complete
(%)
1
28.7*
15.9
2
10.3*
5.5
3
4.7
3.6
4
3.2
2.6
5
1.8
1.1
*p<.001
Percent ever refused (reluctant respondent or final
refusal) by Round and NHIS interview completion
status: RU level MEPS Panel 12
40
35
Percent
30
25
20
15
10
Partials
5
Completes
0
R1*
R2*
R3*
Round
R4
R5
Percent of noncontacts and true contacts by
NHIS interview status: MEPS Panel 12, Round 1
Paradata
Round 1 noncontacts
4+
NHIS
partial
NHIS
complete
Signif
34.7
26.9
p<.001
Round 1 “true” contacts
p<.001
1
2-3
4-5
6-10
11-14
15+
11.1
42.1
21.5
18.5
3.7
13.3
51.2
17.5
13.8
2.5
3.0
1.6
Other paradata measures by NHIS
interview status: MEPS Panel 12, Round 1
Paradata
Reluctant respondent (%)
Not located (%)
MEPS interview type
In person (%)
Phone (%)
Mean interview time
(minutes)
Any break-offs during the
interview (%)
NHIS
NHIS
partial complete
Signif
14.3
8.4
p<.001
6.1
3.3
p<.001
90.2
9.8
110.3
93.8
6.2
107.3
p<.001
10.7
9.2
p=.006
p=.232
Bivariate analysis of R1 respondent
characteristics (reference person) by NHIS
interview completion status: MEPS Panels 11 &
12 combined
Characteristic
Overall (%)
Partial (%)
Complete(%)
Age of Reference person*
65+
18.1
13.0
19.2
Race/ethnicity *
Hispanic
Non-Hispanic Black
Non-Hispanic Asian
Non-Hispanic Other
19.7
17.4
4.6
58.3
26.5
17.9
6.7
48.9
18.2
17.2
4.2
60.4
Marital Status*
Married
Widowed/separated/divorced
Never married
49.8
28.3
21.8
56.9
23.4
19.6
48.2
29.4
22.3
23.5
14.2
25.1
26.2
11.0
22.8
13.1
24.3
24.5
15.3
23.7
14.5
25.3
26.6
10.0
Year 1 Poverty Status*
In/near poverty
Low income
Middle income
High income
Missing
*significant at alpha = 0.05
Bivariate analysis (cont.) of R1 respondent characteristics
(MEPS reference person) by NHIS interview completion
status: MEPS Panels 11 & 12 combined
(* = significant at alpha = .05)
Characteristic
Overall (%) Partial (%) Complete (%)
Size of RU*
1
2
3
4
5+
28.9
28.5
15.8
14.3
12.4
19.1
25.4
19.2
18.5
17.8
31.1
29.2
15.1
13.4
11.2
Employment status*
Employed
65.3
69.4
64.4
Region*
Northeast
Midwest
South
West
16.3
21.2
38.0
24.5
17.3
18.1
35.8
28.8
16.0
21.9
38.5
23.6
MSA status*
MSA
Non MSA
83.7
16.3
87.9
12.1
82.7
17.3
Logistic regression analysis of Round 1 (RU)
respondent (reference person charac) being a
prior NHIS partial complete: MEPS Panels 11 &
12
Effect
DF
Wald X
Race/ethnicity
Yr1 poverty status
No. of people in RU
Family structure
Employment status
Region
MSA status
3
5
4
5
2
3
1
52.1
41.9
21.8
28.5
15.0
10.2
15.3
2
2
Pr > X
<.0001
<.0001
.0002
<.0001
.0006
.0167
<.0001
Odds ratios of logistic model predicting a
“NHIS Partial Complete” respondent at
MEPS Round 1, Panels 11 & 12
Race/ethnicity
Asian
Black
Hispanic
White/Other
1.61*
1.31*
1.45*
1.00
Reporting Unit Size
1
0.26*
2
0.57*
3
0.77*
4
0.90
5+
1.00
Region
Midwest
Northeast
South
West
0.84*
0.95
0.85*
1.00
MSA
MSA
Non-MSA
1.31*
1.00
Poverty Status
In poverty
Low income
Middle income
Missing income
Near poverty
High income
1.01
0.95
1.01
1.52*
1.10
1.00
Employment Status
Employed
1.14*
Missing/<16 yr 2.42*
Unemployed
1.00
Predicting Round 1 dwelling unit level response
propensity -- 29 potential (NHIS) covariates as
used in MEPS weights production
Demographic
Household
Characteristics
SocioEconomic
Status
Geographic
Age ref. person
DU size
Poverty status
Census region
Health
status
Race/ethnicity
Has phone
Education
MSA size
Need help
Marital status
Working or not
Income
MSA/nonMSA
# nts. in
hospital
Gender
Type of PSU
Employment
status
Urban/Rural
Healthcare
coverage
Any Asian
Type of home –
house, apt., etc.
Home
ownership
Any Black
Time w/ no phone
Interview
language
U.S. citizen
U.S. born
Healthrelated
Medical
expenses
category
Additional NHIS para data included in logit
regression (predicting round 1 response
propensity) -- * available for MEPS panel 12 only
 NHIS interview completion status (partial vs. complete)
–
Panels 11 and 12
 Household cooperativeness*
 How likely respond to later linked survey*
 # of contacts and # of non-contacts*
 Language problem*
 Health problem*
 Time constraints*
 Content/privacy concerns*
 Hostility mentioned*
Results of logistic regression predicting round 1
DU level response propensity -- MEPS Panel 11
Effect
DF
Predicted poverty
Age
Race/ethnicity
Marital status
Income
DU size
MSA status
Region
Has phone
Home type
Interview language
U.S. born
NHIS interview completion status
1
4
3
1
4
4
2
3
3
1
2
2
1
Wald X2
12.6
10.4
24.6
6.3
48.3
36.5
21.0
18.7
27.0
7.8
6.9
11.8
105.4
Pr > X2
.0004
.0338
<.0001
.0121
<.0001
<.0001
<.0001
.0003
<.0001
.0053
.0325
.0028
<.0001
Results of logistic regression predicting round 1
response propensity -- MEPS Panel 12
(additional NHIS paradata available)
Effect
Race/ethnicity
Marital status
Income
DU size
MSA status
Region
Has phone
PSU type
Interview language
Health status
NHIS interview completion status
Med expend category
HH cooperativeness
Likely to respond to later survey
# contacts w/ sample unit
# noncontacts
Privacy concerns
Hostility mentioned
DF
3
1
4
4
2
3
3
1
2
2
1
4
4
1
6
5
1
1
Wald X2
33.9
4.2
15.3
21.3
6.0
21.0
19.6
4.5
12.3
11.2
3.9
1.4
28.3
10.6
35.5
30.7
20.6
5.5
Pr > X2
<.0001
.0405
.0041
.0003
.0496
.0001
.0002
.0340
.0021
.0037
.0500
.0343
<.0001
.0011
<.0001
<.0001
<.0001
.0194
RU level item nonresponse rates, MEPS Panel 12
(if any respondent in the RU had a missing value
in year 1)
Variable
NHIS
Partial (%)
NHIS
Complete (%)
Signif
Yr1 poverty status
15.8
10.3
p<.001
Education
3.9
2.1
p=.002
High blood pressure (>17)
2.4
0.9
p<.001
High cholesterol (>17)
2.3
1.2
p=.025
Diabetes diagnosis
1.5
0.8
p=.059
How often dental check up
5.3
2.5
p<.001
How long last routine check up (>17)
10.7
6.4
p<.001
How long last flu shot (>17)
9.6
4.9
p<.001
How long since mammogram (>29)
8.6
4.5
p<.001
How long since last PSA (>39, M)
17.0
10.5
p<.001
Usual source of care
5.2
2.2
p<.001
Employment status
3.7
2.3
p=.014
Mental health status
2.1
1.1
p=.014
Health status
1.6
0.9
p=.059
Comparison of selected Year 1 estimates
(percents) according to NHIS interview completion
status: MEPS Panels 11 & 12 (* = signif p <.05)
Person level
health
measure
Panel 11
Panel 12
Partial
Complete
Partial
Complete
Any private
68.3
71.4*
65.0
68.9
Public only
15.2
15.3
17.4
15.9
Uninsured
16.5
13.3
17.6
15.2
Any activity
limitation*
7.3
10.4
6.1
9.5
High cholesterol
dx (>17)*
20.7
25.5
24.4
30.5
Office based
provider visit*
68.0
75.2
65.7
72.6
Rx (including
refills)*
58.6
65.2
55.1
63.4
Total mean
expenditures*
$2,760
$3,815
$3,171
$3,943
5+ contacts
$2,484
$3,390*
$3,021
$3,823
Insurance (<65)
MEPS Year 1 estimates (in percent) for All MEPS
respondents versus excluding NHIS partial cases
(person weights for NHIS completes re-calculated treating partials
as nonrespondents) (* = signif p<.05)
Person level
health
measure
Panel 11
All
Completes
Only (rewt)
Panel 12
All
Completes
Only (rewt)
Insurance (<65)
Any private
70.7
71.1
68.1
68.6
Public only
15.3
15.5
16.2
16.0
Uninsured
14.0
13.5
15.7
15.3
Any activity
limitation
9.8
10.0*
8.9
9.1
High cholesterol
dx (>17)*
24.5
25.1
29.5
30.0
Office based
provider visit*
73.7
74.7
71.4
72.1
Rx (including
refills)*
63.8
64.4
61.9
62.6
Total mean
expenditures
$3,593
$3,723*
$3,802
$3,836
5+ contacts
$3,162
$3,313*
$3,647
$3,702
Summary
 We examined the impact of the carry-over of two types
of prior interview cases in the MEPS as a result of its
integration with the NHIS.
 In particular, we wanted to examine if the partially
completed NHIS cases may bias MEPS survey
estimates and impact survey burden.
 NHIS partial cases had significantly higher ever refused
rates, higher contact rates, lower response rates at
MEPS Round 1, and higher attrition rates.
 Partial versus complete was a significant predictor of
MEPS Round 1 response propensity controlling for
other variables.
Summary (cont.)
 Item nonresponse rates were higher for the
partially completed cases relative to the NHIS
completes for a number of MEPS key variables.
 Estimates for several selected health items were
different between the NHIS partials and
completes as carried over in the MEPS sample.
 Excluding the NHIS partial cases from the MEPS
survey estimates (and re-weighting) resulted in
slightly higher estimates of selected conditions,
medical events, and expenditures.
 Thanks
Discussion
 Initial exploratory analysis
– What additional analyses can be carried out to
assess potential nonresponse bias and
measurement error bias related to the two types of
interview cases in the MEPS?
– What approaches could be explored to correct for
nonresponse or measurement error bias?
 Should MEPS consider excluding or subsampling the
NHIS partials in future panels of the MEPS?
 Ideas/suggestions for future research?
Logistic regression analysis of the insured (<65 years)
reference person: testing for NHIS interview status
effect, MEPS, Panels 11 and 12 (RU level analysis)
Panel 11
DF
Panel 12
Wald F
p-value
DF
Panel 11 + 12
Wald F
P-value
DF
Wald F
p-value
Age
3
9.13
.0276
1
4.75
.0029
3
21.81
<.0001
Sex
1
10.92
.0009
1
7.67
.0056
1
18.61
<.0001
Race/ethnicity
3
99.39
<.0001
1
111.99
<.0001
3
206.47
<.0001
Highest year of
education
4
69.78
<.0001
4
52.75
<.0001
4
122.32
<.0001
Poverty status (Y1)
5
237.16
<.0001
5
174.76
<.0001
5
406.38
<.0001
Region
3
39.01
<.0001
3
53.02
<.0001
3
86.51
<.0001
Total health care
Expenditures
6
233.63
<.0001
6
194.35
<.0001
6
417.21
<.0001
NHIS Interview
Completion Status
1
0.36
.5477
1
4.75
.0293
1
2.93
.0868