Interviewer Performance, Responsive Design, and the
Download
Report
Transcript Interviewer Performance, Responsive Design, and the
Responsive Design on the National Health
Interview Survey: Opportunities and Challenges
Renee M. Gindi
NCHS
Federal Conference on Statistical Methodology
Statistical Policy Seminar
December 4, 2012
National Center for Health Statistics
Division of Health Interview Statistics
1
Objectives
National Health Interview Survey (NHIS) background
Potential features of responsive design on NHIS
Opportunities
Challenges
2
The National Health Interview Survey (NHIS)
Conducted by National Center for
Health Statistics
Nationally representative
Representative monthly sample
In-person interviews
35-40,000 household interviews/year
Fielded by U.S. Census Bureau
~700 interviewers in 6 regional
offices
1 hour face-to-face
interview – no
incentives
3
Refusal Rates, NHIS 1969-2011
20%
12.9%
10%
1.3%
0%
1970
1975
1980
1985
1990
1995
2000
2005
2010
4
Sources of Paradata on NHIS
Contact History Instrument (CHI)
Used on other surveys fielded by Census
Front/Back sections of the survey instrument
Tailored to NHIS
Blaise audit trails
Used to produce item/interview times
5
Recent Paradata Research from NHIS
Using Statistical Process Control to monitor data quality
estimates (item nonresponse, item time) over time
Using CHI variables to estimate response propensity
response propensity and measurement error
response propensity and survey outcomes
6
Looking Ahead: Responsive Design on NHIS
Some elements of responsive design
Monitoring performance indicators
Change design based on monitoring survey outcomes
Target interventions to subsets using response propensity
Timeline: 2016 sample redesign
7
Looking Ahead: Responsive Design on NHIS
Opportunities
Real-time access to operations data
New ways to estimate survey quality
Challenges
Selecting and prioritizing survey outcome estimates
How, when, and where data collection phases should shift
8
Real-Time Access to Operations Data
Census Bureau’s Unified Tracking System (UTS)
More information to make better decisions quickly
Daily data update and historical data
Flexibility in reports
NHIS-specific indicators on tracked on UTS
Demographic
Race
Income
Education
Employment
Health
Usual place of care
Needs help with personal care
Response quality
First /Last Name
Consent for linkage
Adult SSN
Telephone number
9
New Ways to Estimate Survey Quality
Trying to identify measures that can help assess,
reduce, and correct for nonresponse bias in our
health estimates
Adding new interviewer observation questions on
responders and non-responders
Physical condition of the sample unit
Household income, employment status
Health-related indicators
10
Identifying priority estimates:
15 Selected Health Measures
Lack of health insurance
coverage and type of coverage
Usual place to go for medical
care
Obtaining needed medical care
Receipt of influenza vaccination
Receipt of pneumococcal
vaccination
Obesity
Leisure-time physical activity
Current smoking
Alcohol consumption
Human immunodeficiency virus
(HIV) testing
General health status
Personal care needs
Serious psychological distress
Diagnosed diabetes
Asthma episodes and current
asthma
11
“Phase Shifts”: How, When, and Where?
How can we “sufficiently alter” NHIS protocol?
Mode shift? Shift to “core” survey? Introduce incentives?
When should protocol be altered given a monthly sample
and production cycle?
Is a 7-10 day window wide enough to achieve response goals?
Where should protocol be altered?
Nationally? Regional Office? State?
12
Thank you!
Renee M. Gindi, Ph.D.
Email: [email protected]
Phone: 301-458-4502
13
EXTRA SLIDES
14
SPC: Sample Adult Interview Pace (seconds per question)
Control Chart, Regional Office 1, Cluster 4
Standard Deviation
Average
3s Limits:
12
10
8
6
4
1
3
UCL
Z=8.1
LCL
20
UCL
10
S
LCL
0
J
a
n
M
a
r
M
a
y
J
u
l
S
e
p
N
o
v
J
a
n
M
a
r
M
a
y
J
u
l
S
e
p
N
o
v
J
a
n
M
a
r
M
a
y
J
u
l
S
e
p
N
o
v
2
0
0
8
2
0
0
8
2
0
0
8
2
0
0
8
2
0
0
8
2
0
0
8
2
0
0
9
2
0
0
9
2
0
0
9
2
0
0
9
2
0
0
9
2
0
0
9
2
0
1
0
2
0
1
0
2
0
1
0
2
0
1
0
2
0
1
0
2
0
1
0
Month
Subgroup Sizes:
Min n = 6
Max n = 194
15
Response Propensity and Measurement Bias
Social Security, Supplemental Security Income, Medicare
Bias (Survey mean - Record mean)
1.50%
1.00%
0.50%
Medicare
0.00%
-0.50%
SSI
-1.00%
-1.50%
SS
-2.00%
-2.50%
LOW
MEDIUM
RESPONSE PROPENSITY
HIGH
16
Correlations between CHI Measures and Participation and Health
Outcomes : NHIS, 2010
CHI Measure
Absolute Value of
Correlation with
Family/Sample Adult
Participation
Absolute Value of
Correlations with 76
Health Outcomes
Range
Average
Refusal concerns
.73/.50
.00 - .10
.04
Time constraints
.15/.42
.00 - .14
.05
Privacy/trust
.45/.38
.00 - .11
.04
Gate keeping
.27/.32
.00 - .11
.03
Number of concerns
.62/.45
.00 - .09
.03
Number of contact
attempts
.49/.31
.00 - .25
.07
Health problem
.19/.18
.01 - .33
.13
< .30: Weak/very weak
.30 - .69: Moderate
>= .70: Strong/very strong 17