Transcript Interviewer Performance, Responsive Design, and the

Responsive Design on the National Health
Interview Survey: Opportunities and Challenges
Renee M. Gindi
NCHS
Federal Conference on Statistical Methodology
Statistical Policy Seminar
December 4, 2012
National Center for Health Statistics
Division of Health Interview Statistics
1
Objectives

National Health Interview Survey (NHIS) background

Potential features of responsive design on NHIS
 Opportunities
 Challenges
2
The National Health Interview Survey (NHIS)

Conducted by National Center for
Health Statistics

Nationally representative


Representative monthly sample
In-person interviews

35-40,000 household interviews/year


Fielded by U.S. Census Bureau

~700 interviewers in 6 regional
offices
1 hour face-to-face
interview – no
incentives
3
Refusal Rates, NHIS 1969-2011
20%
12.9%
10%
1.3%
0%
1970
1975
1980
1985
1990
1995
2000
2005
2010
4
Sources of Paradata on NHIS

Contact History Instrument (CHI)
 Used on other surveys fielded by Census

Front/Back sections of the survey instrument
 Tailored to NHIS

Blaise audit trails
 Used to produce item/interview times
5
Recent Paradata Research from NHIS

Using Statistical Process Control to monitor data quality
estimates (item nonresponse, item time) over time

Using CHI variables to estimate response propensity


response propensity and measurement error
response propensity and survey outcomes
6
Looking Ahead: Responsive Design on NHIS

Some elements of responsive design
 Monitoring performance indicators
 Change design based on monitoring survey outcomes
 Target interventions to subsets using response propensity

Timeline: 2016 sample redesign
7
Looking Ahead: Responsive Design on NHIS

Opportunities



Real-time access to operations data
New ways to estimate survey quality
Challenges


Selecting and prioritizing survey outcome estimates
How, when, and where data collection phases should shift
8
Real-Time Access to Operations Data

Census Bureau’s Unified Tracking System (UTS)
 More information to make better decisions quickly
 Daily data update and historical data
 Flexibility in reports

NHIS-specific indicators on tracked on UTS
Demographic
Race
Income
Education
Employment
Health
Usual place of care
Needs help with personal care
Response quality
First /Last Name
Consent for linkage
Adult SSN
Telephone number
9
New Ways to Estimate Survey Quality

Trying to identify measures that can help assess,
reduce, and correct for nonresponse bias in our
health estimates

Adding new interviewer observation questions on
responders and non-responders
 Physical condition of the sample unit
 Household income, employment status
 Health-related indicators
10
Identifying priority estimates:
15 Selected Health Measures








Lack of health insurance
coverage and type of coverage
Usual place to go for medical
care
Obtaining needed medical care
Receipt of influenza vaccination
Receipt of pneumococcal
vaccination
Obesity
Leisure-time physical activity
Current smoking







Alcohol consumption
Human immunodeficiency virus
(HIV) testing
General health status
Personal care needs
Serious psychological distress
Diagnosed diabetes
Asthma episodes and current
asthma
11
“Phase Shifts”: How, When, and Where?

How can we “sufficiently alter” NHIS protocol?
 Mode shift? Shift to “core” survey? Introduce incentives?

When should protocol be altered given a monthly sample
and production cycle?
 Is a 7-10 day window wide enough to achieve response goals?

Where should protocol be altered?
 Nationally? Regional Office? State?
12
Thank you!
Renee M. Gindi, Ph.D.
Email: [email protected]
Phone: 301-458-4502
13
EXTRA SLIDES
14
SPC: Sample Adult Interview Pace (seconds per question)
Control Chart, Regional Office 1, Cluster 4
Standard Deviation
Average
3s Limits:
12
10
8
6
4
1
3
UCL
Z=8.1
LCL
20
UCL
10
S
LCL
0
J
a
n
M
a
r
M
a
y
J
u
l
S
e
p
N
o
v
J
a
n
M
a
r
M
a
y
J
u
l
S
e
p
N
o
v
J
a
n
M
a
r
M
a
y
J
u
l
S
e
p
N
o
v
2
0
0
8
2
0
0
8
2
0
0
8
2
0
0
8
2
0
0
8
2
0
0
8
2
0
0
9
2
0
0
9
2
0
0
9
2
0
0
9
2
0
0
9
2
0
0
9
2
0
1
0
2
0
1
0
2
0
1
0
2
0
1
0
2
0
1
0
2
0
1
0
Month
Subgroup Sizes:
Min n = 6
Max n = 194
15
Response Propensity and Measurement Bias
Social Security, Supplemental Security Income, Medicare
Bias (Survey mean - Record mean)
1.50%
1.00%
0.50%
Medicare
0.00%
-0.50%
SSI
-1.00%
-1.50%
SS
-2.00%
-2.50%
LOW
MEDIUM
RESPONSE PROPENSITY
HIGH
16
Correlations between CHI Measures and Participation and Health
Outcomes : NHIS, 2010
CHI Measure
Absolute Value of
Correlation with
Family/Sample Adult
Participation
Absolute Value of
Correlations with 76
Health Outcomes
Range
Average
Refusal concerns
.73/.50
.00 - .10
.04
Time constraints
.15/.42
.00 - .14
.05
Privacy/trust
.45/.38
.00 - .11
.04
Gate keeping
.27/.32
.00 - .11
.03
Number of concerns
.62/.45
.00 - .09
.03
Number of contact
attempts
.49/.31
.00 - .25
.07
Health problem
.19/.18
.01 - .33
.13
< .30: Weak/very weak
.30 - .69: Moderate
>= .70: Strong/very strong 17