Transcript Folie 1

Nonresponse and Measurement Error in
Employment Research
Gerrit Müller (IAB)
joint with: Frauke Kreuter (JPSM – U Maryland), Mark Trappmann (IAB)
1
Research Questions
 Do survey respondents recruited with extra effort, provide
answers of lower quality?
 Are cooperators more motivated to provide accurate data?
 Or, are late respondents hampered by recall deficits?
 How does extra effort affect total bias?
2
Survey Data
 Panel Study “Labor Market and Social Security” (PASS)
 Dual frame survey (benefit recipients / residential population)
 Wave1: 12,000 HH
20,000 P
 RR1: 30.5% (within HH: 85%)
 Mixed mode survey (sequential CATI -> CAPI)
3
Record Linkage I
 Individual survey data linked with individual administrative data
(80% of all Rs agreed; 72% successfully matched)
 Administrative records on: employment, earnings,
unemployment, labor market programs
yR,survey , yR,records
 Contact data on HH-level only
4
Record Linkage II
 Administrative data linked with paradata for the gross sample of
recipients (from unemployment register)
 Contact data on HH-level only
 Indicator for Respondents / Nonrespondents on HH-level only
yR NR ,records , yR,records
5
Hypotheses about Measurement Error
(response process model, Tourangeau ’84)
 Unemployment benefit (UBII)
 July 2006
 Nov 2006
 at time of interview
Relationship between ME
and response propensity
(number of contact attempts)
 Income in month prior to interview
 Occupation
 Educational degree
6
Contact Quintiles and Follow-Up Efforts
# of contacts:
Q1 (high contactability)
Q2
Q3
Q4
Q5 (low contactability)
_min
1
3
5
8
≥15
_max
2
4
7
14
 Transfer CATI to CAPI
 CATI NR follow-up of “soft refusals”
7
Measurement Error (in percent)
by Contact Quintiles and Follow-up Efforts
Q1 (high contactability)
Q2
Q3
Q4
Q5
To CAPI
UB II
July 2006
12
13
14
16
20
18
UB II
Interview Date
11
12
13
17
15
14
CATI NR follow-up
21
14
8
Measurement Error
by Contact Quintiles and Follow-up Efforts
Q1 (high contactability)
Q2
Q3
Q4
Q5
To CAPI
Income
(abs. dev.)
311
343
370
335
380
451
Years of Edu.
(% mismatch)
25
28
27
26
24
25
CATI NR follow-up
407
27
9
10
5. NR-ME Bias Decomposition
 for Recipient sample only
 HH-level variables only (!)
total bias( y )  ( y NR  R ,records  yR ,survey )
 ( y NR  R ,records  yR ,records )  ( yR ,records  yR ,survey )
 UBII in Jul06 not feasible for bias decomposition
 UBII in Nov
 UBII at date of interview
11
Figure 1. Cumulative mean over (quintiles of) no. of contact
attempts; UB II recipience in Nov 06
1
UB II recipience in Nov 06
0,95
0,9
Records
0,85
Target
Survey reports
0,8
0,75
0,7
1 (High)
2
3
4
5 (Low)
NR f'up
to CAPI
Contactability
12
Figure 2. Cumulative mean over (quintiles of) no. of contact
attempts; UB II recipience at date of interview
UB II recipience at date (month) of
interview / last contact
1
0,95
0,9
Records
0,85
Target
Survey reports
0,8
0,75
0,7
1 (High)
2
3
4
5 (Low)
NR f'up
to CAPI
Contactability
13
“Pick your brains”
 ME-Model for UBII in July06 (handout)
Puzzle: high ME for the young? HH-interview by target head?
 Administrative data not always „Gold Standard“ (error-free)
Assumption: ME in register data unrelated to ME in survey
reports and response propensity
14
“Pick your brains”
 Decomposition findings statistic-specific
 Extend analyses to P-level variables (e.g. employment,
income)
 Problem: yR,records unknown on individual level
 How to go ahead?
15