Transcript EFFECTEN VAN NON-RESPONS

Increasing response rates
Ineke Stoop
SCP
Quantitative Methods in the Social Sciences (QMSS) Lugano 25-26 August 2005
Is it possible?
Quantitative Methods in the Social Sciences (QMSS) Lugano 25-26 August 2005
and is it effective?
Quantitative Methods in the Social Sciences (QMSS) Lugano 25-26 August 2005
Nonresponse research
• How to increase response rates?
• How to measure and correct for
nonresponse bias?
Quantitative Methods in the Social Sciences (QMSS) Lugano 25-26 August 2005
SCP-survey
• Face-to-face questionnaire + long drop
off for every 6+ years household
member
• Use of social and cultural services,
amenities, facilities (no values, opinions)
• Very low non-contact rate
• High rate of refusal conversion
• Final response rate 1999 66%
Quantitative Methods in the Social Sciences (QMSS) Lugano 25-26 August 2005
Response rates AVO
70
60
50
40
30
20
10
0
1979
1983
1987
1991
1995
1999
2003
Quantitative Methods in the Social Sciences (QMSS) Lugano 25-26 August 2005
Cumulative response rates
70
60
50
40
30
20
10
0
1
2
3
4
5
6
7
8
9
10 11 12 13 14 15 16 17 18 19 20
1995
1999
Quantitative Methods in the Social Sciences (QMSS) Lugano 25-26 August 2005
Increase response rates
Describe response process
Contactability
Reluctance
Quantitative Methods in the Social Sciences (QMSS) Lugano 25-26 August 2005
Paradata
•
•
•
•
•
•
Call records (nr, timing, outcome)
Characteristics interaction
Interviewer observations
Sample frame data
Low-level geographic data bases
Interviewer characteristics
Quantitative Methods in the Social Sciences (QMSS) Lugano 25-26 August 2005
Contactability
TNC/NAH/CFC
Call (nr, time, mode)
First
contact
Quantitative Methods in the Social Sciences (QMSS) Lugano 25-26 August 2005
At home, interviewer calls,
contact rate
morning
100
afternoon
evening
90
70
80
60
70
50
60
50
40
40
30
30
20
20
10
10
0
0
6
12
18
24
Quantitative Methods in the Social Sciences (QMSS) Lugano 25-26 August 2005
Timing contacts
• Interviewers prefer working hours
• Contact rates higher in the evening
• Cooperation independent of timing
• Fewer calls if evening calls only
• Length fieldwork period/costs?
Quantitative Methods in the Social Sciences (QMSS) Lugano 25-26 August 2005
Contactability
• Easy
– Evening calls
– Phone number
available
– Detached dwelling
– Child at home
• Difficult
– Big city dweller
– Poor maintenance
neighbourhood
– Young
– Single
– Employed, student
– High cultural participation
Quantitative Methods in the Social Sciences (QMSS) Lugano 25-26 August 2005
Continuum of resistance
• Extrapolate from hard to reach
respondents to noncontacts?
– Socio-demographics
– Core variables
– Two types of hard to contact (short
term/long term)
Quantitative Methods in the Social Sciences (QMSS) Lugano 25-26 August 2005
Reluctance
Respondent has been contacted
Quantitative Methods in the Social Sciences (QMSS) Lugano 25-26 August 2005
Reluctance
Refusal
Establish
first contact
No
Yes
Hard converted
refusal
Request to
cooperate
Request to
cooperate
Request to
cooperate
Yes
Interview
No Follow Yes
-up
No
Refusal
Yes
Cooperative
respondent
Interview
Yes
Contact
No
Refusal
Interview
No
Yes
Soft converted
refusal
Follow
-up
Yes
Contact
No
No
Refusal
Refusal
Quantitative Methods in the Social Sciences (QMSS) Lugano 25-26 August 2005
What happens in the AVO after a
first refusal?
Interview
Refusal at
first contact
Reapproached
Final
refusal
Final
refusal
Quantitative Methods in the Social Sciences (QMSS) Lugano 25-26 August 2005
Reluctant respondents
• Many among
– Big city dwellers
• Few among
– Men
– Singles
– Higher educated
Quantitative Methods in the Social Sciences (QMSS) Lugano 25-26 August 2005
Cooperative and reluctant respondents
Big cities
Mid east
single male
couple
total
0
10
20
30
40
50
immediate
60
70
80
90
converted
Quantitative Methods in the Social Sciences (QMSS) Lugano 25-26 August 2005
100
Refusers, reluctant and cooperative
respondents
Big cities
Mid east
single male
couple
total
Big cities
Mid east
single male
couple
total
0
10
20
30
40
immediate
50
converted
60
70
80
90
refuse
Quantitative Methods in the Social Sciences (QMSS) Lugano 25-26 August 2005
100
Region and refusal conversion
7
45
48
Amsterdam,
Rotterdam,
The Hague
25
15
51
-34
Mid east
14
-40
-20
0
20
still in the field
immediate nonresponse
re-issue, no 2nd refusal
immediate cooperation
converted nonrespondents
40
60
re-issue, 2nd refusal
Quantitative Methods in the Social Sciences (QMSS) Lugano 25-26 August 2005
Results refusal conversion
• Males, singles and higher educated
underrepresented among converted
refusals
– Do interviewer strategies work for
everybody?
• Are only promising cases re-contacted?
– May worsen final sample composition
• Effect refusal conversion
– Works only for honest burghers, Mr. and
Mrs. Average?
Quantitative Methods in the Social Sciences (QMSS) Lugano 25-26 August 2005
Conclusions regular AVO
• Contactability related to sociodemographics, being out of the house
and fieldwork strategy
• Reluctance related to family
composition, gender and education and
persuasiveness
Quantitative Methods in the Social Sciences (QMSS) Lugano 25-26 August 2005
Do nonrespondents differ?
Are final refusals similar to
converted refusals
• Sample frame information
• Neighbourhood information
• Fieldwork information
– Late respondents (similar to non-contacts?)
– Reluctant respondents (similar to refusers?)
• Socio-demographic information
Quantitative Methods in the Social Sciences (QMSS) Lugano 25-26 August 2005
Response and nonresponse
F
r
a
m
e
F
i
e
l
d
w
o
r
k
O
b
s
e
r
v
a
t
i
o
n
Easy
Hard to contact
Reluctant
Follow-up
Quantitative Methods in the Social Sciences (QMSS) Lugano 25-26 August 2005
Follow-up survey
• Small subsample persistent refusers
• 1 person, 20 minutes, multi-mode
• Experienced, motivated, well paid
interviewers
• Wide range of incentives
• 235 successful interviews
• 70% cooperation rate
Quantitative Methods in the Social Sciences (QMSS) Lugano 25-26 August 2005
Why did refusers cooperate?
• High quality interviewers (and telling them
they are the best)
• Extensive briefing
• Trust (money for incentives) and support
(newsletter)
• Importance (newspaper article)
• Wide range of incentives
• Better payment
• Commitment sponsor and fieldwork
organisation
Quantitative Methods in the Social Sciences (QMSS) Lugano 25-26 August 2005
Idiosyncratic choice of incentives
Quantitative Methods in the Social Sciences (QMSS) Lugano 25-26 August 2005
Final analysis
Hard to contact respondents
Reluctant respondents
Refusers who cooperated in
follow-up survey
Quantitative Methods in the Social Sciences (QMSS) Lugano 25-26 August 2005
Follow-up survey
• Sample of persistent refusals
• Socio-demographic differences (made
up for composition regular respondents)
• Small differences in survey variables
(mostly related to socio-demographics)
• Lower participation in classical culture
• Less PC ownership
• Fewer sports activities
• Slightly less active in many aspects
• Not at all similar to converted refusals
Quantitative Methods in the Social Sciences (QMSS) Lugano 25-26 August 2005
refuser
reluctant
hard to contact
Intercept
-2,42
-0,04
-0,38
Age (years)
-0,01
0,00
-0,01
Age: absolute deviation mean
(years)
-0,01
-0,01
-0,01
0,50
-0,10
-0,26
Amsterdam, Rotterdam, The Hague
-0,22
0,69
0,63
Good maintenance
-0,21
-0,26
-0,26
Detached dwelling
0,00
-0,42
-0,28
-0,16
-0,76
-0,05
0,14
-0,37
0,29
-0,30
-0,11
-0,37
0,17
-0,13
0,31
-0,15
-0,51
-0,03
Both classical and popular
-0,21
-0,03
0,13
Classical
-0,56
0,23
0,17
Popular
0,17
-0,07
0,15
0,41
-0,03
0,00
-0,87
0,18
0,00
Use of internet, e-mail
0,94
0,07
0,14
No religious affiliation
0,38
-0,11
0,00
Listed phone number
Male
Single
Child at home
Employed, student
Higher education
Cultural participation
No sports activities
PC in household
Quantitative Methods in the Social Sciences (QMSS) Lugano 25-26 August 2005
2
25+% ethnic
ethnic minority
1,6
Amsterdam
Rotterdam
The Hague
1,2
poor maintenance
multi-unit
single male
0,8
single
no phone
parent
vey low income
single female
-1,6
75+
primary
16-24
HARD TO
CONTACT
academic
cultural popular
employed, student
omnivore
internet
no vcr
many sports
REFUSER
avid
reader
no
nature
RELUCTANT
no pc
very
0
no sports
nature
Randstad
pc off line
?education?
high
atheist
-0,8
0 Mid east
0,8
no books
income
no culture
couple
not employed,
terraced
EASY
couple
South-0,4
student
?income?
pc not used children
classical
North east
excellent
-0,8
no ethnic minorities
detached
0,4
religious
-1,2
family composition
video
ethnic minority
ethnic neighbourhood
region
dwelling
telephone
age
sports
culture
PC
response
maintenance
employed, student
religion
nature conservancy
books
education
income
Quantitative Methods in the Social Sciences (QMSS) Lugano 25-26 August 2005
Reducing bias instead of
increasing response rates
• Increasing response rates MAY not lead to
better survey estimates
• Difficult respondents may not be similar to
final nonrespondents
– Obtain information on the process (cause
of/reason for nonparticipation)
• Collect independent information on
nonrespondents
– Frame information + registers
– Central question procedure
– Follow-up survey among refusers
Quantitative Methods in the Social Sciences (QMSS) Lugano 25-26 August 2005
Quantitative Methods in the Social Sciences (QMSS) Lugano 25-26 August 2005