Transcript a UK ESS experiment

Feasibility of Using Web to
Survey at a Sample of
Addresses:
a UK ESS experiment
Ana Villar
City University London
MANY THANKS TO
NatCen team
Alison Park (UK ESS NC)
Maya Agur
Alun Humphrey
City University London team
Rory Fitzgerald
Lizzy Gatrell
Yvette Prestage
www.europeansocialsurvey.org
RESULTS ARE VERY
PRELIMINARY AND SHOULD
BE TAKEN WITH CARE; DATA
WERE DELIVERED ONLY TWO
WEEKS AGO AND WE ARE
STILL IN THE PROCESS OF
CLEANING THE DATA.
www.europeansocialsurvey.org
Study Design and Goals
Co-funded by the ESS (EC) and the ESRC
Feasibility of implementing a web-F2F survey using
an address-based sample
• Careful contact and communication design
• Within-household random self-selection
• Monetary incentives
Compare to ESS main stage F2F
Sequential:
Web then F2F (N = 500)
Web only (N = 1,500)
www.europeansocialsurvey.org
Experiments
Within-household random self-selection:
• Asking household member with next/last birthday to
go to URL and complete survey
• Household informant fills out roster, respondent is
randomly selected
Incentive upon completion (choice between evoucher and postal voucher):
• £15
• £35
www.europeansocialsurvey.org
Contact Protocol
Web stage (full sample):
•
•
•
•
Pre-notice letter
Invitation letter with £5 voucher and instructions leaflet
Reminder letter
Upon completion  Thank you + Incentive
F2F stage (mixed-mode subsample):
• Of the 500 sampled units from the subsample, those
who did not reply to web survey were contacted at
home by an interviewer following ESS protocol
www.europeansocialsurvey.org
Pre-notice letter
Advance notification of
invitation
Survey topic
Advance notification of
unconditional incentive
Thank you
www.europeansocialsurvey.org
Invitation/Cover letter
Reminder of previous letter
What the ESS is about;
“opportunity to have a say”
Legitimacy – NatCen/ESRC
Scarcity; refer to leaflet
Enclosed incentive and
conditional incentive
Thank you, signature,
confidentiality assurance
www.europeansocialsurvey.org
Instructions Leaflet:
Household Roster
Survey topic
Who is the respondent
How to + incentive
URL and unique ID
Smartphone not allowed
Privacy assurance,
password, thank you
www.europeansocialsurvey.org
Instructions Leaflet:
Last/Next Birthday
Survey topic
Who is the respondent
How to + incentive
URL and unique ID
Smartphone not allowed
Privacy assurance,
password, thank you
www.europeansocialsurvey.org
Household roster
Step 1 – Number of adults
www.europeansocialsurvey.org
Household roster
Step 2 – Listing all adults
www.europeansocialsurvey.org
Household roster
Step 3 – Confirmation of list
www.europeansocialsurvey.org
Household roster
Step 4a – Computer selects HH informant
www.europeansocialsurvey.org
Household roster
Step 4b – Computer selects another HH member
www.europeansocialsurvey.org
PRELIMINARY FINDINGS
www.europeansocialsurvey.org
Response Rates (RR1)
80%
70%
60%
55%
50%
39%
40%
Web
Web + F2F
30%
21%
20%
10%
0%
F2F
Response Rates
www.europeansocialsurvey.org
(RR1)
Response Rates by Selection Method
HH ROSTER
(N = 1,000)
BIRTHDAY
(N = 1,000)
Followed link
NA
NA
Completed roster
NA
NA
21.1%
20.4%
20.1%
20.0%
19.2%
19.1%
Started (Q1)
Completes + Partials
Completed Web Survey
www.europeansocialsurvey.org
Response Rates by Incentive
£15
(N = 1,000)
£35
(N = 1,000)
Followed link
NA
NA
Completed roster
NA
NA
19.4%
18.7%
18.5%
21.7%
20.9%
20.7%
Started (Q1)
Completes + Partials
Completed Web Survey
www.europeansocialsurvey.org
No Interaction Effect
www.europeansocialsurvey.org
Compliance with instructions (n = 112)
30% not applicable – single households or only
adult
50% comply
20% do not comply
Rest a combination of refusals, break-offs,
impossible to tell
So far number of “cheaters” is similar for both last
and next birthday
www.europeansocialsurvey.org
Respondent Behaviour
Average duration: paradata not yet available
Typical number of sessions: 1
About half started before the first reminder was sent
www.europeansocialsurvey.org
Gender by Mode
90%
80%
70%
60%
57%
60%
54%
50%
F2F
40%
Web
Web + F2F
30%
20%
10%
0%
www.europeansocialsurvey.org
Female
Age by Mode
50%
44%
45%
40%
42%
36%
34%
35%
29%
30%
30%
27%
25%
F2F
21%
20%
15%
10%
Web
Web + F2F
11%10%
8%
10%
5%
0%
www.europeansocialsurvey.org
15-24
25-44
45-64
65 or more
Years of Education by Mode
60%
49%
50%
42%
40%
30%
26%
27%
22% 22%
29%
24%
22%
F2F
Web
Web + F2F
20%
10%
7%6%
3%
11%
5% 6%
0%
Less than 9 9-12 years 13-15 years 16-19 years 20 or more
www.europeansocialsurvey.org
years
years
Job Situation by Mode
80%
71%
70%
60%
54%
53%
50%
F2F
Web
Web + F2F
40%
30%
20%
10%
0%
Has a paid job
www.europeansocialsurvey.org
Household Income by Mode
45%
39%
40%
35%
30%
35%
31% 31%
27%
30%
25%
25%
24%
F2F
22%
Web
20%
17%
15%
10%
6%
5%
2%
5%
4%
1%
0%
www.europeansocialsurvey.org
Low
Medium
High
DK
Refused
Web + F2F
Political Interest by Mode
45%
40%41%
38%
40%
35%
33%
30%
27%
30%
25%
23%
20%
15%
15%16%
13%
12%12%
10%
5%
0%
Very interested
Quite
www.europeansocialsurvey.org
interested
Hardly
interested
Not at all
F2F
Web
Web + F2F
Conclusions and Lessons Learned
It is feasible to get a substantial number of people to
complete an online survey
There seem to be some differences in sample
composition by mode
Costs and time necessary to program survey tend to
be largely underestimated
www.europeansocialsurvey.org
Thank you
[email protected]
www.europeansocialsurvey.org
Income by Mode (without item NR)
45%
40%
35%
30%
41%
39%
40%
34%
32%
28%
31%
28%
27%
25%
F2F
20%
Web
Web + F2F
15%
10%
5%
0%
www.europeansocialsurvey.org
Low
Medium
High
Job Situation by Mode
80%
71%
70%
60%
53%
54%
50%
47%
46%
F2F
40%
30%
29%
20%
10%
0%
www.europeansocialsurvey.org
Has a paid job
No paid job
Web
Web + F2F
Gender by Mode
70%
57%
60%
50%
40%
43%
60%
54%
46%
40%
F2F
Web
Web + F2F
30%
20%
10%
0%
Male
www.europeansocialsurvey.org
Female
Economic Hardship by Mode
90%
81%
81%
80%
70%
70%
60%
50%
F2F
40%
Web
30%
30%
20%
19%
Web + F2F
19%
10%
0%
www.europeansocialsurvey.org
Difficulties
Coping or Comfortable
Urbanicity by mode
90%
80%
79%
81%
74%
70%
60%
50%
F2F
40%
Web
30%
20%
21%
26%
19%
10%
0%
www.europeansocialsurvey.org
Urban
Web + F2F
Rural
www.europeansocialsurvey.org
Number of completes by date (roster)
50
45
40
Reminder sent
35
30
25
20
15
10
5
0
www.europeansocialsurvey.org
Number of completes by date (birthday)
45
40
35
Reminder sent
30
25
20
15
10
5
0
www.europeansocialsurvey.org