Transcript On the computation of response metrics for online panels

Computing response metrics for
online panels
Mario Callegaro
Charles DiSogra
Knowledge Networks
DC AAPOR Workshop on Web Survey Methods, September 9th 2009
© 2009 Knowledge Networks, Inc.
What metrics for what panel
Pre-recruited probability-based online panels
 Response rates can be calculated because the frame is known
(AAPOR, 2006)
Volunteer opt-in panels
 “Response rates” cannot be computed (AAPOR, 2007)
 However, other metrics can be calculated, e.g. completion rate
1
Current status
• Volunteer, non-probability (opt-in) panels, widely used in
market research, outnumber probability-based Web panels
• More and more probability-based online panels being built
 2007-2009 American National Election Studies (ANES) Panel
 Face-to-Face Recruited Internet Survey Platform (FFRISP, 2008)
 Dutch Long-term Internet Study for the Social Science (LISS) panel (2007)
• Still no officially agreed standard on how to compute
response rates for online panels
2
Review of current standards
Many efforts and proposals by different national and
international organizations:
European Society for Opinion and Marketing Research – ESOMAR
European Federation of Associations of Market Research Orgs. –EFAMRO
Interactive Marketing Research Organization – IMRO
Advertising Research Association – ARF quality initiative
Bob Lederer proposal endorsed by the American Marketing Association
(AMA)
 Latest effort by ISO (standard #26362) touches on subject





3
Journal recommendations
Some journals are giving guidelines on how response rates
should be computed specifically for online surveys (not
necessarily online panels)
 Journals enforcing AAPOR standards: (e.g. POQ, IJPOR…)
 Journal of Medical Internet Research
Journal of Medical Internet Research (Eysenbach, 2004):
In online surveys, there is no single response rate.
Rather, there are multiple potential methods for calculating a
response rate, depending on what are chosen as the numerator
and denominator.
As there is no standard methodology, we suggest avoiding the
term “response rate” and have defined how, at least in this journal,
response metrics such as, what we call, the view rate, participation
rate and completion rate should be calculated.
4
ESOMAR and IMRO examples
• ESOMAR (2005) metrics:
 “Response based on the total amount of invites (% of full numbers)
per sample drawn (country, questionnaire)
 % questionnaire opened
 % questionnaire completed (including screen-out)
 % in target group (based on quotas)
 % validated (the balance is cleaned out, if applicable)” (p. 20).
• IMRO (2006) metrics:
 Response rate is “based on the people who have accepted the
invitation to the survey and started to complete the survey. Even if
they are disqualified during screening, the attempt qualifies as a
response” (p. 13).
 Completion rate “is calculated as the proportion of those who have
started, qualified, and then completed the survey” (p. 13).
5
AMA platform for data quality progress:
Platform for Data Quality Progress, Bob Lederer
(under AMA umbrella, Nov 2008)
ResponseRate 
T ot # of attemptedresponses
T otalinvitations/intercepts - (Bouncebacks, Errorsor Request for removal)
Qualification Rate 
T otal# passingscreeeningcriteria
T otal# attemptedresponses
Completes
CompletionRate 
T otal# passing screeningcriteria
Mid - T erminatio
n Rate 
# Quittingor pausing without resuming
T otal# passingscreeningcriteria
6
ISO 26362:2009
• Participation rate: ‘number of panel members who have
provided a usable response divided by the total number of
initial invitations requesting members to participate (p. 3)
• Usable response is one where the respondent has provided
answers to all the questions required by the survey design
• The term “response rate’ cannot be used to describe
respondent cooperation for access panels
7
Necessary information to compute
response metrics
• In order to compute response metrics for online panels we
need to understand how panel members are recruited and
what stages are used to build a panel
• Volunteer-opt-in design
• Probability-based design
8
Generalized volunteer opt-in panel design
• Stage 1: Encounter, discover, or seek out to join
• Stage 2: Provide profile information
• Stage 3: Get and do surveys
9
Volunteer opt-in panels: Stages
Postoaca, 2007
10
Stages for probability-based online panels
• Stage 1: Recruitment from frame
• Stage 2: Welcome and get profiled
• Stage 3: Active membership, ready for surveys, actual
study
11
Common steps in building a probabilitybased panel
1. Recruitment Rate (RECR): the recruitment of potential
panel members

Recruitment rate calculation will depend on the recruitment mode:
face -to-face, telephone, mail
2. Profile Rate (PROR): empanelling recruited persons



This stage counts panel members that answered their profile survey,
generally a questionnaire collecting background information and
welcoming respondents to the panel
The computation of the profile rate (a.k.a., connection rate) will
depend on the data collection mode
Profiled members are considered to be “active members” in the
pool from which study samples can be drawn
12
Probability-based design features
Implications for computing response rates
1. Single recruitment cohort (one-time effort) vs. multiple
recruitment cohorts (on-going recruitment)
2. Within-household selection to recruit one person vs. whole
household recruitment of all eligible persons
3. The data collection mode used for non-internet households
(no access to online surveys at time of recruitment)
13
Methods of dealing with non-Internet
households
14
Probability- based web panels: Recruitment
15
Probability- based web panels: Profile
16
Probability- based web panels: Actual study
Same design for volunteer-opt-in panels
17
Active panel dynamics
18
Stage 1 of probability-based web panels
IC
Recruitment (RECR) =
Rate
IC + (R + NC + O) + e(UH + UO)
Example: P_RECR = .4 x 100% = 40%
Refusal (REFR) =
Rate
R
IC + (R + NC + O) + e(UH + UO)
IC = Initial consent
R = Refusal
UH and UO = Unknown if household or unknown “other”
NC = Non-Contact
O = Other non-interview
e = Estimated proportion of unknown eligibility cases
19
Stage 2 (more likely for probability-based panel)
Profile Rate (PROR) =
(I + P)
(I + P) + (R + NC + O)*
Example: PROR = .6 x 100% = 60%
Refusal to Profile (REFP) =
R
(I + P) + (R + NC + O)*
I = Profile survey complete
P = Profile survey partial but acceptable
* Opt-in panels may not know the denominator components.
20
Stage 3 Specific Study Rates
Completion Rate (COMR) =
(I + P)
(I + P) + (R + NC + O)
Example: COMR = .7 x 100% = 70%
Break-off Rate (BFR) =
Study
Refusal (SREF) =
Rate
BF
(I + P) + BF
R
(I + P) + (R + NC + O)
BF = Break-offs -- when the number of answers is below the definition of
partial interview, it can be considered a break-off.
R = Other than for the break-off rate, R includes break-offs as refusals
21
Cumulative Response Rate
Only for pre-recruited probability-based online panels
A multiplicative function
Cumulative RR (CURR) = P_RECR x PROR x COMR
Example CURR= .4 x .6 x .7 = .168 x 100% = 16.8%
Cumulative RR2 (CURR) = P_RECR x PROR x RETR x COMR
Example CURR2= .4 x .6 x .8 x .7 = .134 x 100% = 13.4%
P_RECR = Person recruitment rate
PROR
= Profile rate
COMR = Completion rate for the single study
RETR = Retention rate
22
Computing CUMRR with 1 cohort
RECR
PROR
The computation of a CUMRR is
straightforward when the panel
is built with a single recruitment cohort
RETR
COMR
Study
Respondents
23
Computing CUMRR with 3 cohorts
Unequal cohort
contributions to a
study sample
selected from
among all active
members
24
Formulas dealing with multiple cohorts (1.)
RECR, PROR, RETR are calculated as the weighted average
of the size contribution of each cohort
Example to calculate RECRtotal
RECRtotal
Wc1RECRc1  Wc 2 RECRc 2  Wc3 RECRc 3  ...Wcn RECRcn

Wc1  Wc 2  Wc 3  ...Wcn
Where Wcn = the number of cases contributed to the sample
from cohort n
25
Example of RECR with 3 cohorts
Cohort 1
Cohort 2
Cohort 3
Size in the
final sample
200
100
50
Recruitment
rate (RECR)
.35
.27
.15
RECR total
200 .35   100 .27   50.15 


200  100  50
RECR total
70  27  7.5 104 .5


 .2985
350
350
26
Formulas dealing with multiple cohorts (2.)
RECRtotal
Wc1RECRc1  Wc 2 RECRc 2  Wc3 RECRc 3  ...Wcn RECRcn

Wc1  Wc 2  Wc 3  ...Wcn
PRORtotal
Wc1PRORc1  Wc 2 PRORc 2  Wc 3 PRORc3  ...Wcn PRORcn

Wc1  Wc 2  Wc3  ...Wcn
RETRtotal
Wc1RETRc1  Wc 2 RETRc 2  Wc3 RETRc 3  ...Wcn RETRcn

Wc1  Wc 2  Wc3  ...Wcn
27
Full example with 3 cohorts
Size
RECR
PROR
RETR
Cohort 1
200
Cohort 2
100
Cohort 3
50
___Rtotal
.35
.57
.50
.27
.65
.67
.15
.70
.85
.299
.611
.599
Assume a survey completion rate (COMR) of .713
CUMRR1total  .299 .611 .713  .130  100%  13.0%
CUMRR2total  .299 .611 .599 .713  .078 100%  7.8%
28
Computing completion rate (COMR) when
multiple data collection modes are used
Completion rates need to be
computed separately for each mode
 Web survey
 Mail, phone or IVR
These rates should also be
combined as a weighted average
29
Technical condition in order to compute
response metrics
• In order to compute response metrics each panel
organization must keep an historical database with rates for
each member
• More specifically for probability-based online panels it is
necessary that:
 Each panel member ever recruited must have a record of his/her:
– Recruitment rate cohort value
– Profile rate cohort value
– Retention rate cohort value
30
Which formula for which panel?
Metric
Recruitment
Refusal to be recruited
Profile
Refusal to profile
Screening
Eligibility
Completion
Break-off
Refusal
Cumulative Response
Probabilitybased
Yes
Volunteer
opt-in
N/A
Yes
Yes
Yes
Yes
N/A
Maybe
Maybe
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
N/A
31
Which formula for which panel? II
Metric
Attrition cross sectional
Attrition longitudinal
Reinterview
Pre-recruited
Volunteer
Yes
Yes
Yes
Yes
Yes
Yes
32
Dutch study (Vonk, van Ossenbruggen, & Willems, Esomar 2006)
Panel Management or Manipulation?
33
Some factors affecting each rate
Recruitment rate
 Recruitment methods
 Incentives
Profile rate
 Incentives
 Panel management efforts
Retention rate
 Time elapsed since recruitment
 Incentives
 Panel management efforts
Survey completion rate
 Field time
 Incentives
 Reminders
34
References
• Callegaro, M. and DiSogra, C. (2008). Computing response
metrics for online panels. Public Opinion Quarterly, 72, pp.
1008-1032.
• DiSogra, C. and Callegaro, M. (forthcoming). Computing
response rates for probability based web panels. In
American Statistical Association (Ed.). Proceedings of the
joint statistical meetings: section on survey research
methods [Cd-Rom]. Alexandria, VA: American Statistical
Association.
35
Future work
• Recruitment level computed at a household or at a person
level (when recruiting multiple members per household)
• Attrition rates for cross sectional design
• Attrition rates for longitudinal designs
• Response rates for longitudinal designs
36