Transcript Slide 1

Responsive Collection Design (RCD)
for CATI Surveys and
Total Survey Error (TSE)
François Laflamme
International Total Survey Error Workshop (ITSEW)
Quebec, June 2011
Statistics Canada • Statistique Canada
18/07/2015
Outline
Introduction
RCD surveys
RCD strategy
Active management for RCD
 Monitoring and decision making tools
Highlights and lessons learned
RCD and TSE
Next Steps
2
Statistics Canada • Statistique Canada
18/07/2015
Introduction
Responsive Collection Design (RCD) is an
adaptive approach that uses the information
available prior and during data collection to adjust
collection strategy for the remaining in-progress
cases
Trade-off between quality, cost, productivity,
responding potential of in-progress cases, survey
mode and interaction between surveys
3
Statistics Canada • Statistique Canada
18/07/2015
RCD Surveys
Two experiment surveys with RD and control groups
 Households and the Environment Survey (HES 2009)
● Dwelling survey with a cross-sectional design
● Canadian Community Health Survey (CCHS 2009) sampling
frame
 Survey of Labour and Income Dynamics (SLID 2010)
● Longitudinal survey (Complex survey design)
One full RCD survey - SLID 2011
 Embedded experiment for the first call
 Objectives: proof of concept, improve efficiency
and quality
4
Statistics Canada • Statistique Canada
18/07/2015
RCD Strategy
1) Planning phase
 Analysis of previous data collection cycle
 Data collection phases and strategies
● RCD objectives, staffing plans and response
propensity model for each survey
 Sample validation
 Active management tools and reports
● New key indicators and communication plan
 Control group to assess RCD impact
5
Statistics Canada • Statistique Canada
18/07/2015
RCD Strategy (cont’d)
2) Initial data collection phase
 Use strategic improvement opportunities
previously identified
● New time slice strategy and intermediate cap
 More likely to collect easy cases
 Monitor key indicators to identify start of RCD
Phase 1
● Response rate, productivity, cost (proportion of budget
spent) and responding potential of in-progress cases
● By Regional Office (RO)
6
Statistics Canada • Statistique Canada
18/07/2015
RCD Strategy (cont’d)
3) RCD Phase 1 - Daily overnight job
 Categorize and prioritize cases to improve overall
response rates
● Probability of completion (propensity) - logistic regression
model (sampling frame and sequence of calls information)
 Monitor key indicators to identify start of RCD Phase 2
● Representativity indicator and previous key indicators
● By Regional Office (RO)
4) RCD Phase 2 - Daily overnight job
 Prioritize cases to improve sample representativity
● Priority to domains of interest with lower response rates
7
Statistics Canada • Statistique Canada
18/07/2015
RCD Strategy for HES and SLID
Planning
Phase
Initial Collection
Phase
Control
group
Responsive Collection
Design Phase 1
Responsive Collection
Design Phase 2
Control
Priority
groups
nHES = 10,000 units
nSLID = 16,805 units
Sample
nHES = 20,000 units
nSLID = 33,306 units
No contact
RD group
Other
cases
High probability
nHES = 10,000 units
nSLID = 16,801 units
Miscellaneous
Intermediate Cap on calls
Special
group
Denotes a reassessment of the sample, after which cases will be assigned to a new group.
1) For SLID 2010, a group called “High probability-Tracing” was used during RD phase 1
2) For SLID 2011, another group called “High probability-Refusal” was added during RD phase 1
3) The intermediate and global caps on calls were (20, 25) for HES and (30, 40) for SLID
8
Statistics Canada • Statistique Canada
18/07/2015
Active Management for RCD
 Set of plans and tools to manage data collection while
in progress
 By Regional Office (RO)
 Paradata and data sources used
 Blaise Transaction file (BTH) (i.e. calls and contact information),
interviewer payroll hours, budget and target figures, previous
and current collection cycle information, response propensity
model results
 Key indicators
 Used to identify when to start RCD Phase 1 and Phase 2
 No survey estimates monitoring so far
9
Statistics Canada • Statistique Canada
18/07/2015
When to initiate RCD Phases?
 Decision based on survey progress in terms of response rate,
productivity, proportion of budget spent (cost) and responding
potential of in-progress sample
Indicators to Identify Start of Responsive Collection Design Phase 1,
Sturgeon Falls, HES 2009
Response rate
Proportion of budgeted system time
Proportion of regular in-progress cases
Daily productivity
100%
Average productivity over the last 5 days
Proportion of budgeted payroll hours
Average number of calls made on 'regular' cases / cap on calls
80%
60%
40%
20%
0%
1
3
5
7
9
11
13
15
17
19
21
23
25
27
29
31
33
35
37
39
41
43
45
47
49
Data Collection Days
10
Statistics Canada • Statistique Canada
18/07/2015
RCD Dashboard -
Example for RCD Phase 1
 Dashboards are used to identify when to start both RCD phases
to facilitate interpretation and objective decision-making

RCD phase 1: 6 conditions, RCD phase 2: 7 conditions
 Yellow and red lights signal when many conditions are met
Response Rate
RO
Productivity
(Average over last 5
Response Cond
Cond
Initial Current
Rate
1
2
Budget and Cost
% of
Budget
payroll
hours
Cond
3
% of
Budget
System
Time
Propensity of In-Progress Cases
Avg. # of calls
% of regular
Cond
Cond for regular in- Cond Sum of
In-progress
4
5
6
conditions
progress
cases
cases / cap
EDM
52.2%
1
64.3% 41.6%
1
56.6%
1
62.4%
1
28.3%
0
7.2
0
HFX
52.6%
1
63.6% 45.5%
1
62.8%
1
69.8%
1
27.0%
0
7.7
0
SHER 46.3%
0
66.9% 58.3%
1
49.8%
0
50.7%
0
34.5%
0
3.4
0
STURG 64.2%
1
74.4% 46.6%
1
67.2%
1
65.6%
1
13.9%
1
11.3
1
TOR
52.6%
1
65.7% 53.6%
1
58.0%
1
55.4%
1
27.9%
0
6.8
0
WIN
62.4%
1
69.9% 46.8%
1
69.8%
1
69.0%
1
12.5%
1
13.0
1
11
Statistics Canada • Statistique Canada
4
4
1
6
4
6
18/07/2015
When to initiate RCD Phase 2?
 Similar dashboard than RCD Phase 1 is used (7 conditions)
 Condition 7: average response rate increase over the last 5 days
 Representativity indicator is also used
 Representativity indicator is a measure of variability of response rates
between domains of interest
 National versus regional objectives
Representativity Indicator, National and some Regional Offices, HES 2009
1.000
0.990
0.980
0.970
0.960
0.950
0.940
0.930
National
Sherbrooke
Sturgeon Falls
Edmonton
0.920
0.910
0.900
1
3
5
7
9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53
Data Collection Day
12
Statistics Canada • Statistique Canada
18/07/2015
Other Active Management Tools for
RCD - Examples
 Interviewing progress, results and effort
 Response and resolved rates, tracing and refusal conversion results
and effort, refusal at the first contact, refusal and no contact rates
 In-progress cases
 Distribution of calls (for cap on calls monitoring), distribution of cases by
Blaise group, browser use, intensive tracing, cases with high
propensity but few calls
 System time (effort)
 By period of day, phase and group (RCD, CG)
 Expected distribution by Blaise group between phases
 Useful for staff planning
 Several other ad hoc tools
 Used to identify problems or emerging issues
13
Statistics Canada • Statistique Canada
18/07/2015
Active Management Challenges
Large amount of information and reports available
 In the past not enough info, currently too much
 Need to concentrate on major issues (not on good to know info)
● Can spend a lot of time on something not broken
● Some analysis can wait at the end of collection
Analysis and communication
 Often require an extra analytical step before the information
is communicated
● Reports are not enough
● Real challenge is to analyse, summarize and communicate
● In the case of RCD, only the main reports were distributed
● Other reports were only used when required
14
Statistics Canada • Statistique Canada
18/07/2015
Highlights and Lessons Learned
 Higher overall response rate when RCD is used
 Compared to previous survey cycle
 RD group achieved same response rate with less effort (~2%)
 Sample representativity generally improved
 High probability and priority groups had positive impact
 SLID 2011
 Better response rate and contact rate (over 2%) for cases for which the
first call was forced to be in the same time slice of previous interview
 Pilot test demonstrated the technical feasibility of RCD
 Active Management and communication
 Essential for any RCD
 Need timely and accessible paradata
 RCD is not a “magic” solution
 Need to be used in conjunction with other initiatives
15
Statistics Canada • Statistique Canada
18/07/2015
RCD and TSE
 RCD objectives in terms of TSE
 Reduce variance and non-response bias
 Currently, more likely the variance
 But might depend on the non-response adjustment strategy
 Did RCD introduce a potential non-response bias?
 Active Monitoring of the non-response bias requires survey
estimates during data collection - but can be done after
 Did RCD change the responding propensity of cases?
 Soft treatments - More meaningful grouping of cases
● No incentives, no sub-sampling of respondents
16
Statistics Canada • Statistique Canada
18/07/2015
Next Steps
 Implement RCD for other CATI surveys
 Developed with a research rather than a production perspective
 Improve current RCD strategy
 Propensity models
 Gradually phase-in of RCD phase1
 Integrate new conditions for decision making
● Representativity, survey estimates, non-response bias
 Include cost-efficiency objective
 RCD for CAPI surveys
 Potential benefits of RCD to improve cost-efficiency
 Feasible study : communication flow, potential actions, concurrent
surveys, operational and technical constraints.
 RCD for multi-mode surveys
 RCD theoretical framework
17
Statistics Canada • Statistique Canada
18/07/2015
For more information, please contact
Pour plus d’information, veuillez contacter
François Laflamme
[email protected]
18
Statistics Canada • Statistique Canada
18/07/2015