Transcript Multi-Mode Data Collection: Why, When, How International Conference on Establishment Surveys Montreal June 18-21, 2007 Richard Rosen US.

Multi-Mode Data Collection:
Why, When, How
International Conference on Establishment
Surveys
Montreal
June 18-21, 2007
Richard Rosen
US. Bureau of Labor Statistics
[email protected]
202-691-6524
Any opinions expressed in this paper are those of the authors and do not constitute policy
of the Bureau of Labor Statistics.
Automated Collection in the Current Employment
Statistics Survey
Goals for Paper



To provide insights/experiences based
on CES
To review the collective experience of all
BLS Establishment-based surveys
To draw conclusions about the relative
benefits of multi-mode collection
Current Employment Statistics Program

Monthly survey of employment, payroll, and hours conducted by US
Bureau of Labor Statistics

A Federal-State cooperative system

A sample of 300,000 business establishments

Data are published after only 12 collection days

Limited number of data elements (but greatly increased in 2006)

Multi-mode since 1984
Automated Collection Methods in the CES
In 1984, BLS began to examine alternative collection
methods.
New automated collection modes:
Year
Computer Assisted Telephone Interview (CATI)
Touchtone Data Entry (TDE)
Voice Recognition (VR)
1984
1986
1988
Electronic Data Interchange (EDI)
FAX
Electronic Mail/Internet/WWW
1994
1995
1996
Multi-mode Myths




It will lower my collection cost
It will improve my response rate
Implementation will be easy
It will solve all of my problems
Cost





Depends on what mode is being replaced
Development costs
Initial start-up costs
Economies of scale
Cost structure change over time
Estimated CES Collection Costs
by Mode
Mode:
Unit cost per month:
CATI
$6.43
Mail
$2.10
FAX
$1.60
TDE
$1.50
Web
$0.80
EDI
$0.50
Response Rates

Depends on the mode being replaced




Mail vs any mode = improved response
Electronic Data Interchange = generally
lower initial response; time lag
CATI can achieve high response rates
TDE, FAX, Web, E-mail: Self-response


Lower response than CATI
Higher response than Mail
Easy





Requires R&D
Acquisition of new HW/SW
IT Support
Prototype
Testing
Solve All Problems

Create new set of problems




New protocols
Integration with other collection modes
Integration with other survey operations
Impact on current staffing/workflow
TDE Respondent Contact Program
CES has implemented procedures designed to maximize response
Contact Type:
 Advance Notice
 Nonresponse Prompt
 “Last Chance” prompt
 Secondary NRP
 Long-term NRP
 Refusal Conversion
Contact Method:
FAX or Postcard
FAX or Call
FAX
Call
Call
Letter/Call
Example of Protocol Effect

Effect of Advance Notice and Nonresponse
prompting on TDE Response


Elimination of Advance Notice:
 Reduced response by 10 % points
Elimination of Nonresponse message:
 Reduced response by 10 % points
Rosen and Hertwig, “The Impact of Prompting on Response Rates: Experience with
Touchtone Reporting in the CES Program,” American Statistical Association, August
2002.
Comparison of CES Collection Methodologies
Data
Collection Reporting
Method
Procedures
Touchtone
Data Entry
(TDE)
CATI
FAX
Web
Mail
Electronic
Data
Interchange
(EDI)
Respondent calls toll free
800 number, connects to
computer, enters data
using touchtone phone
Respondent called by
interviewer, data entered
into computer
Respondent FAXes
completed form
Respondent links to
secure web site, enters
data and submits
Respondent mails
completed form
Respondent creates a
file which is transmitted
to BLS electronically
Monthly
Advance
Notice
Non-Response
Prompting
-Phone call (if large)
-FAX (if small with FAX)
Postcard or FAX
-Postcard (if small, no
FAX)
CATI call serves as
Postcard
prompt
Blank Form
E-mail
Blank Form
----
-Re-FAX survey form
-Phone call
E-mail
-Re-mail survey form
-Postcard
-Letter
-Phone call
Personal phone call
What Factors to Consider?

Characteristics of your survey




Periodicity
Survey length/complexity
Sample composition
Characteristics of your respondents



Knowledge/education
Environment (office vs mobile)
Commitment/willingness to report
Distribution of CES Sample by
Collection Mode
March 1993
1%
4%
March 2000
1%
1%
8%
6%
6%
16%
1
6
7%
5%
2
3
4
4
3
86%
59%
5
2
6
Mail
CATI
TDE
Tape/Diskette
5
Other
1
Mail
Tape/Diskette
Web
CATI
FAX
TDE
EDI
Distribution of CES Sample by
Collection Mode
March 2000
Tape/disk 7%
May 2003
Web 1%
Web 1%
Mail 4%
EDI 6%
FAX 7%
Mail 16%
CATI 5%
EDI 23%
CATI 18%
FAX 15%
TDE 33%
TDE 59%
Tape/disk 6%
Comparison of CES Collection Rate, Employment Received,
and Revisions
(Nov. 1988-Jan. 2001)
100
160
90
80
120
100
60
50
80
40
60
30
40
20
20
10
Jan-01
Jul-00
Jan-00
Jul-99
Jan-99
Jul-98
Jan-98
Jul-97
Jan-97
Jul-96
Jan-96
Sep-95
May-95
Nov-94
Jul-94
Jan-94
Sep-93
May-93
Nov-92
Jul-92
Jan-92
Sep-91
May-91
Nov-90
Jul-90
Jan-90
Sep-89
0
May-89
0
Nov-88
Percent
70
Date
Percent Employment Received at First Closing
Absolute Value of Total Private Revisions from 1st-3rd Closing
Percentage of Registry Received at First Closing
Absolute Value of Revisions
(1000)
140
Profile of CES Population and Collection
Methods
Range of
worksites
1-4
5-9
10 - 19
20 - 49
50 - 99
100 - 249
250 - 499
500+
Total
Number of firms Percent of
(EINs)
firms (EINs)
208,831
93.6%
6,365
2.9%
3,826
1.7%
2,326
1.0%
820
0.4%
559
0.3%
192
0.1%
139
0.1%
223,058
100.0
TDE/Web/E-mail
Percent of
worksites
29.4%
5.2%
6.5%
8.8%
7.1%
11.0%
8.2%
23.6%
100.0%
Fax/XLS/Fillable Form
Percent of
Employment
39.2%
9.0%
7.4%
9.5%
7.3%
8.2%
5.2%
14.1%
100.0%
EDI
Total
Mail
TDE
CATI
Fax
Web
Feb-07
Oct-06
Jun-06
Feb-06
Oct-05
Jun-05
Feb-05
Oct-04
Jun-04
Feb-04
Oct-03
100
90
80
70
60
50
40
30
20
10
0
Jun-03
Collection Rate
CES Collection Rates for Preliminary Estimates by Mode
Jun 03 - Mar 07
EDI
CES Collection Rates by Mode, Final Estimates. Jun 2003 - Mar 2007
100.0
90.0
70.0
60.0
50.0
40.0
30.0
20.0
10.0
pr
-0
3
Ju
l-0
3
O
ct
-0
3
Ja
n04
A
pr
-0
4
Ju
l-0
4
O
ct
-0
4
Ja
n05
A
pr
-0
5
Ju
l-0
5
O
ct
-0
5
Ja
n06
A
pr
-0
6
Ju
l-0
6
O
ct
-0
6
Ja
n07
0.0
A
Collection Rate
80.0
Total
Mail
TDE
CATI
FAX
Web
EDI
Web Collection: Some Advice

KISS (keep it simple stupid)


Beware of overbearing security requirements


Digital Certificates, complicated passwords, passwords that
must be updated, pose significant barrier
Keep edits simple



Respondents are not sophisticated computer users
Don’t try to replicate all of your edit checks on-line
Edit failures when data are correct will frustrate respondents
In April 2007, BLS launched a more streamlined
Website for data reporting (Web-lite).
Ja
Fe n0 4
M b04
a
Apr04
M r0
a 4
Juy0 4
n
Ju 04
Au l0
4
S e g0 4
p
O 04
ct
No 04
De v 0
4
Ja c04
Fe n0 5
M b05
a
Apr05
M r0
a 5
Juy0 5
n
Ju 05
Au l05
S e g0 5
p
O 05
ct
No 05
De v 0
5
Ja c05
Fe n0 6
M b06
a
Apr06
M r0
a 6
Juy0 6
n
Ju 06
Au l06
S e g0
p 6
O 06
c
No t 06
De v 06
Ja c06
Fe n0 7
M b07
ar
07
TDE vs Web Response Rates for Preliminary
Estimates, Jan. 2004 - Mar. 2007
100%
90%
80%
Begin Web-lite
April 2006
70%
60%
50%
40%
TDE
Web
E-mail vs Web

Easier for respondent



Eliminates login process
Can’t forget your account number of password
Can be done securely with HTTPS




Can embed HTML form or PDF directly into E-mail
Respondent fills out form and hits a “submit” button
Data sent via Browser to agency server using HTTPS
Several products on market
E-mail Drawbacks



Limited editing capability
With PDF, size of file may be an issue
With HTML,



Number of data items/questions is limited
(single page)
Not all respondents have HTML E-mail
Products available don’t offer “total
solution” so must develop back-end support
TDE vs Web vs E-mail
Collection Rate Comparison
2007
All
February 73%
March
68%
April
69%
TDE
73%
73%
69%
Web
66%
64%
*60%
*CES experienced web server problems during April 2007
E-mail
74%
67%
70%
EDI: Some Comments

Takes time to work with firms



Have a standard file format but be
prepared to take what they have


Most Gov’t surveys are voluntary
Must “get in line” for IT resources
Data item response can be an issue
New directions: XML
Review of BLS Establishment
Surveys

Eight surveys




Four monthly
Two Quarterly
Two Annual
Over the past 10 years, all have
adopted multi-mode collection
Survey Questions








Distribution of modes used
Reason for using multiple modes
Do you target certain populations for specific
modes
Benefits and Drawbacks
Process used to determine new modes
New modes being considered
Key factors when considering new modes
Advice to other survey organizations
Distribution of Modes, BLS Establishment
Surveys, 2007
Mode:
CES
IPP
JOLTS
PPI
MWR
NCS
SOII
OES
M
M
M
M
M
Q
A
A
41%
13%
54%
84%
19%
46%
85%
Mail
8%
CATI
23%
40%
Fax
18%
18%
20%
28%
30%
1%
Touchtone Data Entry
Electronic data
Web
1%
Average
44%
8%
45%
56%
13%
2%
1%
12%
6%
1%
15%
26%
1%
Telephone
4%
2%
46%
28%
3%
10%
13%
11%
5%
Personal Visit
5%
1%
Combination
10%
1%
Other
3%
M=Monthly
3%
Q=Quarterly
A=Annual
1%
Reasons Using Multi-mode
Reason for using multi-mode:
CES
JOLTS
PPI
IPP
1
1
NCS
Respondent preference
3
Reduce cost
2
6
Improve response rates
1
1
1
Better Data Quality
5
Quicker Turnaround
3
2,3
MWR
SOII
3
1
OES
Score
3
39
1
18
1
5
2
18
9
4
7
Technology and resources available
2
Problems with Mail delivery
Support GPEA
5
2
5
2
5
Reduce Respondent Burden
2
State Partner Feedback
Ease of Use for Respondent
Improve Productivity
5
2
3
5
4
4
3
Do you Target Modes?


CES: Take into account the characteristics of the firm in terms of size, number of units/reports,
past reporting history in terms of timeliness and ability to self-report.
MWR: Large multi-state employers encourage to use EDI Center; small multi-unit respondents
2-30 worksites offered Web. Also target software developers and outsourcing firms to include
electronic reporting in their systems or services.

NCS: Mode of collection is determined by size of establishment, location of establishment,
reporting capability of the firm, and level of cooperation.

SOII: All modes are available to all respondents. Certain respondents get booklets designed to
encourage internet and other electronic methods or reporting.

IPP: During the visit respondents are told that web is the preferred mode of collection but that
mail/fax is also available if that is their preference. IPP wants the respondent to choose
whatever method they feel most comfortable with. Since September 2005, just over 70% of
IPP respondents have selected web.

PPI: prefers electronic data collection, which currently is limited to Fax. For important
respondents, we allow them to email in spreadsheet with pricing data.

JOLTS: With the exception of our largest respondents, we normally encourage our respondents
to provide data via TDE after a 6 month period of data collection using CATI.

OES: COCs or establishment with high weight; previous response via a particular mode;
availability of email or web site addresses
Drawbacks



CES: Time spent in researching and developing new mode. Need to develop new protocols and
procedures for each mode. Often end up with separate databases for each mode that need to be
managed. Almost like running multiple/separate survey operations.
MWR: Requires sizable amount of staff resources to maintain systems and deal with coordination and
timing issues. Keeping States and State systems and employers in sync can be a challenge.
NCS: It is virtually impossible to isolate and evaluate the effectiveness of any single mode. Have to
maintain instructions and enforce protocols in each area. Additional tracking and maintenance is required
to keep up with each firm’s collection mode. Need to customize update materials and data requests to
match the mode used by each firm. Electronic collection carries risks of confidentiality breaches.

SOII: For some reason electronic collection has a slightly lower response rate than respondents receiving
the standard booklets.

IPP: Additional costs/resources to maintain multiple modes.

PPI: The major drawback for PPI is the BLS limitations on email, not having web repricing and
systems limitations on broadcast fax pricing.

JOLTS: None.

OES: Increased occupational coding burden on State analysts; Internal and external
security/confidentiality issues; Costs.
Process to Determine New Mode


CES: Initial research on mode, possible use/benefit. Proof on concept project. If appears to be
successful, limited production. Then full implementation. Constant evaluation and monitoring.
MWR: Evaluated pros and cons of various collection modes in terms of expected costs, accuracy,
security, periodicity of collection. For example, if survey is only done every 3 years do you want to
invest in new costly collection mode.

NCS: The last mode to be added, electronic collection through email to a secure server, arose due to
demand from respondents and field economists. It was then developed and tested.

SOII: Does the new mode reduce cost and/or help capture additional narrative.



IPP: More respondents started requesting additional electronic alternatives, such as Internet-based
collection. Initially piloted the web survey with a small sample of reporters. Results of the pilot were
very favorable.
PPI: Broadcast fax was a pilot project for several years with limited respondents. When the anthrax
problem disrupted mail pricing, then we started repricing forms by fax and gradually increased our
broadcast fax capabilities
OES: Feedback; Research and IT consultation; Security review; Pilot testing and process refinement
New Modes Under Consideration








CES: E-mail collection either with embedded
HTML form or fillable XLS/PDF
MWR: Fillable forms
NCS: secure, encrypted data files
SOII: None
IPP: Fillable PDF survey forms via email
PPI: Web repricing
JOLTS: Secure E-mail
OES: Fillable forms, Web
Factors to Implement






CES: Ease of use for respondent; respondent
acceptance. Data security.
MWR: Cost, employer acceptance, State acceptance,
employer familiarity with the survey.
NCS: If the test is successful, it is relatively easy to use,
and respondents find it acceptable.
IPP: Respondent demand, costs and other measurable
compared to IPPs current collection alternatives.
JOLTS: Cost of using the central BLS facility for secure
email. Ease of use for respondent.
OES: Feasibility to collect meaningful data; Cost; Security
Study Mode Effects

CES: Mostly response rate effects and data item response. Some review of data quality.

MWR: Web collection has higher response rates, but this may be somewhat biased as we
restricted initial solicitation to good reporters.

NCS: No

SOII: response rates, processing times

IPP: With web repricing the IPP gets substantially faster data turnaround than mail/fax;
Clerical and quality edits on the front-end of the web survey yield better quality and more
usable data than mail/fax; response rates have been consistently greater for web than
mail/fax respondents.

PPI: No

JOLTS: No

OES: Response Analysis Survey will include review of mode effects
Advice







CES: Surveys need to modernize their collection. Carefully consider alternative collection modes, do
the needed development work and evaluation. Don’t expect the new mode to solve all of your
problems. If you can solve just one problem or meet one objective that may be enough.
Constantly evaluate your “mix” as things change, and you have to adapt and add new modes and
de-emphasize others.
MWR: Consider costs, employer familiarity with the survey, timing and coordination issues.
NCS: Evaluate how secure any new mode of collection will be and whether any policy precludes a
specific mode from use. Assess what information is available in more than one mode and whether
respondent burden or collection costs can be reduced, and whether efficiency or quality is
improved by adding a new mode. Need to have technology support and staff training on any new
collection mode employed.
SOII: Do it.
IPP: Know your respondents; get support from upper management; Iterative development and
pilot testing; Usability testing a must; keep stakeholders informed; provide resources for ongoing
project management.
PPI: Be prepared to budget in advance as developing electronic collection methods is costly.
OES: Talk to others to find out their lessons learned; establish and maintain a good working
relationship with IT; lots of research; have a tech close by; Pilot testing; Training
Summary







Most BLS surveys are using multi-mode
Respondent preference/acceptance is top
priority
Cost should NOT be primary goal (secondary or
side benefit)
Improved response is a possible benefit
Improved timeliness is possible/likely
Additional modes add complexity to survey
operations
Continuous evaluation of effectiveness; can’t
stand still