Transcript Web Usage in a Business Panel Survey ICES-III, Montreal Canada June 21, 2007 David Marker and Janice Machado, Westat (U.S) [email protected].

Web Usage in a Business Panel Survey
ICES-III, Montreal Canada
June 21, 2007
David Marker and Janice Machado,
Westat (U.S)
[email protected]
Overview
 Terrorism Risk Insurance Program (TRIP) Surveys were
conducted by Westat for the U.S. Department of the
Treasury
 Data needed to report to the US Congress on:
 The effectiveness of TRIP
 Capacity to offer terrorism coverage after the TRIP
sunsets, availability, & affordability
 Terrorism risk insurance premiums
11/6/2015
2
Overview (contd.)
 Collected panel survey data on property, casualty, and
workers’ compensation insurance from national samples
of 3 types of organizations
 Re-insurers (suppliers of insurance to insurers)
 Insurers (suppliers of insurance)
 50 page questionnaire
 1 month calendar time
 Insureds/policy holders (purchasers of insurance)
 25 page questionnaire
 1 to 4 hours to complete
11/6/2015
3
Overview (contd.)
 Surveys conducted in 3 waves
 Nov’03-Feb ’04
 Oct-Dec ’04
 Feb-Mar ’05
 Several versions of the instrument for Waves 2 and 3
 Data collected via multiple modes:
 Web
 Mail
 Facsimile
11/6/2015
4
Topics





On-line vs. back-end logic checks
Handling multiple respondents per instrument
Relative use of hard copy vs. web
Getting complete information from partial completes
Analyzing data quality issues during data collection
11/6/2015
5
On-line versus Back-End Checks
 Hard copy and web surveys were returned
 Hard copy returned surveys were entered into the web by
Westat staff
 Ensured that both web and hard copy surveys went
through the same edit checks
 However, the Westat staffer, not being the respondent,
was not able to change a response that failed an online edit
11/6/2015
6
On-line versus Back-End Checks (contd.)
 If we could think of an edit prior to the start of data
collection, we added it to the on-line web program.
 If we missed an edit, we added it during post-data
collection (back-end) processing
 We added to on-line edits in subsequent waves based on
our experience in earlier waves
11/6/2015
7
On-line versus Back-End Checks (contd.)
 Mostly soft edits were programmed for items
 Respondents were prompted when they entered a
response that failed an edit
 If the response entered on the 2nd attempt failed a soft
edit, the web program simply accepted the respondent’s
2nd entry provided it fell within the hard edit
 Hard edits required allocated revenue across regions to
add to 100%, or it was not accepted
11/6/2015
8
Types of On-line Edits
 Edits included:
 Range edits which specified an upper and lower value
for each item
 Skip edits that jumped the respondent to the
appropriate next question based on the response
provided to an earlier question
 Logic edits that prompted the respondent when a
subsequent response contradicted an earlier response
11/6/2015
9
Final On-Line Edit
 Edits prior to permitting the respondent to exit the survey
 The program checked to see if there were valid, nonmissing responses to all “critical” questions in each
survey
 If missing or invalid, the questions and responses
were displayed and one final attempt was made to
obtain or correct the information recorded
11/6/2015
10
Back-End Edits
 These were mainly limited to logic edits found to be
necessary post-data collection
 The edits reviewed responses provided to dependant
questions
 If the responses were inconsistent, one or more
responses were set to missing or the organization was
contacted and a new value for the item was obtained
11/6/2015
11
Handling Multiple Respondents
 Questionnaires asked for organizational, financial, and
insurance data
 Often one person in the organization could not respond
to all topics
 Allowed multiple respondents to complete
 Recorded names and contact information
 Also recorded one person who could respond to any
follow-up questions
11/6/2015
12
Handling Multiple Respondents (contd.)
 Provided the Web User ID and Password to just one
senior contact in each organization
 Responsible for dissemination (even multiple
locations)
 Ensuring completion and internal QC
 Program allowed a new respondent to jump to the
section they wanted to complete using pre-programmed
tabs
 Program required some sections/questions to be
completed before others
11/6/2015
13
Web vs. Hardcopy Responses
Insured W2
Insured W3
Insurer W2
Insurer W3
Web
Not Web
%Web
2,051
510
96
72
1,853
452
80
38
53%
53%
55%
65%
Web includes multiple mode
Not web includes mail and fax
11/6/2015
14
Obtaining Missing Information
 Web surveys permitted respondents to indicate they did
not know or did not want to provide a response to any
question
 Client identified a list of questions that needed a
response for the survey to be considered a complete
 E-mail and phone communications used to obtain
missing information
11/6/2015
15
Obtaining Missing Information (contd.)
 Respondents were allowed to re-access their completed
survey and enter missing information
 Westat analysts also updated the web survey with
missing information retrieved activating any on-line web
edits
 Back-end edits were then run on the data
 Increased the number of completed surveys
11/6/2015
16
Real-Time Data Quality




Review item nonresponse from early cases
Identify very problematic questions
Reminder email to answer all subparts
Reduced item nonresponse and need for data retrieval
and imputation
11/6/2015
17
Item Response Rates
B2. Please indicate if the statements below describe the
organization. Is the organization selected a …
a. Subsidiary of another U.S. firm?
b. Subsidiary of a foreign-owned firm?
c. Headquarters of your organization
in the U.S.
d. Something else?
(Specify) ___________
YES
NO
1
1
2
2
1
2
1
2
11/6/2015
18
Effect of Email Clarification
Sent to 1/4th of sample for whom we had email addresses
“When answering question B2, please answer 1 or 2 for
EACH of the four subparts of this question.”
Table 1. Percent of respondents who needed coders to clean up question B2.
Percent of responses requiring coder cleaning
Before email (n=2,269)
After e-mail (n= 986)
B2a
10.4%
6.6%
B2b
11.7%
6.3%
B2c
2.9%
1.0%
B2d
34.7%
23.9%
11/6/2015
19
Updating Skip Patterns
 Respondents edited earlier responses
 Need to update skip patterns real time to identify newlyrequired items
11/6/2015
20
Prefill Items with Information
from Prior Waves
 Each survey was conducted in 3 waves
 Surveys collected extensive numeric data (financial,
insurance, etc.)
 Respondents changed between waves
 To reduce errors in how values were reported between
waves and by different respondents subsequent waves of
the survey showed responses that were provided in
earlier waves
11/6/2015
21
Prefill Items with Information
from Prior Waves (contd.)
 Prefills done only on select items where there was an
increased probability of error. For example, one
respondent providing numbers in ‘000s, and another in
millions
 Showing a previous response helped improve the quality
of data collected on the subsequent round
 Also allowed respondents to fill in earlier blank cells in
the matrix
11/6/2015
22
Conclusions




Web provides opportunities and dangers
Ease burden on respondents, especially in large org.
Allows for real-time data quality improvements
Lose control on data after they are initially received from
respondent
11/6/2015
23