Transcript Effectiveness of Monetary Incentives and Other Stimuli Across Establishment Survey Populations ICES III 2007 Montreal, Quebec Canada 4.30.07 Danna Moore and Mike Ollinger.

Effectiveness of Monetary Incentives and Other
Stimuli Across Establishment Survey
Populations
ICES III 2007
Montreal, Quebec Canada
4.30.07
Danna Moore and Mike Ollinger
Objective
(1) Why is this study important?
(2) Gaps in Establishment Survey Literature
(3) Provide suggestions for implementing
establishment surveys
Establishment Respondents:
What type of respondent are they?
Large business or Org
Respondent
Multiple locations
Gate keepers
Record system
Household
Respondent
e.g. small
farmer, small
business
owner
Can vary along the continuum
Why is this important?









Questionnaire variation
Converting
Mandatory reporting
R’s Vary over time
Risk/Difficulty -- cash incentives
Population based sample sizes
Experimental design
Cost/Benefit
Generalize
Stimuli Tested
• Cash Incentives
• Cash plus special postage/package
• Multiple modes
• Mode sequencing
• Mode preferences
• Visual Design— color background vs. none
• Answer boxes stand out
Some Answers Towards the Big Question
Do incentives help or hinder in obtaining
survey responses from businesses?
• Crosses types of establishments and
industries.
• Experiment based
• Population based
• Random assignment
Understanding Why People Participate
Several theories
(Dillman, 1978; Gouldner, 1960; Biner and Kidd, 1994; Groves
et al., 1999).
• Social exchange
Leverage Saliency Theory—Groves et al. (1999)
 Decision to participate is a series of interactive additive
factors.
 Some are survey specific and others are person specific.
Incentives are viewed as an inducement used to compensate
for absence of some factors (i.e., saliency or sense of duty).
Leverage Saliency Theory
Of Survey Participation
1999 POQ vol 64
Features of Establishment Surveys That
Often Lead to Survey Errors
1.
2.
3.
One person selected to 4.Respondent’s characteristics
in relation to the
represent
establishment influences
establishment.
their ability to respond.
Burden increases as
5. intermediary between the
questionnaire and the
they answer as a
characteristics of the record
representative.
system.
There is a respondent
6. Organizational environment
questionnaire
7. Extenuating survey situation
interaction.
Factors Influencing Response
• Businesses differ across industries by size,
structure, and organizational environment.
• Each survey may have a situation or
circumstance that impedes contact.
• These differences and circumstances often
influence how well a survey request can
penetrate an establishment
Gaps
• Lack of monetary incentive studies:
 Few experimental treatments for definitive comparisons
and outcomes.
 Few comparisons of cash versus “cash like” (e.g. checks,
ATM cards) studies—could be especially important for government.
 No empirical demonstrations of effectiveness of
incentives across types of industries and firm size.
 Few comparing effects of various size incentives.
 Not much on effectiveness of survey modes and mode
sequencing
 Response attributable to survey mode sequencing.
2001 HMO Survey Physicians N=1474
80%
100%
80%
**
57%
***
63% 66% 64%
70%
43%
60%
40%
20%
0%
Control T1: FC, FC
T5 Priority $10,FC
T2:Priority No, FC $10 1 T3: Priority $10, FC
T6: Priority $10 2x
T4; Fc, Priority $10
2004 Priority $10 2x
2003 USDA Nationwide Meat Manufacturers N=1,705
Response rates achieved by experimental treatment group
80%
70%
60%
50%
67%
***
***
70%
57%
56%
47%
44%
45%
40%
30%
23%
20%
10%
0%
Pilot-Redm eat Mf.
Pilot--Poultry
Mf.
Control: 2x FC No $
Full-- Redm eat Full Poultry Mf
Mf
Trt 2: 2x priority
Trt 3: 2x Priority $5
2006 Snake River Grain Warehouse and Shipper Survey
Response by Experimental Group (n=424 elevators)
N.S.
80%
70%
72%
76%
63%
60%
***
64%
57%
58%
49%
50%
N.S.
43%
40%
30%
20%
10%
10%
0%
Grain
companies
Port Shippers
Control Trt 1: 2X FC No $
Grain Elevators
Chi SQ 6.7
P < .01
Trt 2: FC $5 , FC No $
Trt 3: 2x FC $5
2007 Evaluation of WA Plastic Pesticide Container
Use/Recycling Control versus Treatment, N=1,986
Chi Sq. 15.86
***P<.001
60%
50%
40%
52%
42%
35%
Chi Sq. 33.5
***P< .0001
22%
30%
20%
10%
0%
Commercial
Applicators/Dealers Licensed
for Pesticides
Control Trt1: No incentive, FC
Licensed Producers
Trt 2: $5 incentive, Priority
2006 Oregon Business Environmental Management Survey
Type and Experimental Treatment Group Initial sample n=1964
60%
50%
***
51%
51%
46%
40%
42%
**
30%
19%
***
25%
*** 47%
42%
40%
**
24%
23%
15%
20%
10%
***
52%
43%
41%
33%
28%
**
18%
8%
3%
0%
Construction:
n=394
Ctrl T1: FC 2x
Manufacturing
n=752
Transportation
n=343
Trt 2: Priority No $ 2x
Accomodation
n=475
Trt 3: Priority $5, FC
All Response
Trt 4: Priority $5 2x
2006 Trade Adjustment Assistance Survey, All Qualifying Industries N=6,429
Exp. Incentive Treatment vs. Control ---Completion Rates
53%
Percentage
60%
50%
***
34%
40%
30%
20%
10%
Sig.
Chi Sq. 51.07
P<.001 ***
0%
1
Control t1: FC 2x No Incentive
Ttrt 2: FC $5 2x
Trade Adjustment Assistance (TAA)
2006 Survey of Program Nationwide – Fisheries N=5,592
70%
60%
***
66%
**
58%
65%
***
62%
48%
41%
59%
54%
**
48%
50%
40%
***
54%
70%
45%
41%
35%
29%
31%
Catfish
WA
AK
TX
LA
AL
FL
GAMS
NC
SC
Salmon Salmon Shrimp Shrimp Shrimp Shrimp Shrimp Shrimp Shrimp Shrimp
30%
20%
10%
0%
Control T1: FC 2x
Trt 2: $5, Priority 2x
Trade Adjustment Assistance (TAA)
2006 Survey of Program Nationwide – Commodities
N=837
NS
88%
100%
80%
73%
NS
NS
77%
74%
59%
76%
59%
62%
60%
40%
20%
0%
Grapes
CA Olives
ID-Potatoes Blueberries
Control T1: FC 2x
Trt 2: $5, Priority 2x
Lychees
TAA Survey—Percentage of Completes Associated with
Incentive Experimental Assignment and Survey Mode
Telephone Completes
Web
Completes
1%
2%
13%
12%
3%
Mail Completes
69%
Mail--No incentive
Mail--$5 priority
Web -- No incentive, FC
Web--$5 priority
Tel.-No incentive, FC
Tel-$5 priority
Suggestions for
implementing establishment surveys
Effective Practice:
• Contact respondents multiple times
• Contact respondents in multiple modes.
• Allow for survey mode preference.
• Design surveys that reduce burden
• short, conditional branching, ease
• Use leverages
Visual Design Effect – FARW Commercial Dealers N=1600
Background Shading w/ Visible Answer Boxes
Does it make a difference?
Is there an interaction effect w/ Incentives?
***
56%
***
60%
48%
50%
36%
34%
40%
Bkg. Color
No $5
Bkg. Color $5
30%
No Bkg Color
No $5
No Bkg Color
$5
20%
10%
0%
1
Visual Design Test
FARW Growers With Pesticide License
***
***
45%
44%
**
39%
Bkg. Color No $5
40%
35%
30%
***
Bkg. Color $5
24%
21%
25%
20%
15%
10%
No Bkg Color No
$5
5%
No Bkg Color $5
0%
1
• All 3 survey modes generated completes
• Exper. treatment (Cash incentive &
priority mail) stimulated more responses
in all 3 survey modes—large interaction
effect.
• Offering web as an alternative option
garnered 15%
• Telephone last 3% – still effective
Influential Circumstances
Saliency: Topic interest
 area of business emphasis for entity.
 High personal interest for respondent.
 High level of public or political concern
Role of survey sponsor.
 Regulator or source of certification.
 Mandatory reporting.
 Source of program $$ or sponsorship
Response Burden
 Complexity, length, multiple reports
What was learned
from experimental trials
•
Token cash incentives were effective across types of establishment and
industry populations.
•
•
•
•
•
2-day Priority mail was more effective than first class mail.
•
Establishment population characteristics and the selected respondent
characteristics need to be considered jointly in explaining response.
•
The survey circumstances and situation impact establishment response.
Priority mail alone just slightly better than First class
Cash incentives combined with priority mail was synergistic.
Mixed mode strategies are very helpful and work.
Respondents may have mode preferences.