Transcript Data Collection Considerations

Asking the Right Questions:
Developing Effective Surveys
An evaluation capacity-building training
from the Tobacco Control Evaluation Center
by Robin Kipke & Travis Satterlund
June 10, 2011
1
What We’ll Be Covering
10:00 Introductions, take knowledge pre-test
10:40 End-use strategizing
11:35 Question types
11:50 Dillman principles for writing questions
12:15 Lunch break
1:15 Dillman continued
1:35 Writing Questions -- Practice
2:25 Survey sequencing & construction
3:00 Field testing surveys
3:10 Learning recap, setting action plans
3:20 Complete exit survey
3:30 Optional consultation with TCEC associates
2
Training Objectives
Participants can explain and apply these concepts:
o A survey is like a conversation
o The 1st step of developing a survey is end-use
strategizing
o The aim of survey design is to reduce nonresponse and measurement error
3
A survey is like a conversation in
that…
Photo by Robert Thivierge
o
o
o
o
o
o
o
o
Is communication with a purpose
Begins with an introduction
Needs to capture respondent’s interest
Starts with easy-to-answer ?s, builds to more substantial
Follows logical order, uses transitions to change topic
Finds a delicate way to raise sensitive issues
Winds down with less consequential subject
Indicates end with a sign off
4
Where Survey Design Fits In
Source: Youth Media Evaluation Toolkit 2005
5
What Surveys Can Tell You
To inform your project about
o Need for education or outreach
o Community priorities or policy options
o Level of support or opposition among stakeholder
groups
o Makeup of local populations
o Extent of any change effected
o Satisfaction with services
6
Thinking about What Information to
Collect
7
Using Reverse Logic to Develop
Questions
Apply process to evaluation planning and development of
data collection instruments
o Purpose of data
o Target audience for the data
o How information to be used
o Likely critics
o Credibility threshold
o Pieces of data
o Data sources and formats
8
How It Works
o An example
o Try it out – pair activity
o Think of the project you work on and how a
survey could inform your efforts
o Discuss with your team what you might want
a survey to tell you, how it could be used
o Work together to fill in each of the boxes of
the End-use Strategizing worksheet
9
Questions –Information Types
o Attitudes—What one wants or prefers
o Beliefs—What one thinks to be true
o Behavior—What one does or has done
o Attributes—What one is
10
Questions—Structure Types
o Open-ended—No answer choices are
offered
o Closed-ended—Answer choices are
offered
11
Open-Ended Questions
Examples:
o Short Answer
o How long have you lived in this apartment?
o Clarification (as part of skip pattern)
o If you answered “yes” to the previous
question, please explain why...
o Comments
o Please write any additional comments you
may have about the potential smoke-free
policy.
12
Open-Ended Questions
Drawbacks:
o Respondents may find it difficult to express
their feelings
o They take more time
o Can yield inadequate answers without
probing, follow-up questions
o Analysis is time consuming and difficult
13
Closed-Ended Questions
Examples:
o Yes/No questions
o Have you used any tobacco products in the last
30 days?
 Yes  No
o Multiple Choice
o How many bedrooms does this apartment have?
 Zero (Studio)  One  Two  Three
 Other _________________
14
Closed-Ended Questions
o Likert-type Scales
o To what degree would you favor or oppose a
policy to make at least half of the individual units
in this apartment complex non-smoking?
Strongly
Favor
Somewhat
Favor
Neither
Favor nor
Oppose
Somewhat
Oppose
Strongly
Oppose
1
2
3
4
5
o How often do you attend coalition meetings?
All of the
time
1
Most of the Some of the
time
time
2
3
Rarely
Never
4
5
15
Closed-Ended Questions
Examples (continued):
o Ranking
o On a scale of 1 to 5, rank the issues that matter most to
you with 1 being most important.
___ Health care
___ Environmental protection
___ Safe neighborhoods
___ Quality of education
___ Fair wages
16
“Survey design is all about motivating people
adequately so they complete the cognitive
steps necessary for answering questions
accurately and return the questionnaire.”
~ Don Dillman
17
The Dillman Principles
o The aim of survey design is to minimize
measurement and non-response error
o Measurement error: poor question wording or
formatting leads to inaccurate answers
o Non-response error: people who respond to the
survey are different from those who did not
Don A. Dillman. 2007. Mail and Internet Surveys: The
Tailored Design Method, 2nd ed. Hoboken, NJ: John Wiley
& Sons.
18
Creating Respondent Buy-in
o Motivate people to begin and complete survey
o Use introduction/cover letter
o Give compelling reason to participate
o Ask interesting first question
o Build trust by demonstrating competence
o Employ good survey design
o Make it easy to understand what to do
19
Example Introductions
1.
The Bonanza County Public Health Department is
interested in finding out about how tenants feel about
being around tobacco smoke. There are no right or
wrong answers and they will be kept anonymous. We
hope you will share your opinions with us.
2.
To help protect Bonanza County multi-unit housing
(MUH) residents from the dangers of secondhand
smoke, the county Tobacco Prevention Program will
be working with MUH owners and managers to adopt
and implement a voluntary policy that prohibits
smoking in…
3.
The Bonanza County Tobacco Education Coalition is
seeking public opinion on the effects of exposure to
secondhand smoke and possible smoking policies.
Your opinions are very important to us.
4. Smoke-Free Apartment Complexes – Tenant Survey
20
Managing Cognitive Energy
o Respondents will devote only a limited amount
of mental effort to complete your survey
o Make survey easy to understand,
navigate and respond to
o Cluster related topics
o Cluster similar response types
o Weigh need for survey length vs.
complexity of questions
21
Keep Population of Interest in Mind
o What cultural characteristics might affect their
ability to understand the survey?
o What is their literacy level?
o What language do they feel most comfortable with?
o How much time will they be willing to spend on
taking the survey?
o Is this a topic that interests them?
22
Dillman #1: Keep It Simple
o Choose simple rather than specialized words
o Use as few words as possible
o (Sometimes these two rules conflict)
Use
Rather Than
Customers
Patrons
Smoke from other people’s
cigarettes
Secondhand smoke
Apartments or condominiums
Multi-unit housing
People who live here
Occupants of this household
23
Dillman #2: Say What You Mean
o Use complete sentences to ask questions
(even when they seem self-evidently clear)
Problematic:
 Male
 Female
1. What is your gender?  Male
 Female
1. Please check one:
2. Age: ____
Revised:
2. How old are you? _____ years
24
Dillman #3: It’s All about Timing
o Avoid vague qualifiers
o Instead use more specific frames
o How often do you dine out at a restaurant?
Problematic:
Revised:
 Never
 Not at all
 Rarely
 A few times
 Occasionally
 1-2 times a month
 Regularly
 3 or more times a month
25
Dillman #4: Balancing Act
o To avoid bias, state both sides of the attitudinal
scale in the question stem
o Would you favor or oppose a city policy to make all
parks non-smoking?
o Balance scales with equal number of positive
and negative choices
Problematic:
o Very likely, somewhat likely, not very likely
o Neutral position is different from undecided
o Neither supportive nor unsupportive vs. don’t know
26
Balancing Act Example
Good Example:






To what extent do you agree or disagree with the
statement “Smokers have a right to smoke”?
Strongly agree
Somewhat agree
Neither agree nor disagree
Somewhat disagree
Strongly disagree
I don’t know
27
Dillman #5: Pick Me!
o Beware of primacy effects in “mark all that
apply” lists
o Lists are OK for factual questions but not
attitudes or preferences
o Make each issue a yes/no or scalar question
Ok
for
list
1.
Which type of organization do you represent in the
coalition? (mark all that apply)
 Local lead agency
 Law enforcement
 Other ____________
 Service organization
 Educational institution
 I don’t represent an
organization
28
Problematic:
Which of the following areas in the apartment complex do you think
should be made non-smoking? (Mark all that apply)
 courtyard
 pool area
 barbeque areas
 balconies/patios
 mailboxes
 laundry rooms
Revised:
Would you like any of the following areas of your apartment
complex to be made non-smoking?
Courtyard
 yes
 no
Pool area
 yes
 no
Barbeque areas
 yes
 no
Balconies/patios
 yes
 no
Mailboxes
 yes
 no
Laundry rooms
 yes
 no
29
Dillman #6: Framing the Issue
o Use cognitive techniques to improve recall
o Guided imagery
o Layer questions
Problematic:
When you have watched movies where actors
were smoking, did it make smoking seem more
appealing to you?
30
Framing the Recall
Revised:
Think about the last three movies you saw.
1. What type of movies were they? (mark all that apply)
 Action  Comedy
 Drama  Documentary
2.In any of those movies, did any of the characters smoke?
 Yes
 No (go to question 4)
3. Did seeing the characters smoking make cigarettes seem
more appealing to you?
 Yes
31
 No
Framing Sensitive Issues
o Save more sensitive questions towards the end
o Once you’ve established a rapport
o In case respondents refuse to proceed
o Soften the impact of potentially objectionable
questions
o Preface personal questions with more general
ones about the issue or ask about other people
32
Problematic:
1. Have you ever shoplifted any tobacco products from a store?
Revised Versions:
The questions which follow are being asked to help us
understand where young people get tobacco products when
they are underage. We really appreciate your help and that
of students all over the state who have been asked to complete
this survey honestly.
V1. Have you ever taken any tobacco products from a store
without paying for them?
V2. Have you ever obtained tobacco products from…
a friend or relative?
 yes  no
a store without paying?
 yes  no
33
Dillman #7: Six of One, Half Dozen of
the Other
o Make sure answer choices are mutually
exclusive (numbers, conceptual overlaps)
Problematic:
How old are you?
 15-18 yrs. old
 30-50 yrs. old
 18-30 yrs. old
 50+ yrs. old
When you tried to quit, where did you turn to for help?
 my doctor
 support group
 online service
 quitline
 clinic or hospital
 on my own
34
Dillman #8: It Doesn’t Add Up
o Ensure that respondents can answer the
question
o Is it beyond the knowledge of respondents?
o Were you aware that the California Air Resources
Board has declared secondhand smoke to be a toxic
air contaminant?
o If the city were to pass a tobacco retail ordinance,
how much should a license cost?
o Avoid excessive specificity
o How many cigarettes have you smoked within the
last 30 days?
35
It Doesn’t Add Up continued
o Use timeframes within memory
In the last year, how many tenants complained
about drifting tobacco smoke?
o Avoid unnecessary calculations
How long have you lived
in this apartment?
______ months
36
Dillman #9: Don’t Be Negative!
o Avoid asking respondents to say “yes” in order
to mean “no”
Problematic:
In which areas of the apartment complex should
people not be able to smoke?
Revisions:
v1 Which areas of the apartment complex would
you like to be non-smoking?
v2 In which areas of the apartment complex
should smoking be prohibited?
37
Dillman #10: Over a Barrel
o Avoid double-barreled questions where
two things are being asked in the same
question
o Watch out for “and”
o Instead collapse into illustrative category,
split into two questions, use “or”
38
Over a Barrel?
Is either of these double-barreled?
o If the downtown area was free of secondhand
smoke and cigarette litter, do you think you
would be more or less likely to shop and attend
events there?
o Do you think retailers should have to pay for a
license to sell tobacco which would earmark a
portion of the funds to cover the expense of
enforcement?
39
Now You Try it
o Write one question using each of these formats:
o
o
o
o
Yes/no
Multiple choice
Scalar
Open-ended
40
Survey Design Considerations
o Create a clear navigational path
o Provide signposts to guide respondents
o Use graphic elements as clues
START ❶ Circle the response
o Keep format visually uncluttered
41
Survey Design Considerations
o Cluster like topics (e.g., knowledge,
preferences, support)
o Also cluster similar response formats
(Likert scales, statements of agreement, etc.)
o Use consistent scale directions throughout
o Organize choices vertically, not horizontally
o Beware of response set effect
o Use pamphlet layout
42
43
Sequencing Questions
o
o
o
o
o
o
o
Remember a survey is like a conversation
First engage interest, build trust, ensure success
Move from easy-to-answer to complex
General to more specific
Less personal to more sensitive
Balance open-ended vs. closed-ended
Save demographic questions for the end
44
Field Testing Your Survey
o
o
o
o
o
Why it’s important
Who to involve
How to go about it
What to look for
What to do with the feedback
45
TCEC: Your Resource Center
o Recorded webinars & training modules:
o
o
o
o
o
o
o
o
o
o
Online Surveys: Techniques & Tips (12/2/10)
Reducing Error: Designing Surveys that Work (9/30/10)
Journey of a Survey (1/28/10)
Developing a Survey Instrument (3/26/09)
End-use Strategizing for Creating DCIs (12/10/08)
Public Opinion Surveys (mini training)
End Use Strategizing Checklist
Tips & Tools #2 on writing questions
Hundreds of survey instruments in repository
Individualized help from Evaluation Specialists
46
Reflecting on Learning
o
o
o
o
o
o
o
o
Surveys are more than a brainstormed list of ???s
Start with end-use strategizing process
A survey is like a conversation
Follow design principles to reduce error
Need to motivate respondents
Manage finite amount of cognitive energy
Create navigational path
Make it easy to complete!
47
To reach us:
Email: [email protected]
http://programeval.ucdavis.edu
Main phone line: 530.752.9951
48