Transcript WIM Fall 2013 Intro to Questionnaire Design

Introduction to
Questionnaire Design
2013-2014 Workshop in Methods
November 8, 2013
Ashley Bowers, Stacey Giroux, and Lilian Yahng
Why Questionnaire Design Matters
• Do you think the sports media treat African American
athletes differently than white athletes?
– YES
– NO
60% of respondents answer “YES, DIFFERENTLY”
• Do the sports media treat African American athletes and
white athletes the same or differently?
– SAME
– DIFFERENTLY
40% of respondents answer “DIFFERENTLY”
Center for Survey Research
November 8, 2013
Cognitive Aspects of Survey Methodology
Question-Answering
Process:
•
•
•
•
Comprehension
Retrieval
Judgment
Reporting:
• Formatting into
response options
• Editing answer
Tourangeau (2000)
Center for Survey Research
Example: Do you think the sports
media treat African American
athletes differently than white
athletes? (Yes/No)
•
•
•
•
•
Captures both the ideal and nonideal
case – e.g., misunderstanding, satisficing
Respondents can go wrong at any of
these stages
Not necessarily an orderly progression;
several can be happening simultaneously
Survey as artificial construct
Survey within a social context
•
•
•
(social exchange theory)
(cooperative/conversational principle)
(social desirability)
November 8, 201
TOP TEN
1.
2.
3.
4.
5.
Center for Survey Research
Conceptual Variability
Problematic Wording
Vague Quantifiers
Response Options
Order Effects
6. Recall/Retrieval
Difficulty
7. Estimation Difficulty
8. Attitudinal Questions
9. Sensitive Topics
10.Formatting
November 8, 2013
#1 - Conceptual Variability
• Words have many meanings:
• variability across respondents
• divergence from researcher’s definition
• Suessbrick, Schober & Conrad (2000) administered CPS Tobacco
Use Supplement followed by post-test about interpretation:
Most frequent interpretation held by only 53.8%!
23%
Only cigarettes you
finished
Cigarettes you finished
or partly smoked
54%
23%
Center for Survey Research
Even just one puff
November 8, 2013
Do you think children suffer any ill effects from
watching programmes with violence in them, other than
ordinary Westerns?
Belson (1981) determined that respondents interpreted children, ill
effects and violence in numerous ways
• e.g. “children”: < 8 years, < 19 - 20 years
• children as students
• only 8% interpreted question as intended
Center for Survey Research
November 8, 2013
The “same” question, with some definitions:
Do you think that children suffer any ill effects from watching TV with
violence in them, other than ordinary Westerns? By children I mean
people under 14, by ill effects I mean increased aggression at school
or at home, increased nightmares, inability to concentrate on routine
chores, and so on. By violence I mean graphic depictions of
individuals inflicting physical injuries on themselves or others,
depictions of individuals wantonly damaging property or possessions,
abusive behaviors or language to others, and so on.
Improved?
Additional words can clarify intended meaning but this may lead to
unwieldy questions, as above.
• Tradeoff between clarity and complexity
• Possible confusion when everyday terms are defined (Gricean
implicatures)
Center for Survey Research
November 8, 2013
#2 - Problematic Wording
Are you satisfied with our prices and customer service?
AVOID:
Do you favor or oppose not allowing drivers to use cell
phones while driving?
•
•
•
•
•
Given the world situation, the government protects too
many documents by classifying them as SECRET or TOP
SECRET. (Agree/Disagree)
How many minutes does it usually take you to commute to
work?
In the past month, have you crossed the street from one
side to another in order to avoid going near someone you
thought was frightening?
Now that you've seen how you can save time, would you
buy our product?
Center for Survey Research
Double-barreling
Negatives
Complex syntax
Hidden assumptions
Leading questions
Aim for a ~6th grade
reading level.
November 8, 2013
#3 - Vague Quantifiers
• Non-numerical terms for quantity have different numerical
interpretations
Belson (1981) found “few” (in over the last few years) meant:
• “no more than two years” (7/59 respondents)
• “seven or more years” (19/59)
• “ten or more” (11/59)
• Particularly problematic in response options
How often do you feel really excited? Very often, pretty often, not
too often or never?
If R says more than never, Schaeffer & Bradburn asked for
number
For educated and younger Rs, “pretty often” and “very often”
were associated with larger numbers
Center for Survey Research
November 8, 2013
#4 - Response Options
Why Response Options Matter
Response Alternatives and TelevisionWatching (Schwartz & Bienias, 1990)
Center for Survey Research
Don’t Forget to Check:
 Balanced scale
 Number of response options: 5 to 7?
(Krosnick et al.)
 Label all scale points
 Mutually exclusive and exhaustive
 Midpoint (e.g., Neither/Neutral) option?
 “Other (Please Specify)” option?
 “Don’t Know” option?
 “Not Applicable” option?
November 8, 2013
#5 - Order Effects
• Ordering of Questions
• Part/whole effects (marital satisfaction/general satisfaction)
– ask general question first
• Related content – Fewer people say taxes are too high
when first asked several questions about whether
government spending for various programs should be
increased
• Ordering of Response Options
• Recency effect: tendency to endorse last option in list
• most likely when interviewer reads to respondent
• Primacy effect: tendency to endorse first option in list
• most likely when respondent reads to self (self-administered)
or predictable scale used
Center for Survey Research
November 8, 2013
#6 - Recall/Retrieval Difficulty
When did you
purchase your
car radio?
Center for Survey Research
Tourangeau (2000)
July 17, 2015
• Recall tasks can involve a
complicated process for respondents
(cognitive burden)
• Telescoping
• Situating an event between landmarks
• More recent, more likely to recall
• Greater impact/salience,
more likely to recall
• Is the respondent in a position to know the answer?
(e.g., parent’s income on college student survey)
Center for Survey Research
July 17, 2015
#7 - Estimation Difficulty
•
•
•
In the past twelve months, since [DATE], how many different times have you been admitted to a
hospital as a patient overnight or longer?
In the past year, how many times have you seen or talked with a medical doctor or a physician’s
assistant about your health?
During the past 4 weeks, how much have you been bothered by emotional problems (such as feeling
anxious, depressed, or irritable)?
•
•
•
How many hours of television did you watch yesterday?
How many hours of television did you watch last month?
In a typical week, how many hours of television do you watch?
•
We all, from time to time, compare our lives with those of other people, such as co-workers, family
members, friends, neighbors, people you went to school with, and so on. We would now like to ask
you some questions about how you see yourself in relation to others.
• Counting, general impressions, and hypotheticals can be
difficult for respondents
Center for Survey Research
• Reference period/point?
• Ask in a series of questions?
• Try a longer question (using familiar words, examples)?
November 8, 2013
#8 - Attitudinal Questions
•
•
•
Do you think marijuana should be made legal, or not?
On the average, [racial minority] have worse jobs, income, and housing than white
people. Do you think these difference are mainly due to discrimination?
How satisfied are you with how your life has turned out so far? Please use a scale of
1 to 10, where 1 is extremely dissatisfied and 10 is extremely satisfied.
• Factual vs. attitudinal questions
• Measuring subject states of respondents
•
•
•
•
Center for Survey Research
“No right or wrong answers”
May not be fixed
No mental “card catalog” of topics
Capturing weak attitudes and nonattitudes
– “…or have you not thought much about this?”
– Midpoint, No Opinion, Don’t Know options
November 8, 2013
#9 - Sensitive Topics
The Casual Approach:
Do you happen to have murdered your wife?
Do you happen to jog or not?
(Gallup)
The Numbered Card Approach:
Would you please read off the number on this card that corresponds to what
became of your wife? (Hand card to respondent.)
(1) Natural death (2) I killed her
(3) Other (What?)
The Everybody Does It Approach:
As you know, many people have been
killing their wives these days. Do you
happen to have killed yours?
In talking to people about elections, we often find
that a lot of people were not able to vote because
they weren’t registered, they were sick, or they just
didn’t have time. How about you—did you vote in
the elections this November? (ANES)
The Other People Approach:
Do you know any people who have
murdered their wives? How about yourself?
Center for Survey Research
Bradburn (2004)
Some people say that most people can be
trusted. Others say you can’t be too careful in
your dealings with people. How do you feel
about it? (GSS)
November 8, 2013
• Sensitive answers
• A short list of topics shown to be sensitive:
•
•
Pap smears, mammograms, disease prevention, library card
ownership, voting, reading, exercise, income, charitable giving,
drinking, gambling, sexual activity, illegal activity.
Note: List possibly susceptible to culture, time, gender, etc.
• Demographics: early or later in questionnaire?
• Income: try broad ranges, if initially refused.
Center for Survey Research
November 8, 2013
#10 - Formatting
KEEP IT SIMPLE
Center for Survey Research
November 8, 2013
Visual Cues and Heuristics
Special Issue:
Topics on Survey Measurement
and Public Opinion
POQ (2013) 77 : S1
Center for Survey Research
For web surveys:
1. Middle means typical
2. Left and top mean first
3. Near means related
4. Up means good
5. Like means close
November 8, 2013
The Good, the Bad, and the Ugly
20
• Do you have access to high speed internet, either in your home, at
work, or somewhere you can easily get to like your public library
or a nearby relative?
• Do you agree or disagree that teens should not be fined for not
obeying the local outdoor smoking ordinance?
• What is the status of this house? (Indonesian Family Life Survey)
–
–
–
–
–
Self-owned
Occupying
Occupying illegally without rent
Occupying illegally with rent
Rented/contracted
• Do you agree or disagree with the following statement: the federal
government has gotten totally out of control and threatens our
basic liberties unless we clear house and commit to drastic
change. (FOX News Poll)
Center for Survey Research
November 8, 2013
Workshop
• Scenario: The Happiness Task Force has hired
your team to develop a survey to gauge
respondents’ happiness.
• Rules: Draft five survey questions, and you
must include at least:
•
•
•
•
Center for Survey Research
One demographic question
One question that uses a Likert scale (e.g.,
1=strongly disagree …. 5=strongly agree)
One additional closed-ended question
One open-ended question
July 17, 2015
Cognitive interview
Cognitive expert review
Focus group
Field pretest
Center for Survey Research
November 8, 2013
Focus Group
http://lovestats.wordpress.com/dman/
Center for Survey Research
July 17, 2015
Focus Group
• Guided discussion among 6-10 led by a
moderator
• How do people think about this topic?
• Have we missed something?
• Language people use
• Technology: Online, teleconference, twoway
• Study protocol development
Center for Survey Research
November 8, 2013
Sample Focus Group Protocol
• What is your job title and what kind of
work do you do?
• Main activities?
• Change much from day to day?
• When you think about job skills, what
kinds of things come to mind?
• Skills shared with others in this
group?
• Skills required?
Center for Survey Research
November 8, 2013
Cognitive Expert Review
• Experienced survey
researcher/methodologist or cognitive
psychologist
• Not substantive expert
• Written review with follow-up discussion
• Measurement problems
• Solutions (e.g., rewording, reordering)
• Testing options
Center for Survey Research
November 8, 2013
Center for Survey Research
November 8, 2013
CSR: Might need to be a little more specific – an alternative
from the Current Population Survey is: Are you employed by
government, by a private company, a nonprofit organization,
or are you self-employed or working in the family business?
Center for Survey Research
July 17, 2015
CSR: This will be a tough question and may be subject to
inaccuracy. Would recommend asking if they have provided
help in a specific area (e.g., help with food) – and then, how
many times have they provided help in that specific area and
to how many different people.
Center for Survey Research
July 17, 2015
CSR: Do you need both Q5 and Q6? I am not sure how much
additional information Q6 gains you for additional respondent
burden.
CSR: Might be good to be consistent across the items – you
use two years here, then five years, then last year – may
confuse respondents.
CSR: Will respondent know the level of political influence of
other people?
Center for Survey Research
CSR: Large number of scale points. Would be good to ensure
that respondents are able to make distinctions between such
a large number of points. I would expect that there would be
concentration of responses at a limited number of points.
Might also consider giving the respondent a showcard with
the response scale as this might make it easier for them to
provide a response that uses more of the scale.
November 8, 2013
Cognitive Interview
• One-on-one interview with trained
interviewer(s)/researcher(s)
• Protocol: survey questions + item-by-item
probes
• Respondent “thinks aloud” as he/she
answers each question
• Directed probes after questions
• Protocol just a guide
Center for Survey Research
November 8, 2013
“Thinking Aloud”
• Windows example before interview
• Early probing: “Tell me more about that”,
“Tell me what you are thinking”
• Concurrent
• May be difficult
Common Directed Probes
(from Willis 2005, p. 48)
Cognitive Probe
Example
Comprehension/
interpretation probe
What does the term “outpatient” mean to you?
Paraphrasing
Can you repeat the question I just asked in your own
words?
Confidence judgment
How sure are you that your health insurance covers
drug and alcohol treatment?
Recall probe
How do you remember that you went to the doctor five
times in the past 12 months?
Specific probe
Why do you think that cancer is the most serious
health problem?
General probes
How did you arrive at that answer?
I noticed that you hesitated. Tell me what you were
thinking.
Was that easy or hard to answer?
http://lovestats.wordpress.com/dman/
Center for Survey Research
November 8, 2013
Examples – AVS
• Revised: Scientists should modify human genes to cure
serious diseases.
Center for Survey Research
November 8, 2013
Examples – AVS
• Original: How good does it make you feel when you see
the American flag flying?
EXTREMELY GOOD
VERY GOOD
NOT VERY GOOD
NOT AT ALL
Finding: R feels neutral
Revised: Changed scale and included neutral option
• Original: If I have worked hard, I deserve to have time for
fun and pleasure.
Finding: Pleasure not viewed positively
Revised: If I work hard, I deserve time to enjoy life.
Center for Survey Research
November 8, 2013
Pilot/Field Pretest/Pretest
• Small scale version of main study (n=2030+)
• Mirror procedures of main study
• Use debriefing questions
• Identify common problems from debriefing
forms (observe/talk with interviewers)
• Examine your survey data
Center for Survey Research
November 8, 2013
Debriefing Questions
• Were there any words or phrases in any of the questions
that you felt were unclear?
• Please list any questions that you felt were not easy to
read or required you to read them more than once.
• Were there any questions that you felt you did not have
enough knowledge to answer?
• Were there any questions that you felt were too personal
or were uncomfortable answering?
• One of our major challenges in conducting this study will
be obtaining participation from busy executives like
yourself. Please provide any thoughts you might have
about what we could do to encourage participation.
Center for Survey Research
November 8, 2013
Debriefing Questions
• Did you have technical difficulties with any of the following
tasks while completing this survey? (Select all that
apply.)
Seeing the text of a question
Seeing all of the responses for a question
Entering an answer to a question
Going back to a previous question
Changing an answer to a question
Exiting the survey
Finding out how to contact us
Figuring out how far along you were in the survey
I didn’t have any technical difficulties with the survey
Center for Survey Research
November 8, 2013
• Original: How important to you are greenways and trails?
• Finding: Rs do not understand greenway.
• Revised: Add a definition.
• Original: How satisfied or dissatisfied are you with
greenways and trails?
• Finding: New residents unable to answer.
• Revised: Must live in the city for 6+ months.
• Original: What is your date of birth?
• Finding: Rs uncomfortable with question.
• Revised: Use age range.
• Original: How would you rate: Timely arrival of products
on back order?
• Finding: Never had back ordered product.
• Revised: Asked screener question. Added reference
period.
Center for Survey Research
November 8, 2013
Blank
Center for Survey Research
November 8, 2013
http://lovestats.wordpress.com/dman/
Center for Survey Research
November 8, 2013
Reliability and Validity
(from Singleton and Straits 2005, pp. 90-105)
• Reliability
• Internal consistency
• Test-retest reliability
• Construct validity
• Intercorrelations
• Discriminant validity
• Convergent validity
• Known groups
Center for Survey Research
November 8, 2013
References
Questionnaire design:
Bradburn, N. et al. (2004). Asking Questions.
Converse, J., and Presser, S. (1986). Survey Questions: Handcrafting the
Standardized Questionnaire.
Fowler, F.J. (1995). Improving Survey Questions.
Tourangeau, R. et al. (2000) . Psychology of Survey Response.
Focus groups: Krueger, R., and Casey, M. (2000). Focus Groups.
Cognitive interviewing: Willis, G. (2005). Cognitive Interviewing: A Tool for
Improving Questionnaire Design.
Reliability/validity: Singleton, R.A., Jr., and Straits, B.C. (2005).
Approaches to Social Research.
Data collection:
Dillman, D et al. (2009). Internet, Mail and Mixed Mode Surveys.
Couper, M. (2008). Designing Effective Web Surveys.
Oishi, S.M. (2003). How to Conduct In-Person Interviews for Surveys.
Center for Survey Research
November 8, 2013
Thank You!
For more information:
Ashley Bowers
Stacey Giroux
Lilian Yahng
[email protected]
[email protected]
[email protected]
Center for Survey Research
http://csr.indiana.edu
Center for Survey Research
November 8, 2013
Cognitive Testing Plan
• Recruiting and scheduling subjects
• How? Convenience sample, usually recruit through
personal networks, Craigslist, social media, flyers,
newspaper ads, community groups
• What types? Representative of the population you are
studying, make sure some respondents will go down
all paths in questionnaire
• Incentive usually given (unless volunteers) - $25-$50
• Generally, 5-25+ subjects per round (often only time
and $$ for 1 round but 2-3 is desirable)
• Train multiple interviewers
• Involve in questionnaire development
• Learn background on technique and probing
Center for Survey Research
November 8, 2013
Cognitive Testing Plan (cont’d)
• Mode of data collection? In-person may work
better but can do by telephone
• Where to conduct? Quiet lab or non-lab
setting
• Record interview and/or take notes for each
interview
• How long of an interview? 15 minutes – 2 hours.
1 hour recommended (may be able to get
through a 20 minute questionnaire)
• Hold debriefing and may write a report
Center for Survey Research
November 8, 2013