Transcript Cognitive Survey Research Methodology

Introduction to
Questionnaire Design
Spring 2005 Seminar Series
Survey Research Laboratory
University of Illinois at Chicago
www.srl.uic.edu
Questionnaire design
encompasses:
1. How to write questions
2. How to draft and organize a
questionnaire
The Art of Asking Questions
You must ask the right question
Respondents must understand your
question
Respondents must know the answer
Respondents must be willing and
able to tell you the answer
Three Simple Rules for Writing
a Good Questionnaire
1) Think through your research questions and
objectives before you write questions
2) Prepare an analysis plan before you write
questions
3) Ask yourself, in relation to points #1 and #2
above, if each question on your list is
necessary? Even if the data would be
‘interesting’ it has to ultimately be used in
analysis to make the cut!
Types of Survey Questions
1) Those that ask about behaviors or facts
 Non-threatening behavior questions
 Threatening behavior questions
 Demographics
2) Those that ask about psychological states
or attitudes
3) Those that ask about knowledge
What Is A Good Question?








One that yields a truthful, accurate answer
One that asks for one answer on one dimension
One that accommodates all possible
contingencies of response
One that uses specific, simple language
One that has mutually exclusive response options
One that produces variability in response
One that minimizes social desirability
One that is pretested
Ask questions one at a time
Bad question:
In the past 6 months, what
major appliances has your
household purchased new
from the store?
Better question:
Now I’m going to read a list
of household appliances. As I
read each one, please tell me
whether or not your household
has purchased this type of
appliance new from the store
during the past 6 months.
How about…
a refrigerator?
a kitchen range or oven?
a microwave?
Ask questions one at a time
Bad Question:
Compared to one year ago, are
you paying more, less, or about
the same for your auto and life
insurance?
Better Question:
Compared to one year ago, are
you now paying more, less, or
about the same for …
a. auto insurance?
b. life insurance?
Specify
Specify who, what, when, where and how.
For example, whose income? What’s included?
Over what period of time? Example:
“In 2002, what was your total household income,
before taxes? Please count income from all
members of your household, including wages
from employment, disability, social security, and
public aid.”
Specify through cues
Example
People drink beer in many places – for
example, at home, at restaurants, at bars,
sporting events, at friends’ homes, etc.
During the past 30 days, did you drink
any beer?
Use words with singular meanings
Ambiguous:
How would you
compare how close you
are to family in your
current neighborhood
to how close you were
in your old
neighborhood? Would
you say your family is
closer here, further
here, or the same?
More Clear:
Compared to your last
neighborhood, do you
now live closer to your
family, are you further
from your family, or are
you about the same
distance?
Social Desirability
 Respondents will try to represent
themselves to the interviewer in a way that
reflects positively on them
As questions become more threatening,
respondents are more likely to overstate or
understate behavior, even when the best
question wording is used
Minimizing Social Desirability
• For socially desirable behavior, it is better to ask whether
respondents have ever engaged in the behavior before asking
whether they currently engage in the behavior
• For socially undesirable behavior, it is better to ask about
current behavior first, rather than ask about their usual or
typical behavior
• Train interviewers to maintain a professional attitude
• Self-administered computer-assisted procedures can reduce
question threat and improve reporting on sensitive questions
• Longer questions reduce sensitivity when obtaining
information on frequencies of socially undesirable behavior
Open vs. Closed Questions
General rule: closed questions are usually
better
–
–
–
–
Easier for the respondent
Less coding later
Better to have respondent do categorizing
Categories help define the question
Disadvantages of
Closed Questions
• Categories may be leading to respondents
• May make it too easy to answer without
thinking
• May limit spontaneity
• Not best when
– asking for frequency of sensitive behaviors
– there are numerous possible responses
General Principles for
Response Options
• Response categories should be consistent with
the question
• Categories must be exhaustive, including every
possible answer
• Categories must be mutually exclusive (no
overlap)
• If appropriate, include a “don’t know” category
Constructing Response Scales
• Respondents can generally only remember a maximum
of 5 responses unless visual cues are used
• Using graphic images such as thermometers and
ladders and using card sorting for compex ratings is
effective
• Number of points in scale should be determined by
how you intend to use the data
• With scales with few points, every scale can be labeled;
in longer scales, only the endpoints are labeled
Ordering Response Categories
• Usually better to list responses from the lower
level to the higher level
• Associate greater response levels w/ greater
numbers
• Start with end of a scale that is least socially
desirable
Should you use a middle category?
– Common practice is to omit it to push respondents
(Rs) toward one end or the other, on the theory that
few individuals are truly in the middle on a particular
issue
– Evidence from empirical studies shows that use of
an explicit middle alternative will often be taken by
Rs in a forced choice situation if offered; at the same
time, it does not affect the ratio of ‘pro’ to ‘con’
responses or the size of the don’t know category
– Our usual recommendation is to include it unless
there are persuasive reasons to exclude
Part 2:
How to draft and organize your
questionnaire
Ordering the Questions
•
•
•
•
•
Segment by topic
Ask about related topics together
Salient questions take precedence over less salient ones
Ask recall backwards in time
Use transitions when changing topics – give a sense of
progress through the questionnaire
• Leave objectionable questions (e.g., income) for the end
• Put demographic questions at the end
How to Start the Questionnaire
• Start with easy questions that all respondents
can answer with little effort
• First questions should also be non-threatening
• Don’t start with knowledge or awareness
questions
• First questions should be directly related to the
topic as described in the introduction or
advance/cover letter
Survey Intro/Cover Letter
Introduction should indicate:
–
–
–
–
–
who is conducting the survey
the topics to be covered in the survey
an assurance of confidentiality
any IRB stipulations
whether you offer how long it will take
depends on mode, topic, population
Physical Format of the SelfAdministered Questionnaire
• Careful formatting is necessary to
decrease errors and increase motivation
• Respondent’s needs must always take
priority, followed by interviewer and data
processors
Physical Format Checklist
• Number all questions sequentially
• Use large, clear type; don’t crowd
• ‘White space:’ Place more blank space between
questions than between subcomponents of questions
• List answer categories vertically instead of horizontally
• Avoid double/triple ‘banking’ of response choices
• Be consistent with direction of response categories
• Be consistent with placement of response categories
Physical Format, Continued
• Don’t split questions across pages. If
necessary (e.g., question requires 1.5 pages),
restate question and response categories on
next page
• Put special instructions on questionnaire as
needed, next to question
• Distinguish directions from questions
• Precode the questionnaire (vs. check boxes)
Mail questionnaires
• Include a cover letter and contact information if
the respondent needs help
• Use a booklet format
–
–
–
–
–
Easier to turn pages
Prevents lost pages
Permits double-page formats
Looks more professional
Include a title, graphic, name/address of sponsor on
cover
Testing the Questionnaire
• Preferable to test the questionnaires with
people like those in your main study study
population
• Test in same mode to be used for main
study
• Consider cognitive pretesting
Questionnaire Design Steps
1.
2.
3.
4.
5.
6.
7.
Decide what information is needed
Search for existing questions
Focus groups
Draft new questions/revise existing ones
Sequence the questions
Get peer evaluation
Revise and test on self/co-workers
Questionnaire Design Steps
8.
9.
10.
11.
12.
Think-aloud interviews
Revise/eliminate questions
Prepare interviewer instructions for
pilot test
Pilot test (10-20 cases)
Revise eliminate questions based on
respondent & interviewer comments
Questionnaire Design Steps
13.
14.
15.
16.
17.
Pilot test again, if necessary
Prepare final interviewer instructions
Be prepared to modify questionnaires if
interviewer training raises problems
After interviewing is complete, debrief
interviewers for potential problems
Use experience from one study for
future planning
Key References To Get Started
Dillman, Don. Mail and Internet Surveys: The
Tailored Design Method. New York: John,
Wiley & Sons, Inc, 2000.
Bradburn, N, Sudman, S. and Wansink, B..
Asking Questions: The Definitive Guide to
Questionnaire design – for Market Research,
Political Polls, and Social and Health
Questionnaires. San Francisco: Jossey Bass,
2004.
See also past issues of Public Opinion Quarterly