Transcript Research Method Lab - University of Chicago

Research Method Lab
Designing and Writing Effective
Surveys
Overview

This talk will address the following:




Types of survey questions
14 tips for improving your survey
writing
Common survey pitfalls
10 criteria for a good survey
Types of Survey Questions

Open-ended or Verbal




The expected response is a word,
phrase, or an extended comment
Responses can produce useful
information but analysis can present
problems
Some form of content analysis may be
required unless the information
obtained is for special purposes
Use sparingly or stick with interviews
Types of Survey Questions

Closed-ended, Likert Scale

Not very important

1


2
3
Very Important
4
5
6
7
The respondents must indicate by
selecting the number that best
represents their attitude
A quick and easy way to measure more
nuanced ranges of opinions
Types of Survey Questions


Closed-ended, Multiple Choice
Which
 A.
 B.
 C.
 D.
learning activity did you like best?
Small group
Role play
Mapping
Computer modeling
When you want respondents to pick
the best answer or answers,
consider this type
Be sure to include specific directions
Types of Survey Questions


Closed-ended, Ordinal
Please write a number between 1 and 5 next to each item below.
Put a 1 next to the item that is most important to you in selecting
an on-line university course. Put a 5 next to the item that is
LEAST important. Please use each number only once.





___
___
___
___
___
availability of instructor for assistance
tuition cost for the course
ability to work in groups with other students
quality and quantity of instructor feedback
number of students enrolled
Great for rating things in relation to
other things
Again, be specific with instructions
Types of Survey Questions

Closed-ended, Categorical

How many hours do you study on a school night?

0-1
2-3
3+
Be sure that categories do not
overlap
Be sure to break down items into
distinct and clear categories
Each respondent must “belong” in a
specific category
Types of Survey Questions


Closed-ended, Numerical
How many years have you taught
full time?



When the answer must be a real
number, ask a numerical question
Use these, especially with large survey
populations, to do an analysis of age
range, years taught, etc.
Not as effective with smaller samples
Types of Survey Questions












Cumulative/Guttman Scale
Here the respondent checks each item with which they agree
The items are constructed so that they are automatically
cumulative– if you agree to one, you probably agree to all of the
ones above it on the list
Can be a good way to gauge how people feel about controversial
topics
Requires care when writing so that it doesn’t seem leading
Example (from a survey guide):
Please check each statement that you agree with:
__ Willing to permit immigrants to live in the U.S.
__ Willing to permit immigrants to live in your community.
__ Willing to permit immigrants to live in your neighborhood.
__ Willing to have an immigrant as a next door neighbor.
__ Willing to let your child marry an immigrant.
Which One Do I Use?







Use Open-ended questions…when you want to get
the respondent’s own words
Use Likert-Scale questions to assess a person’s
feelings about something
Use Multiple-Choice questions when there are a
finite number of options
Use Ordinal questions to rate things in relation to
other things
Use Categorical questions when the respondent
must fall into one section
Use Numerical questions for real numbers like
age, number of months, etc. for large surveys
Use Cumulative questions to gauge degrees of
feelings about controversial or complex issues
Contingency Questions






Contingency questions can be any of the styles
previously discussed
Typically used when you want to filter out certain
respondents
Respondents are directed to other questions in
the survey
Too many jumps can confuse the reader
Example:
Have you ever been bullied in school?
Yes
 No
If “yes,” go to question #4

14 Tips for Improving Survey Writing




1. Remember your survey’s purpose
All other rules and guidelines are based
on this one
There is a reason you decided to spend
time and effort doing a survey and you
should ensure that every question you
ask supports that reason
If you start to get lost while writing your
questions, refer back to this rule
14 Tips for Improving Survey Writing




2. If in doubt, throw it out
This is another way of stating the first
rule
A question should never be included in a
survey just because you can’t think of a
good reason to discard it
If you cannot come up with a concrete
research benefit that will result from the
question, don’t use it
14 Tips for Improving Survey Writing




3. Keep your questions simple
Compound sentences force respondents
to keep a lot of information in their
heads, and is likely to produce
unpredictable results
The following question is too complex for
a clear, usable answer and needs to be
broken down into component parts
Example: Imagine a situation where your department chair is out sick, two
new students have enrolled mid-term, and the district’s test scores are
about to be released next week. How supported do you feel by the
administration?
14 Tips for Improving Survey Writing



4. Stay focused—avoid vague issues
If you ask “please rate your satisfaction
with the school’s discipline policy” the
answers will not lead to any specific
action steps
Particular elements of the school’s
discipline policy must be probed if
responses are to result in specific
recommendations
14 Tips for Improving Survey Writing



5. If a question can be misinterpreted, it
will be
“What time do you normally eat dinner”
will be answered differently by people
living in different regions because it can
refer to the midday or evening meal
Be clear, concise, always be aware of
imprecise language, and avoid double
negatives
14 Tips for Improving Survey Writing



6. Include only one topic per question
“Please rate your satisfaction with the
implementation and enforcement of your
school’s dress code” combines two issues
You need to break this question into two
smaller ones in order to get responses
that can lead to recommended actions
14 Tips for Improving Survey Writing




7. Avoid leading questions
It is easy, and incorrect, to write a
question that the respondent believes has
a “right” answer
“Most P.E. teachers believe that exercise
is good for you. Do you agree?” is an
example of a leading question
Even the most well-meaning researcher
can slant results by including extra
information in a question
14 Tips for Improving Survey Writing




8. Consider alternative ways to ask sensitive
questions
Income, drug or alcohol consumption, sexual
habits, religious beliefs, and political views are
obvious examples of sensitive topics
But even topics like teaching styles, identifying
biases (racial and gender), classroom
management enforcement, and coworker
relations can be sensitive
Questions like “did you vote in the last election”
forces respondents into a corner—they might be
unwilling to admit they did not vote because of
civic pride or embarassment
14 Tips for Improving Survey Writing



9. Make sure the respondent has enough
information
Asking respondents “How has the school board
affected your classroom” isn’t as effective as “Last
month the school board passed a ruling that
requires a district committee composed of parents
and teachers to review all textbooks. Did you
know this?” followed by “What kind of changes to
your department’s curriculum have you seen?”
Break these kind of questions into two parts- a
screening item and a follow-up question
14 Tips for Improving Survey Writing





10. Response questions need to be mutually
exclusive and exhaustive
If response questions are not mutually exclusive,
the respondent will have more than one
legitimate place for their answer
The response choices “1-2,” “2-3,” and “more
than 3” pose a problem for someone who answers
“2”
You must also ensure that the response options
you provide cover every possibility
Providing a list of “reading,” “math,” and “science”
for “What is your toughest class?” isn’t exhaustive
enough
14 Tips for Improving Survey Writing





11. Keep open-ended questions to a
minimum
While open-ended questions are a
valuable tool, they should not be
overused
They can result in respondent fatigue
where you’ll only get short answers
These short answers can be avoided by
using a set of well-designed, closedended questions
Open-ended questions also pose problems
in terms of coding and analysis
14 Tips for Improving Survey Writing




12. People interpret things differently,
particularly when it comes to time
Trouble spots include “always,”
“sometimes,” and “never”
You must build in a temporal frame of
reference to ensure that all respondents
are answering in the same way
Example: “I am going to give you a list
of magazines. For each one, please
indicate whether you have read it
regularly. By regularly I mean at least
twice this week.”
14 Tips for Improving Survey Writing





13. Consider a “don’t know” response
It is useful to allow people to say they simply do
not have an opinion about a topic
Some researchers worry that people will opt for
that choice, reducing the ability to analyze
responses
Evidence shows this fear is largely unfounded
If you only want information from those with an
informed opinion about an issue or interest in a
topic, offer a “don’t know” choice
14 Tips for Improving Survey Writing






14. Provide a meaningful scale
The end points of Likert scales must be anchored
with meaningful labels
The number of scale points can have little effect
on the conclusions you draw later
An odd number of points provides a middle
alternative and provides a good way for
respondents to head for the center
If measuring extreme opinions, use a scale with a
greater number of points
You generally gain nothing by having a scale with
more than 7 points
Common Survey Pitfalls: Bad
Questions Stink!





Well written questions are critical
Participants must stay interested
If your respondents start to feel alienated by
threatening, emotional, lengthy, or difficult
questions, response rates are likely to go
down
Respondents can get frustrated if your questions
do not provide answer choices that match their
opinions or experiences
The quality of your collected data will suffer, your
analyses will be less meaningful, and the whole
research process will be useless
Common Pitfalls: Writing Good
Questions is an Art




There are infinitely good ways to create
surveys- no one person holds all the
knowledge on the subject
There are also a number of common
pitfalls, which if avoided, will increase the
usefulness of survey data
Nothing compensates for poorly worded
questions
The objective is to keep survey questions
focused on single issues and topics, to
keep them short, and make sure
everyone can understand the questions
Barreling vs. Specificity



Barreled questions ask respondents
to rate 2 or more behaviors or
issues in a single question
It is impossible to attribute one
answer to either issue
This is the most common problem
with survey questions
Let’s correct the barreling…



Bad example: “When making
assignments, my supervisor gives
clear, achievable goals that are
within my control”
How can we improve this question?
Good example: “When making
assignments, my supervisor gives
me achievable goals”
Jargon vs. Clarity




Specialized terminology is not generally
understood
Specialized or unfamiliar words prevent
those not familiar with the terminology
from understanding and answering the
question in a responsible way
Using acronyms is a possible use of
jargon
Stick with common speech
Let’s correct the jargon…



Bad example: “How would you rate
your child’s WISC-III assessment
experience?”
How can we improve this question?
Good example: “How would you
rate your child’s experience taking
the Wechsler Intelligence Scale
test?”
Loose Bundling vs. Anchoring



This occurs when questions are too
general and not grounded in specific
behaviors
To improve loosely bundled
questions, a more specific behavior
must be identified in the question
What does the behavior look like?
Let’s correct the loose bundling…



Bad example: “I am interested in
my students”
How can we improve this question?
Good example: “I regularly share
information about my students’
progress with their parents”
Complexity vs. Simplicity


Complexity occurs when sentence
phrasing is so long and labored that
respondents become confused
Keep questions short and focused
on single issues
Let’s correct the complexity…



Bad example: “What is the frequency of
the overall interpersonal, informal, and
formal communication between the focal
group and the targeted semiautonomous, functionally specialized
groups?”
How can we improve this question?
Good example: “What is the frequency of
the formal communication between the
focal group and your group?”
Lack of Knowledge vs. Knowledge


Refers to when respondents either
lack the knowledge necessary to
accurately answer the question or
have never bothered to organize
their knowledge around the subject
Never ask people more than they
know
Let’s correct the lack of knowledge…



Bad example: “The in-service
presenter was an expert in
differentiated instruction”
How can we improve this question?
Good example: “The in-service
presenter effectively answered
questions from the audience”
Social Desirability vs. Realism



This occurs when questions have an
obviously “correct” or socially desirable
answer
When respondents worry about giving the
“right” answer, the data will be distorted
Make sure the question prompts
respondents to answer with a reasoned
opinion instead of an emotional response
Let’s correct the social desirability…





Bad example: “I frequently belittle and
talk down to my fellow employees”
How can we improve this question?
Good example: “People in my work group
frequently belittle and talk down to their
fellow employees”
Or
“There are times when the work pressure
gets so bad that I have been known to
belittle and talk down to my fellow
employees”
Leading & Loaded Questions vs. Fact


Leading and loaded questions are
worded so that they influence
respondents’ answers, creating bias
Avoid using emotionally charged
wording
Let’s correct the leading & loaded
questions…



Bad example: “Don’t you think that
the Pledge of Allegiance should be
required in school because it
teaches patriotism?”
How can we improve this question?
Good example: “The Pledge of
Allegiance should be required in
school”
Halo Effect vs. Respondent Autonomy



Watch out for questions that link a
position with a particular person or
group
The respondent’s attitude about the
person or group may influence their
attitude about the position
Eliminate the link to a specific
person or group by creating an
unnamed group or dropping the link
Let’s correct the Halo effect



Bad example: “Do you agree with
President Bush that schools need to
be accountable to taxpayers?”
How can we improve this question?
Good example: “Do you believe
that schools need to be accountable
to taxpayers?”
Conclusion: 9 Criteria for a Good
Survey





1. The target population (the group of
people from whom you want feedback) is
well-defined
2. The actual people you survey matches
your target population
3. The group you survey is large enough
to gain useful data
4. Good follow-up minimizes nonresponse
5. The type of survey used is appropriate
Conclusion: 10 Criteria for a Good
Survey




6. The questions are well-worded
7. The survey is properly timed,
both in the amount of time it takes
to complete it, as well as when it is
distributed
8. The people giving the survey are
well-trained
9. The survey answers your
research questions