Transcript Writing Survey Questions

Writing Survey Questions
Research Methods for Public
Administrators
Dr. Gail Johnson
Dr. G. Johnson,
www.reseachdemystified.org
1
Survey Options: Amount of
Structure
 Structured
 Close-ended
questions
 Semi-structured
 Open-ended
questions
Dr. G. Johnson,
www.reseachdemystified.org
2
Structured Questions
 How satisfied or unsatisfied are you with the graduates of
the teachers college?
 Very Unsatisfied <---- > Very Satisfied
 How helpful or unhelpful have the agricultural consultants
been in working with you in the past year?
 Very Unhelpful <------- > Very helpful
 How useful, if at all, has the program evaluation workshop
been in helping your evaluate your program?
 Of little usefulness <------- > Very Useful
Dr. G. Johnson,
www.reseachdemystified.org
3
Semi-Structured Questions
 What are the greatest assets you have
observed in the MPA graduates?
 What knowledge and skills should MPA
graduates have for the 21st century?
 What are the three things that you learned
from the MPA program that you use most
frequently?
Dr. G. Johnson,
www.reseachdemystified.org
4
Structured vs Semi-structured
Which to Choose?
 Structured, check-a-box questions are easier
for participants to complete and easier for
researchers to analyze.
 But
hard to develop and sometimes feels
impersonal.
 Unforgiving of mistakes: if you ask the wrong
question, use the wrong word or have a type or
grammatical error on a written survey: there is
no way to change it. You have to live with it.
Dr. G. Johnson,
www.reseachdemystified.org
5
Structured vs Semi-structured
Which to Choose?
 Semi-structured, open-ended questions are harder
for participants to complete because it takes much
more time to write-in the answers.

But they get to tell their story in their own words.
 Harder for researchers to analyze because the
English words can have many meanings

But it may provide researchers with new insights: ideas
and views that had not considered
Dr. G. Johnson,
www.reseachdemystified.org
6
Structured vs Unstructured
Which to Choose?
 My advice:
 If
you have a lot of open-ended questions,
better to interview in person than ask them to
complete a self-administered survey.
Dr. G. Johnson,
www.reseachdemystified.org
7
Writing Closed-Ended Survey
Questions
 Harder than it might appear
 Takes more time that people think
 Requires an in-depth knowledge of the
topics to be covered
 Requires a comprehensive awareness of the
words that people are likely to use when
talking about the topics
Dr. G. Johnson,
www.reseachdemystified.org
8
Writing Closed-Ended Survey
Questions
 “Every questionnaire, must finally be
handcrafted. It is not only that
questionnaire writing must be ‘artful’; each
questionnaire is also unique, an original.”
--Converse and Presser
Dr. G. Johnson,
www.reseachdemystified.org
9
Close-ended: Check a Box
 Single question with a response set:
 How frequently, if at all, have you visited
the community center in the last 3 months?
 Did
not visit (0 times)
 1-3 times
 4-6 times
 7-9 times
 10 or more times
Dr. G. Johnson,
www.reseachdemystified.org
10
Close-ended: Check a Box
 Single question with a response set:
 Including all sources of income, what was your
gross household income last year?








Less than $15,000
$15,000 to $25,000
$25,001 to $50,000
$50,0001 to $75,000
$75,001 to $100,000
$100,001 to $125,000
$125,001 to $150,000
Greater than $150,000
Dr. G. Johnson,
www.reseachdemystified.org
11
Using Intensity Scales
 Multiple-choice answers to survey questions are
sometimes called response sets
 Intensity scales are better than simple agree or
disagree responses, better than simple yes or no
responses
 Intensity scales provide a way to gage the intensity
of feeling:

Distinguish between strong feelings from the
mediocre/whatever feeling
Dr. G. Johnson,
www.reseachdemystified.org
12
One-Way Intensity Scale
 One-Way Scale: tries to capture a single
dimension (or construct) that goes from low
to high with a meaningful middle category
Always or Almost Always
 Mostly
 About Half of the Time
 Occasionally
 Never or Almost Never

Dr. G. Johnson,
www.reseachdemystified.org
13
Soften the Ends of the Scales
 Soften the ends of the scales by phrasing “Always
or Almost always” and “Never or Almost never”
 You want to encourage people to use the full range
of the scale

It is rare for something to be absolutely never or always



I almost never watch TV but it would be untrue to say that I
never watch TV
In my mind, I watch TV less than occasionally but that would
be truer than saying never
Absolute ends limits the usefulness of using a scale
Dr. G. Johnson,
www.reseachdemystified.org
14
A Goldilocks Intensity Scale
 Would you say the amount of elective
courses offered are too many, too few or
about right?
Much too many
 Somewhat too many
 About right
 Somewhat too few
 Much too few

Dr. G. Johnson,
www.reseachdemystified.org
15
A Goldilocks Intensity Scale
 “About right” is meaningful information—if
fact, the program director would want to see
a majority of students reporting that the
number of elective courses are “about right”
Dr. G. Johnson,
www.reseachdemystified.org
16
Intensity Scales with a Neutral
Middle
 Likert Scale
Strongly Agree
 Agree
 Neither Agree or Disagree
 Disagree
 Strongly Disagree

Dr. G. Johnson,
www.reseachdemystified.org
17
Intensity Scales with a Neutral
Middle
 Effectiveness/Ineffectiveness Scale
Very Effective
 Generally Effective
 Neither Effective Or Ineffective
 Generally Ineffective
 Very Ineffective

Dr. G. Johnson,
www.reseachdemystified.org
18
Survey: Intensity Scales
 Yes/No Intensity Scale
Definitely Yes
 Probably Yes
 Neither Yes or No
 Probably No
 Definitely No

Dr. G. Johnson,
www.reseachdemystified.org
19
Exits: No Opinion, Not Applicable,
Don’t Know
 Questionnaires/survey instruments are
designed for everyone to answer
 But sometimes people are asked questions
that they cannot answer because the
question does not apply or they truly have
no opinion
Dr. G. Johnson,
www.reseachdemystified.org
20
Exits: No Opinion, Not Applicable,
Don’t Know
 For example, all MPA students might be
asked about the quality of elective courses
but some may have not yet taken any
 You do not not want them offering an
opinion
Dr. G. Johnson,
www.reseachdemystified.org
21
Exits: No Opinion, Not Applicable,
Don’t Know
 So, you add an exit: “no opinion, or not
applicable, or don’t know” to the response
set
 Do not ever delude yourself into thinking
you can force people to answer your
question if you do not give them an exit
 Remember,
their participation is always
voluntary!!
Dr. G. Johnson,
www.reseachdemystified.org
22
Why Exits are Needed
 The middle category conceptually is trying to
capture an opinion.
 A person may not really care, one way or another,
if the MPA program decides to change its name.
The person is indifferent but that is still an opinion
 The exit allows the person to say I have no
information about the proposed name change to
have an opinion. They are not indifferent.
 Subtle distinction but in some situations it might
matter
Dr. G. Johnson,
www.reseachdemystified.org
23
Adding “Exits” to Scales
 Likert Scales
Strongly Agree
 Agree
 Neither Agree or Disagree
 Disagree
 Strongly Disagree
 Don’t know, No Opinion

Dr. G. Johnson,
www.reseachdemystified.org
24
Do Exits Work?
 Sometimes
 At
best, the provide
 A way for people to complete the survey
with minimal frustration (and one goal is to
make it easy for people to complete the
survey)
 Meaningful information (and avoid junk
data)
Dr. G. Johnson,
www.reseachdemystified.org
25
Do Exits Work?
 But some people may not want to admit they don’t
know
 People have been asked their opinions about
public programs that do not exist
 Rather than say “don’t know”, they find the
middle category: “I give a 3”; This is junk
data
 This suggests that analysts need to be very
careful about how they analyze middle
categories
Dr. G. Johnson,
www.reseachdemystified.org
26
Big Exits: Skip Questions
 Researchers can use skip questions when
they want to target a handful of questions to
specific participants.
 For example, in an organizational survey,
the researchers might want to ask a few
questions that specifically apply to
supervisors.
Dr. G. Johnson,
www.reseachdemystified.org
27
Big Exits: Skip Questions
 They
ask: do you supervise people.
– This is a time to use a simple yes or no
response set.
 If
yes, please answer the following 5
questions (questions 20-25).
 If no, please skip to question 26.
Dr. G. Johnson,
www.reseachdemystified.org
28
Big Exits: Skip Questions
 Another example: if conducting a citizen
survey, the researcher might ask:
 Did
you use the library in the past year?__Yes
___ No
If yes, please answer questions 10-13.
 If no, please skip to question 14.

Dr. G. Johnson,
www.reseachdemystified.org
29
Writing Good Closed-Ended
Questions
 Give people permission to provide answers
on either side of scale
 Phrasing
can seem awkward but needed to
avoid bias:

“How likely or unlikely are you to vote in the
election” sends the signal that it is OK to answer
either likely or unlikely.
Dr. G. Johnson,
www.reseachdemystified.org
30
Avoid Ranking Questions!!
 They appear to be easy and meaningful but
appearances are deceiving
 They are very hard to analyze
 They give very limited information
 They can even distort information
 See text for my full and complete rant!!
Dr. G. Johnson,
www.reseachdemystified.org
31
Writing Survey Questions:
My Best Advice
 Locate surveys others have used and see if
there are questions that will work for your
purpose
 Use clear, simple language
 Encourage a range of responses using a 5point scale (or even a 7-point scale)
 Avoid
simple yes or no responses when asking
about opinions
Dr. G. Johnson,
www.reseachdemystified.org
32
Writing Survey Questions:
My Best Advice
 Ask only one question at a time
 Avoid
double or triple-barreled questions
 “Do you think your supervisor
communicates well with staff about their
performance as well as about what is
happening in the agency?”
 She might do one well but not the other.
How would employee answer this question?
Dr. G. Johnson,
www.reseachdemystified.org
33
Writing Survey Questions:
My Best Advice
 Close all the “gates”
 Provide
instructions for skip questions
 Provide mutually exclusive responses
(eg. No overlapping categories for age or
income)
Dr. G. Johnson,
www.reseachdemystified.org
34
Writing Survey Questions:
My Best Advice
 Focus on current experiences: memory decays
over time!

The more distant the time, the more likely their recall
will be inaccurate.
 Leave exits (no opinion, not applicable)
 Ask only the demographic questions that you
actually need, keeping in mind that some people
may be identifiable (eg. the only woman on the
faculty will be identifiable)
Dr. G. Johnson,
www.reseachdemystified.org
35
Writing Survey Questions:
My Best Advice
 Pre-test, pre-test, pre-test
 With
real participants, find out:
 Is each question clear?
 Are there unknown words or phrases?
 Is there a better way to ask each question?
 What questions did they expect to see but
did not?
 Remember: ask them for suggestions
Dr. G. Johnson,
www.reseachdemystified.org
36
Practice: Writing Survey
Questions
 Thinking about the observation of the room
exercise:
 Develop
5 closed-ended questions that would
enable everyone in the class to assess the key
characteristics of the room
 Develop 10 questions to assess student
opinions about the quality of the MPA
program?
Dr. G. Johnson,
www.reseachdemystified.org
37
Practice: Writing Survey
Questions
 Compare with partners
 Notice:
 Similarities and differences?
 How many different topics were covered?
 How many different scales were used?
 While there are guidelines that help, there is
tremendous diversity things we each think
matters most and the words we prefer
Dr. G. Johnson,
www.reseachdemystified.org
38
Survey Questions In The News
 Washington Post-ABC survey (11/09) asked:
 “On another subject, you may have heard about
the idea that the world's temperature may have
been going up slowly over the past 100 years.
What is your personal opinion on this - do you
think this has probably been happening, or do you
think it probably has not been happening?”
Dr. G. Johnson,
www.reseachdemystified.org
39
Right Concept?
 Is it global warming or climate change?
 Do some people think climate change is
happening but not global warming?
 How would they answer this question?
Dr. G. Johnson,
www.reseachdemystified.org
40
Survey Questions In The News
 “There's a proposed system called "cap and trade."
The government would issue permits limiting the
amount of greenhouse gases companies can put
out. Companies that did not use all their permits
could sell them to other companies. The idea is
that many companies would find ways to put out
less greenhouse gases, because that would be
cheaper than buying permits. Would you support
or oppose this system?”
Dr. G. Johnson,
www.reseachdemystified.org
41
Right Question?
 Is this the most accurate description of cap
and trade?
 For those who do not support cap and trade
but do want the government to take action
to mitigate climate change—how would
they answer?
Dr. G. Johnson,
www.reseachdemystified.org
42
Dilemma: Short but Biased?
 My belief: to only ask about a single option
for a complex policy issue is a form of bias
 But
you can also see that the survey would get
quite long if all options were covered in the
midst of a survey about many policy issues
 It is a dilemma: need to keep it short and
simple, but need to be comprehensive to avoid
bias
Dr. G. Johnson,
www.reseachdemystified.org
43
Writing Survey Questions:
Final Words
 Writing questionnaires are harder than might
appear
 The English language is sometimes difficult: the
same words can mean different things to different
people
 It is sometimes hard to get a complex issue nailed
down in one question

It may make take ten.
 Pre-test, Pre-test, Pre-test
 Folks: you won’t write a great survey in a week!
Dr. G. Johnson,
www.reseachdemystified.org
44
Credibility of
Survey Results
 If the issue matters to you, look at the exact
wording of the question and the possible answers





A news report may not give an accurate rendering
Make sure the question did not try to lead to desired
answers (bias)
Make sure there are an equal number of positive and
negative response options
Consider whether they asked the right questions for the
topic
Be on the look-out for fatally flawed questions
Dr. G. Johnson,
www.reseachdemystified.org
45
Credibility of
Survey Results
 At
the same time, keep in mind, it really is hard
to write good survey questions
 The appearance of bias might be the result of
how they decided to handle the comprehensive
vs keep in short dilemma
 It is easy to criticize any survey but after you
have done a few surveys, you will have more
compassion for the work of others
 Perfection is not the appropriate criteria
Dr. G. Johnson,
www.reseachdemystified.org
46
Creative Commons
 This powerpoint is meant to be used and
shared with attribution
 Please provide feedback
 If you make changes, please share freely
and send me a copy of changes:
 [email protected]
 Visit www.creativecommons.org for more
information
Dr. G. Johnson,
www.reseachdemystified.org
47