Transcript No Slide Title

Designing the Questionnaire or
Observation Form
Procedure for Developing a
Questionnaire
Specify what information will be sought
Step 1
Determine type of questionnaire and method of administration
Step 2
Determine content of individual questions
Step 3
Determine form of response to each question
Step 4
Determine wording of each question
Step 5
Determine question sequence
Step 6
Determine physical characteristics of questionnaire
Step 7
Reexamine steps 1-7 and revise if necessary
Step 8
Pretest questionnaire and revise if necessary
Step 9
Open-Ended Questions
1. How old are you? ______
2. How do you feel when you drive your Toyota minivan?
Why do you feel that way?
3. Can you name three sponsors of the Monday night football
games? _____
4. Do you intend to purchase an automobile this year? _____
5. Why did you purchase a Sony brand color television set?
_____
6. Do you own a VCR? _____
Multichotomous Questions
A fixed-alternative question in which respondents are asked to
choose the alternative that most closely corresponds to
their position on the subject.
Example
How old are you?
____ Less than 20
____ 20-29
____ 30-39
____ 40-49
____ 50-59
____ 60 or over
Dichotomous Questions
A fixed-alternative question in which respondents are asked to
indicate which of the two alternative responses most closely
corresponds to their position on the subject.
Example
Do you think laws requiring passengers in motor vehicles to
wear seat belts are needed?
Yes
No
Split Ballot
A technique used to combat response bias in which
one phrasing is used for a question in one-half of the
questionnaires while an alternate phrasing is used in
the other one-half of the questionnaires.
Example
Do you think gasoline will be more expensive or less expensive next year
than it is now?
More expensive
Less expensive
Do you think gasoline will be less expensive or more expensive next
year than it is now?
More expensive
Less expensive
Item Non-Response
A source of non-sampling error that arises
when a respondent agrees to an interview
but refuses, or is unable, to answer specific
questions.
Flame-broiled or Fried?
Original Question: “Do you prefer your hamburgers flame-broiled or fried?”
Result: flame-broiling (Burger King) beat frying (McDonald’s)
by a 3-1 margin
Revised Question #1: “Do you prefer a hamburger that is grilled on a hot
stainless-steel grill or cooked by passing the raw meat through an open
gas flame?”
Result: 53% preferred the grill (McDonald’s)
Revised Question #2: “The chain that grills on a hot stainless-steel griddle
serves its cooked hamburgers at the proper temperature without having to use
a microwave oven. And the chain that uses the gas flame puts the hamburgers
after they are cooked into a microwave oven before serving them. Just knowing
this, from which of these two chains would you prefer to buy a hamburger?”
Result: the grill (McDonald’s) beat the flame (Burger King) by a
5.5 to 1 margin
Leading Questions
A question framed so as to give the respondent
a clue as to how he or she should answer.
Example
Do you feel that limiting taxes by law is an effective way to
stop the government from picking your pocket every payday?
Yes
No
Undecided
Implicit Alternative
A problem that occurs when an important
alternative or response option is not
included in a question (or response
categories).
Example
Would you like to have a job, if this were possible?
Yes
No
Would you prefer to have a job, or do you prefer to do just your housework?
Yes
No
Implicit Assumption
A problem that occurs when a question is
not framed so as to explicitly state the
consequences, and thus it elicits different
responses from individuals who assume
different consequences.
Example
Would you like to double the number of job offers you receive as a senior?
Yes
No
Would you like to double the number of job offers you receive as a senior if
that means devoting an additional 10 hours per week to studying so as to
raise your grade point?
Yes
No
Double-Barreled Question
A question that calls for two responses and
thereby creates confusion for the
respondent.
Example
Are the food and service at the local Pizza Hut good?
Yes
No
Key: Be on the lookout for “and” and “or” in your items!
Guidelines for Question Wording
• Use simple words and questions
• Avoid ambiguous words and questions
• Avoid leading questions
• Avoid implicit alternatives
• Avoid implicit assumptions
• Avoid generalizations and estimates
• Avoid double-barreled questions
Guidelines for Question Sequencing
• Use simple, interesting opening questions
• Use the funnel approach, asking broad
questions first
• Carefully design branching questions
• Ask for classification information last
• Place difficult or sensitive questions near the
end
Guidelines to Question Sequencing (Cont)
Funnel Approach
An approach to question sequencing that gets its name
from its shape, starting with broad questions and
progressively narrowing down the scope.
Example
How would you rate your last experience when eating at
ABC restaurant?
Extremely
Poor
Poor
So-So
Good
Extremely
Good
How would you rate the quality of the service?
Extremely
Poor
Poor
So-So
Good
Extremely
Good
Example of Question Order Bias
• Demand estimation for new product
• Key question: buying interest (5-point
scale, “not at all interested-very
interested”)
Questions asked before buying
interest question
None
Asked about advantages
% very
interested
3
17
Asked about disadvantages
0
Asked about advantages and disadvantages
6
Guidelines to Question Sequencing (Cont)
Branching Question
A technique used to direct respondents to different places
in a questionnaire, based on their response to the
question at hand.
Guidelines to Question Sequencing (Cont)
Branching Question Example
1. Please answer the following question Yes or No.
Did you, personally, give money to
Canopy of Care last year?
{If Yes, go to #3.}
Y or N
2.
We are interested in why people do not contribute. The following is a list of answers
others have given. Please tell me which, if any, apply to you.
a. ___ Someone else in my household had already contributed.
b. ___ I did not have the money at the time.
c. ___ I gave to other charities.
d. ___ I volunteered my services to Canopy of Care instead of contributing money.
e. ___ I volunteered my services to other charities instead of contributing to Canopy of
Care.
f. ___ I did not give because Canopy of Care spends its money inefficiently.
g. ___ None of the above.
3.
To how many different charities do you think Canopy of Care gives money?
a. ___ 0-20
d. 81-100
b. ___ 21-40
e. More than 100
c. ___ 41-80
f. Don’t know.
Pretest
Use of a questionnaire (or observation
form) on a trial basis in a small pilot study to
determine how well the questionnaire (or
observation form) works.
ALWAYS PRETEST DATA COLLECTION FORMS!!!!!
Handling Sensitive Questions
• Don’t ask unless absolutely necessary!
• Give broad response categories
(e.g., income, age)
• Place near end of questionnaire
• “Hide” them in less sensitive questions
• Use counterbiasing statement
• Use randomized response technique
Use of Counterbiasing Statement
Recent studies have shown that a high
percentage of males use their wives’
cosmetics to hide blemishes. Have
you used your wife’s cosmetics in the
past week?
Randomized Response Model
An interviewing technique in which potentially
embarrassing and relatively innocuous
questions are paired, and the question the
respondent answers is randomly determined
but is unknown to the interviewer.
Randomized Response Technique
(example from Weiers 1988)
Initial, “benchmark” study with nonsensitive question:
“Do you likepepperoni pizza?” (65% say “yes.”)
Second survey, involving 400 respondents.
Each respondent flips coin, with flip result
not revealed to researcher.
Heads
Tails
Respondent answers question 1:
“Do you like pepperoni pizza?”
Respondent answers question 2:
“Have you ever been in jail?”
160 “yes” responses
240 “no” responses
Randomized Response Technique
(example from Weiers 1988)
160 “yes” responses
240 “no” responses
“Yes”
“No”
Pizza
130
(.65x200)
70
Jail
30
(160-30)
170
160
240
200 (.5x400)
200
Conclusion: Estimate that 30/200, or 15%
of population has “been in jail.”
Physical Characteristics of
Questionnaire: Suggestions
 questionnaire should appear simple to complete
(white space is your friend!)
 minimize number of pages (smaller fonts are OK,
provided form appears simple)
 mix-up response formats occasionally (avoids
response set bias and breaks monotony)
 use directions as necessary for each group of
items, but keep them short and simple
 number items within each section
 have respondents check boxes rather than lines
 use shading, boxes, lines, etc., to keep it
interesting
Other Issues
 Pretest!
 Know exactly how each piece of
information will be used
 NO “nice-to-know” questions!
 Back translation for international
questionnaires
 Pre-coding