Transcript Questionnaire (Survey) Design

Questionnaire (Survey) Design
Measurement Error: Respondent’s
Cognitive Process
• Multicomponent Approaches
• Two track Theories
– High road (Follows multicomponent)
– Low road (short circuits steps of multicomponent)
Multicomponent Approach
1. Comprehension: of the question
2. Retrieval: of relevant information from the
R’s memory
3. Judgment: using the retrieved information to
judge the question
4. Selection: of an answer
Overview
• What should be asked?
• How should each question be phrased?
• In what sequence should the questions be
arranged?
• What questionnaire layout will best serve the
research objectives?
• Does the questionnaire need to be revised?
What should be asked?
• Relevancy– If you don’t need the info don’t ask
• Accuracy– reliable and valid
How should each question be
phrased?
• Open-ended
Fixed alternatives
Simple dichotomy
– Determinant choice
– Attitude rating scale
– Checklist (Check all that apply)
• Code as individual yes/no questions
– Categories should be mutually exclusive and
exhaustive
Ex. How old are you?
1. 0-12 2. 12-16 3. 16-24
4. 24-85
Choices Matter.
• <30min;30-59min;1-1.5;>1.5-<=2hrs; etc
• <2.5
Choices Matter.
Half of the respondents were given a scale that began
with <30 minutes and then proceeded in half-hour
increments ending with 4.5+ hours. The other
respondents were given the same scale except the first
five answers were compressed so that it began with
<2.5 hours. Only 16 percent of the respondents given
the first set of response alternatives reported watching
more than 2.5 hours of TV per day, but 32 percent of
the respondents given the second set of response
alternatives reported watching more than 2.5 hours of
TV per day.
The Art of Asking Questions
• Avoid complexity
– Ex. In deciding on optimum B & M location, the
franchiser requests community input on the mean
travel distance for its clientele, how far do you
travel?
The Art of Asking Questions
• Avoid leading and loaded questions
– Ex. Culvers sells a new deliciously fattening
custard, in a typical sitting how much would you
like to eat?
• Exception likert scales ask people to affiliate
with attitudes.
The Art of Asking Questions
• Avoid ambiguity
– Ex. How many times have you eaten at culvers?
The Art of Asking Questions
• Avoid making assumptions
– Ex. How would you rate the service at the new
Culvers?
The Art of Asking Questions
• Avoid “double-barreled” items
– Ex. How much do you like the burgers and custard
at Culvers?
The Art of Asking Questions
• Avoid burdensome questions that may tax the
R’s memory
– Ex. What did you order on your first visit to
Culvers?
In what sequence should the
questions be arranged?
• Order Bias- election ballots
– order of answers,
– order of questions
• Funnel Technique- general to specific
questions
• Filter question- a screening question to be
used to avoid making assumptions
– Ex. Have you ever contracted an STD?
• Pivot question- for sensitive info, like income
– Ex. What was your income last year?
• Was your income greater than 10,000?
• Was it greater than 20,000?
What questionnaire layout will best
serve the research objectives?
• Instructions to interviewers should be in CAPS,
italics,
• The survey should have plenty of free space
• Precode the responses
• Clearly direct the interviewer with skips etc.
Does the questionnaire need to be
revised?
• TEST IT!!
Suggested components
• Questionnaire number
• phone number?
• Interviewer ID
• Informed consent (part of email).
• Demographics section
Example Survey Preamble/Email
Dear Respondent:
My name is XXXXX and I am conducting a research project on business advisory boards at colleges and
universities. We are interested in the opinions of faculty from AACSB schools as The University of Somewhere
is in the start-up phase for establishing a business advisory board. Could you please take a few moments and
answer this very short survey on the attached file? Even if you have very little contact with your advisory
board I am very interested in your opinion.
By completing this survey you will be contributing to the goal of identifying best practices for establishing and
maintaining a business advisory board and I will send you the study findings upon request.
All answers will be kept confidential to the extent possible, acknowledging that any form of communication over
the Internet does carry a minimal risk of loss of confidentiality. All information will be reported in the aggregate.
Participation in this study is voluntary, and all participants may refuse to answer any question found on the
survey.
I thank you for your time and consideration in contributing to this project and if you have any questions about
your rights as a research subject you may contact my advising professor name here in the XXX Office of
Sponsored Research at XXXXXX or email here to discuss them.
Sincerely,
XXXXX
Where are you from?_________
What is your income_______?
Are you interested in studying abroad?
(0)Yes (1)No
We are trying to gauge your level of interest in
studying abroad? Are you very interested, somewhat
interested, not very interested, or not at all
interested?
1. Very interested
2. Somewhat interested
3. Not very interested
4. Not at all interested
Is your religion a factor in your decision making?
1. Yes 0. No
Decision guide for questions
• A. Does the survey question measure some aspect of the research
question?
• B. Does the question provide information needed in conjunction with
some other variable?
– {if no to both A and B drop question, if yes to at least one proceed}
• C. Will most Rs understand the Q and in the same way?
– {if no revise or drop; if yes proceed}
• D. Will most Rs have the information to answer it?
– {if no revise or drop; if yes proceed}
• E. Will most Rs be willing to answer it?
– {if no revise or drop; if yes proceed}
• F Is other information needed to analyze this Q?
– {if no Proceed. If yes proceed only if the other information can be collected}
• G. Should this question be asked of all Rs or a subset?
– {If all proceed, if a subset proceed only after appropriate filter Q}
But First a Sampling Experiment
Each group of students should:
1. Pull 5 candies out of the bag
2. Weigh the candies
3. Write down the weight
4. Put the candies back in the bag!!
5. Pass the scale and bag to your neighbors
6. Silently multiply the weight of the 5 candies
by 20.
Type
Weight (g)
Hersheys
43
Hot tamales
14
Jr mint
18
peanut butter kiss
7
Carmels
8