Transcript Developing Questionnaires and Structured Interviews

Developing Questionnaires and
Structured Interviews
Alan C. Acock
Why learn how to do these?
• Questionnaires and structured interviews are the
primary way many researchers measure variables.
• Give you a critical eye to evaluate the work of
others.
• You may have a great theory, review of literature,
and research hypotheses. You may have the best
research design possible. You still need to measure
your variables and this is where questionnaires
and structured interview enters the research
process.
How to Develop a Questionnaire
• Guideline 1
• Create a realistic list of all research
questions and hypotheses you wish to
address.
• List all the variables you need for the study.
• This provides a checklist that can be used to
make sure you have items that address each
research question or hypothesis.
Guideline 2
Include demographic, socioeconomic, and
geographic information.
• You will need to control or adjust for
variables such as
•
•
•
•
•
•
•
•
race
gender
age
income
education
marital status
family size
geographical information
List assumptions about participants
•
•
•
•
•
•
Language skills
What is their motivation to participate?
What are red flags for them?
Where might they misrepresent true beliefs?
What do they know?
What can’t they answer?
Questionnaires in other languages
• Know your participants
• Formal Spanish may not be ideal—mixed
language
• Translate to Spanish with appropriate adjustments.
• Back translate to English using different translator.
• Do practice interviews to find what is not clear—
cognitive mapping.
Cognitive Mapping
•
•
•
•
Ask person like those you will interview.
Person reads the question.
What is the question asking?
Person reads the response options and picks
one.
• Why did you pick that option?
Order counts
• Order should be fixed so all participants
answer questions in same order.
• Easy questions at the start.
• Critical questions next in case somebody
quits before the very end of the survey.
• Hard questions that are not absolutely
critical at the end.
• Sequencing can change answers.
Develop skip map
• Computer assisted interviews where a
person’s answer to one item automatically
routes them to the next appropriate item.
• Written questionnaire limit skips.
• Interviewer training is critical for skips.
• Need to pretest with different types of
people who will answer different sets of
questions.
Develop data entry plan
• Computer assisted interviews automatic—produce
dataset ready to analyze.
• Scanning in answers restricts format, but is
accurate and quick.
• Entering by hand is very time consuming and error
prone.
• Step A—put numbers for each question in right margin.
• Step B—enter the data that is in the margins.
Types of questions—closed ended
• A. strongly agree, B. agree, C disagree, D
strongly disagree.
• Check income category. Check years of
education. Check age category.
• Easy to enter data.
• Easy to compare answers.
Types of questions—open ended
•
•
•
•
Participants own voice
Doesn’t impose researchers categorization
Confounded with language/verbal skills
Often skipped or trite answers—How is your
marriage? Answer—OK, I guess.
• Structured questionnaires don’t have follow-ups to
draw out meaningful responses
• Difficult to analyze
Vague/Double Meaning
• Good questions involve one point or dimension.
• Real bad: Has your spouse yelled at you or hit you
in the last month?
• Problematic: How happy is your marriage?
•
•
•
•
•
Happy regarding parenting
Miserable regarding sexual relationship
Great regarding equitable division of chores
Terrible in terms of career goals
Fantastic in terms of companionship
• Be as specific and unidimensional as possible
Avoid value laden wording
• Do you support the Tax Reform solution
that will benefit so many people including
those on a fixed income, the elderly,
children, and the middle class?
• Do you hate gays? Hate is likely to be
avoided even if the person really does not
support gays.
Mitigate threatening questions
• Have you had unsafe sexual intercourse in
the last month?
• Some people have sex without using a
condom. Did you happen to have sexual
intercourse in the last 12 months without
using a condom?
Formats: Grid
Response set—Reverse order--Format is efficient.
Item
1. I often feel depressed
2. I often feel I have no control
3. I control my own future
4. My boss appreciates the work I do
5. It is often difficult for me to sleep
SA
A
D
SD
DK

























Skip directions for self administered
Teenagers
1. Are you male or female?
 Male (go to question 4)
 Female (go to question 2)
2. Have you menstruated?
 Yes (go to question 3)
 No (go to question 4)
3. How much pain, if any, do you have when you menstruate the first time?
 None
 A little
 Quite a bit
 Extreme
4. How many days of school did you miss in the last month?
____ Enter the number of days
Sample Questionnaire
• The file: sample questionnaire.doc contains
a sample of small part of a structure
interview.
Evaluation of this structured
interview—First questions on chores
• Is participant able to give number of hours
each household member spends on each
task (first section)?
• What about multitasking?
• Would men be as good as women at estimating?
• Diaries might be more useful
Evaluation of this structured
interview—Depression scale
• Standard measure; widely used.
• Response set problem—not bother to even
read last couple items—just check 1 or 7 for
all of them.
• Response set gives researchers false sense
of reliability.
• Between item b and c we could say “have
an especially enjoyable experience”
Evaluation of this structured
interview
• Question 8. Is this threatening?
• Does opening statement mitigate the treat?
• Are respondents able to answer this
question?
• Correlations between parent reports about
their children and the children’s reports are
weak.
Evaluation of this structured
interview
• Question 11. Is yes or know what we need?
• We lose possible variation. If we can get
meaningful responses we need to maximize
variation.
• With some groups, maximizing variation may be
counter productive.
• Can distinguish between yes and no
• Can make meaningful/reliable distinctions between
other response options.