Transcript Document

NEWMOA Workshop on Survey
Instrument Design
Linda Piccinino
The Importance of Questionnaire Design
• “Planning the questionnaire is one of the most critical
stages in the survey development process.”
–
American Statistical Association (1997)
Purposes of Surveys for the EPA
• Measure results of pollution prevention and compliance
assistance efforts
• Show level of compliance with regulations
• Determine assistance needs
• Demonstrate results in a quantitative manner
Types of Survey Mode
• Mail (letter, email)
• Telephone
• In-person
Group Exercise 1
Learning How to Define Survey Content and Scope
– Writing survey questions to address research questions
Types of Survey Administration
• Group interviews
• Structured interviews
• Semi-structured interviews
• Self-administered questionnaires
Other Factors to Consider
• Questionnaire length
• Questionnaire format and appearance, physical size
• Question order
Question Wording and Structure
Comprehension and Clarification
• The question should define words in the way the
researcher intended
• The question wording and structure should be clear about
what event is being asked about
Comprehension and Clarification
Example:
– Were you robbed during the last six months?
Comprehension and Clarification
Revision:
• Was something belonging to you stolen, such as –
a) Things that you carry, like luggage, a wallet, purse, briefcase,
book
b) Clothing, jewelry or calculator
c) Bicycle or sports equipment
d) Things in your home – like a TV, stereo or tools
e) Things from a vehicle, such as a package, groceries, camera
OR
f) Did anyone ATTEMPT to steal anything belonging to you?
Source: National Crime Victim Survey Questionnaire
Comprehension and Clarification
Examples:
– How many hours did you
work LAST WEEK at all
jobs?
– Did you lose any time or
take any time off LAST
WEEK for any reason such
as illness, holiday or slack
work?
– Did you you work any
overtime or at more than
one job LAST WEEK?
–
Source: Schaeffer, 1993
Problems:
– Respondents include
reports about usual hours
worked
– Questions about hours lost
and extra hours follow
request for total hours
Comprehension and Clarification
Revision:
– How many hours do you USUALLY work at your (main)
job?
– LAST WEEK, did you lose or take off any hours from
(work/your main job) for any reason such as illness,
vacation, holiday, labor dispute or layoff?
– LAST WEEK did you work any overtime or extra hours (at
your main job) that you don’t usually work?
– So, for LAST WEEK, how many hours did you ACTUALLY
work at your (main job/other jobs)?
Specifying the Reference Period
• The reference period should be as clear and concise as
possible.
• Memory aids such as a calendar or markers such as
important dates (holidays, birthdays, etc.) might help
respondents to think about the reference period you are
asking about.
Specifying the Reference Period
Example:
Possible Solution:
• In the last 7 days, not counting
• I am going to ask a few
today, how many times did you
smoke at work?
• Source: Belson, 1989
questions about things you
might have done while at work
LAST WEEK. By last week I
mean the week beginning on
Sunday June 4, 2006, and
ending on Saturday June 10,
2006.
– In the LAST WEEK, how
many times did you take a
cigarette break during work
hours?
Tips
• Look at each question and ask yourself “How will this
information be used?”
• Put yourself in the respondent’s place – how would you
answer the question?
Stages of the Survey Response
Process
The Survey Response Process
Four steps in generating a response:
1) Comprehension
2) Retrieval of information
3) Response formation
4) Response mapping and reporting
The Survey Response Process
Extent to which each stage in the response process is
difficult depends on both the question and the respondent.
– How many bowls of cereal did you consume in the past
week? (retrieval process)
– Have you gained 5 or more pounds in the past year?
(judgment process)
Respondent Cognitive Tasks
Comprehension:
• What does the respondent think the question is asking?
• What do specific words or phrases mean?
Retrieval:
• What type of information does the respondent need to recall? Does
the respondent possess the information required?
• What are some difficulties associated with the respondent retrieving
information? What types of strategies does he/she use (e.g., counting
events by recalling each one individually or providing an estimate
based on usual behavior)
Source: Willis, G. Cognitive Interviewing.
Respondent Cognitive Tasks
Response Formation:
•Does the respondent have the motivation to use the retrieved
information to form a response?
•Does response formation pose particularly difficult tasks for the
respondent?
•Does the respondent tell the truth? Does he/she falsify his/her
response? Does the question raise privacy concerns?
Response Mapping:
•Can the respondent match his/her answer to the response
options provided?
Tips
• We focused on comprehension, but respondents can have
difficulties at any stage of the process
• If you know the stage that is causing difficulty, you have a
better chance of fixing the problem
Response Categories
• Frequencies: Absolute and Relative
• Rating Scales
• Dates
• Open and Closed Questions
Frequencies
• How many boats do you shrink wrap a year?
• Source: New Hampshire Dept. of Environmental Services Marina Questionnaire
• How often do you dispose of unwanted mercury from your
dental office?
Rating Scales
• Bipolar
• Interval
• Categorical
Dates
• Exact dates
• Ranges: Approximate dates, durations and estimates
Open and Closed Questions
• Open-ended:
– How did you learn about this program?
• Closed:
– What is your RCRA hazardous waste generator status?
• Large Quantity Generator
• Small Quantity Generator
• Conditionally Exempt Small Quantity Generator
• Don’t know
Common
Problems with Survey Questions
Problems with Survey Questions
Common Question Problems:
•Complicated questions
•Unfamiliar or technical terms
•Ambiguous concepts or words
Question Problems: Complicated Questions
Example:
Over the last month, how many times per month, week, or day did
you drink 100% fruit juice such as orange, apple, grape or grapefruit
juice? Do not count fruit drinks like Kool-Aid, lemonade, HI-C,
cranberry juice drink, Tang, and Twister. Include juice you drank at
all mealtimes and between meals. (never, 1-3 times last month, 1-2
times per week, 3-4 times per week, 1 time per day, 2 times per day,
3 times per day, 4 times per day, 5 or more times per day).
Source: NCI All-Day Screener
Question Problems: Unfamiliar Terms
Example:
Do you think that a compliance assessment is a
satisfactory tool to measure results of
compliance assistance efforts?
Example:
I have a sweet tooth. (strongly disagree,
disagree, neutral/NA, agree, strongly agree)
Source: Eating Behavior Patterns Questionnaire (2003)
Question Problems: Ambiguous Concepts or Words
Examples:
• In the past 3 months, how often did you eat a
vegetarian dinner? (usually/always,
sometimes, rarely, never)
Source: Fat and Fiber Behavior Questionnaire (1997)
•
Do you now…
• Spend half an hour or more in moderate or
strenuous physical activity at least 3 times a
week? (Yes/No)
Source: NMES - SAQ
Question Problems
•Question assumptions not applicable to respondent
•Loaded questions
•Double-barreled questions
Common Question Problems: Assumptions
Examples:
About how many gallons of “perc” do you use in one
year?
Source: EPA Dry Cleaners Survey (2004)
Do you have a Non-Contact Cooling Water Discharge
of 5,000 or more gallons per day?(yes/no/not
applicable/don’t know)
Source: Garment Care Business Information (2004)
How long have you been limiting the amount of fat in
your diet? (less than 30 days, 1-6 months, 7-12
months, over a year)
Source: National Food Stamp Program (1996)
Common Question Problems: Loaded Questions
Examples:
Would you say you are now complying with EPA
standards?
Are you using environmentally preferable
products?
Common Question Problems:
Double-Barreled
Examples:
What did you do to lose weight and maintain your
weight loss?
Source: California Dietary Practices Survey (2001)
During the winterization process, did you store
engines “wet” and “dry?”
Question Problems
•Overlapping response categories
•Recall difficulty
•Sensitive questions
Common Question Problems:
Overlapping Response Options
Examples:
How many boats do you sand per year?
Fewer than 5 boats
5 to 10 boats
10 to 20 boats
More than 20 boats
Which of the following main dishes do you think is
lowest in fat? (fish sticks, pork ribs, barbeque ribs,
chicken)
Source: National Food Stamp Program Survey (1996)
Common Question Problems: Recall Difficulty
Examples:
How many newspapers did you recycle in the past
month?
How many glass bottles, cans of paint, cardboard
boxes or batteries did you dispose of since you
started working at the hospital maintenance
division?
Common Question Problems:
Sensitive Questions
Examples:
If you are not using EPP (environmentally
preferable) or bio-based products, why not?
Source: Connecticut Hospital Environmental Roundtable Follow-up Survey (2005)
Do you maintain your weight at recommended
levels? (Y,N)
Source: Health Behaviors Module (2000)
Practice Example
Identify problems with the following survey
question:
1) Would you describe the amount of butter or
margarine you usually spread on breads and
muffins as ____? (none, light, moderate,
generous)
Source: Diet and Health Knowledge Survey (DHKS) 1994-1996
Group Exercise 2
• Critique of Existing Survey Questions
– Based on what has been learned so far, identify problems
and offer solutions
– Revise questions from Exercise 1
Tips
• Look at each question – Can it be simplified?
• Reconsider words that are not recognized by your
spellchecker
Pretesting
Pretesting Methods
• Methods
– Low cost options
• Consequences of not pretesting
Debriefing
• Final questioning that occurs after the respondent has
completed the questionnaire and reviewed all materials
• Interviewer may ask open-ended question to probe further
into certain comments or observations the respondent
made during the interview
• Respondents may also be asked to provide an
assessment or overall critique of the instrument
Sample Debriefing Questions
• Think about the questionnaire as a whole. Does anything in
particular come to mind that you want to comment on? What are
your general feelings about it? Did you feel you were prepared for
the types of questions asked?
• In general, did you have difficulty remembering what you did?
Sample Debriefing Questions
• I noticed that you chuckled at several points. Can you tell me what
caused you to do so?
• Were there any specific questions that you had trouble answering?
Why?
• Were there any questions that you felt uncomfortable
answering? Why?
• What questions do you think most needed to be revised? Do you
have any suggestions for how they might be better written?
Group Exercise 3
• Pretesting of Questionnaire
• Debriefing
Group Exercise 4
• Revise and retest questions from Exercise 3
• Review of entire questionnaire development process –
what are the key points and problems associated with
each step?
Tips
• Always do some pretesting with some members of the
target population
• Test any changes that you make
Suggested Readings
Converse, JM and S Presser. 1986. Handcrafting the Standardized
Questionnaire. Thousand Oaks, CA: Sage Publications, Inc.
EPA. 2002. Guide for Measuring Compliance Assistance Outcomes.
Revised. Section IV. Washington, DC: EPA, pp. 23-50.
Groves, RM, et al. 2004. Survey Methodology. Hoboken, NJ: John
Wiley & Sons, Inc.
Payne, S. 1980. The Art of Asking Questions. Princeton, NJ:
Princeton University Press.
Sudman, S and NM Bradburn. 1982. Asking Questions. A Practical
Guide to Questionnaire Design. San Francisco, CA: Jossey-Bass
Publishers, Inc.
Willis, GB. 2004. Cognitive Interviewing. A Tool for Improving