Transcript Effective Questionnaire Design

Effective
Questionnaire
Design
Craig Parylo & Annette Lee
What is a questionnaire?
An effective questionnaire…
Asks relevant questions
Question 1
• Questions that will provide the answers to the
research topic in question
Food
Gets valid responses

• Responses given reflect the respondent’s actual
opinions
Gets a representative response
• Makes it more reliable to infer the findings to
wider population
Question 2
Question 3
Introduction
Anyone can design a questionnaire.
An effective questionnaire requires planning
and an appreciation of the psychology of
respondents:
Filling in a questionnaire is a complex
process.
• The job of a questionnaire designer is to
make it easy to respond.
Complex cognitive process
1. Identify
subject
Food
4. Decodes
5. Collects info
Question 1
6. Share?
2. Phrase as
question
3. Reads
7. Encode
9. Respond
8. Option
 No
I prefer tea
Exercise: responding to questions
1.
2.
3.
4.
5.
6.
What is the colour of your car?
What do you usually have for breakfast?
What did you have for breakfast last Friday?
Have you ever not left a tip at a restaurant?
What is your sexuality?
What is the biggest regret in your life?
Choosing the right approach
Photo by Ronit Geller (http://www.sxc.hu/profile/Rogel)
Types of questionnaire
No method consistently outperforms others
Considerations
Your approach should take
consideration of:
• Your topic
• Who you want survey
• The type of information
you want
• Your budget
• How quickly you need
the information
Sample size vs Depth of info
• Better response rates from interviewer-led
surveys than from self-response surveys.
• Self-response surveys are usually cheaper.
• Interviewer-led surveys can produce richer
information.
• Self-response surveys can reach more people.
Don’t confuse quantity for quality.
Children
Is it pitched at
the right age
group?
Timing and convenience
• Will it be completed right after the event, or
sometime after?
• Will it be completed in front of other people
or in private?
• Many people are poor at estimating numerical
info about their lives.
• The more common the event the shorter a
person’s window of recall is.
Question types
Question types
• Open / closed
• Single / multiple
response
• Attitudinal / opinion
questions
• Filtering / routing
Attitude/opinion
• Can be difficult to get the phrasing of the
question right so it means the same thing to
all people.
• Can be tempting to put lots of explanation
into the question – but sometimes this just
confuses things further.
• Use clear, simple English/ language and avoid
being too general.
• Try not to ask ‘Iceberg’ questions.
Mid-point / neutral opinions
Use mid-points in questions on attitude.
Strongly
disagree





Strongly
agree
• Historically controversial as it was thought
that respondents chose a neutral option to
avoid thinking about a question.
• Now shown that including a mid-point
increases reliability and validity
‘How important?’
Following your visit to the clinic today, please rate the
following – first based on your experience and second
how important this is to you?
1
2
3
4
5
6
7
Waiting time
Very poor







Excellent
How important is the length of
waiting time to you?
Not
important







Very
important
Quality of magazines in waiting
room
Very poor







Excellent
How important is the quality of
magazines in waiting room to
you?
Not
important







Very
important
17
Common mistakes
What might
be wrong
with this
question?
Need to cover
all options
Common mistakes
What might
be wrong
with this
question?
Options
should be
mutually
exclusive.
What is your age group?
0-18 18-30 30-45 45+ 
0-18 19-30 31-45 46+

Common mistakes
What might
be wrong
with this
question?
Options
should be
balanced
What did you think about the
waiting time?
Excellent Very Good Average Poor
good
‘Good’ responses
‘Bad’
response
Common mistakes
What might
be wrong
with this
question?
How would you rate the
appointment booking service
using the:
Good Average Poor
Website
Use opt-out
responses
appropriately
Phone
Reception
N/A
Increasing clarity
22
Question wording
• Keep questions as short as possible
– Max 16 words.
• Consider providing medium length (30 word)
introductions to groups of related questions
(context is vital!)
• Use open-ended questions sparingly
23
Double questions
If a question
can be broken
down into
parts then it
should be?
Double negatives
Do you think that it is an unwise policy to
continue sending food aid to Africa.
☐ Yes
☐ No
• If the respondent has to tick ‘No’ to give a
positive response then this can be considered
a double negative.
• Try and phrase questions with a positive slant
so that the positive response is ‘Yes’ rather
than ‘No’.
25
Style, appearance and layout
Design
• Use booklet format with double-sided printing
– A4 folded to A5, or, A3 folded to A4
• Front cover:
– The title of the questionnaire
– Identity of the organisation carrying it out
– Clearly explain the purpose of the questionnaire
– Keep graphics neutral
Aesthetics
• Arial font, size 12 point or larger
• Questionnaire length
– Shorter is generally better
– Also keep plenty of ‘whitespace’
• Be consistent in the use of colour, shapes and
location in order to guide the respondent
through the questionnaire
Other elements to remember
• Make it look professional.
• Give an estimated completion time.
• Put the deadline for return at the beginning
and end of the questionnaire.
• Provide a return address – on the
questionnaire.
• Say ‘Thank you’.
Ordering your questionnaire
• Place easy, non-challenging questions first
• Key filter questions should go in early.
• Use a funnel structure
– Go from basic to more in depth.
– General questions before specific questions.
– Go from the least sensitive to the most sensitive
questions.
– But – try and put important questions in before
the end.
Ordering your questions
• Ask for suggestions for improvement at the
end.
• Check for leading questions.
• Add prompts where required.
• Order of questions can impact heavily on the
amount of time it takes for the respondent to
complete the questionnaire.
Filtering
• Filtering (or routing) helps identify which
respondents should answer a detailed set of
questions about a certain aspect of the topic.
• Can be difficult to achieve – should be VERY
CLEARLY signposted
Filtering Example
About our new website
1. Did you visit our website today?
Yes
No
✓
(If yes please answer questions 2-4, if no go to question 5)
2. Did you like the new look?
3. Did you find it easy to navigate?
4. Would you recommend the site to others?
5. Did you pick up a copy of our latest leaflet?
✓
✓
✓
✓
Filtering Example
About our new website
1. Did you visit our website today?
1a. Did you like the new look?
1b. Did you find it easy to
navigate?
1c. Would you recommend the site
to others?
2. Did you pick up a copy of our latest
leaflet?
Yes
No
Please continue
with Q1a
Please
go to Q2
Tips for increasing response rate
Photos by Sarah Williams & Davide Guglielmo
(http://www.sxc.hu/profile/MeHere) (http://www.sxc.hu/profile/brokenarts)
Improving the response rate (1)
• Shorter questionnaires
• Good communication
– Contact the respondents beforehand
– Follow-up contact with non-responders
• Monetary incentives
• Personal touch
– Handwritten address
– Including respondent’s name in cover letter
Improving the response rate (2)
• Method of delivery
– Include a stamped return envelope
– Send by first class or recorded delivery
– University sponsorship can add credibility
• Assure confidentiality
Key take-home message
Pilot your tool
• This is the MOST IMPORTANT part.
• If a range of respondents from different
backgrounds will be using the questionnaire
make sure you use as many of these different
groups as possible in the trial.
• Go back to your objectives – does the data help
you fulfil your objectives and measure against
the standards you set?
39