Transcript Questionnaire Design Part I

Questionnaire Design Issues
Section B
Disclaimer: The examples used are not necessarily good or recommended but
are used for illustrative purposes only.
Copyright 2010, The World Bank Group. All Rights Reserved.
Structure and Format
Layout can:
• Facilitate data collection, transformation and capture
• Differentiate between questions, responses, sections and
clarify skip patterns
• Encourage better response rates
2
Copyright 2010, The World Bank Group. All Rights Reserved.
Structure and Format
1. Abbreviations and punctuation
a.
b.
c.
d.
Spell out abbreviations
Do not use commas or periods inappropriately
Abbreviated wording only with trained interviewers
Underline or bold critical words
2. Response categories
a. Margin-justify pre-coded and self-coded responses
b. Margin-justify code boxes for write-in responses
3. Space
a. Avoid cluttered appearance
b. Provide enough space for write-in responses
c. Avoid too many thick black lines - try thin gray lines
3
Copyright 2010, The World Bank Group. All Rights Reserved.
Structure and Format
4. Answer spaces
a. Should be clearly identifiable for each question
b. Use columnar or tabular format for answer spaces
c. Ensure each response category box is clearly identifiable
5. Interviewer working space
a. Provide space and instructions on the questionnaire for any
calculations to be performed
6. Differentiation
a. Differentiate between the types of answers required and
provide guides to skip patterns whenever possible.
4
Copyright 2010, The World Bank Group. All Rights Reserved.
Structure and Format
7. Color
a.
b.
c.
Drop-out colors for image data capture
Differentiate between sections and clarify skip patterns
Colorful graphics can help motivate respondents to complete
self-enumerated questionnaires
8. Question numbering
a.
b.
Each question should be numbered sequentially (1, 2, 3)
Each section should be numbered sequentially (I, II, III, A, B,
C…)
5
Copyright 2010, The World Bank Group. All Rights Reserved.
Structure and Format
9. Instructions
a. Should be placed (in order of preference):
–
–
–
–
Above the question
At the beginning of a questionnaire section
On the front of the questionnaire
On a separate sheet or in an instructional booklet
b. Instructions should have a differentiated and standardized font,
location or separation (box)
10. Language
a. Various languages may affect spacing and layout
b. Translations should be rigorously checked
11. Consistency
a. Formatting decisions must be consistent throughout.
6
Copyright 2010, The World Bank Group. All Rights Reserved.
Processing Considerations
Keyed data entry
1.
Appropriate number of code boxes per response
2.
One box per character
3.
Answer boxes margin justified
4.
How many responses should be marked for each multiple choice
question?
5.
How will “not stated” be marked?
6.
How will questions not applicable to a particular respondent be
marked?
7
Copyright 2010, The World Bank Group. All Rights Reserved.
Processing Considerations
Scanned data capture
1. Forms must be kept in good condition
2. If multiple pages, must be separated (bar codes, cutters)
3. Paper color, printing clarity, drop out colors
4. Sufficient white space around answer boxes
5. Proper writing instrument used to complete forms
6. More rigorous testing of questionnaires, early enough to make
changes to software, equipment, and/or forms.
7. Compare against a keyed sample
8
Copyright 2010, The World Bank Group. All Rights Reserved.
Questionnaire Quality Issues with
Scanned Data Capture
Questions should be pre-coded as much as possible.

The major areas requiring quality control are:
1.
2.
3.
Paper (color, weight and size)
Printing (color and clarity)
Format (page layout, response position(s), and clear zones)
9
Copyright 2010, The World Bank Group. All Rights Reserved.
Questionnaire Quality Issues with
Scanned Data Capture
Interviewers, and respondents for self-enumeration, must be
trained, instructed in how to complete the form correctly.
1. OMR bubbles
2. OCR characters
10
Copyright 2010, The World Bank Group. All Rights Reserved.
Methods for Drafting Questionnaires
1.
Unstructured Individual Interviewing
2.
Qualitative Group Interviews
3.
Participant Observation
11
Copyright 2010, The World Bank Group. All Rights Reserved.
Unstructured Individual Interviewing
A discussion of the proposed survey topics
Guided by a topic outline
Used to gain insights into questionnaire structure
Time required: 2 - 6 weeks, Cost: Salary and per diems
12
Copyright 2010, The World Bank Group. All Rights Reserved.
Qualitative Group Interviews
Informal discussions of selected topics
Aids in developing conceptual framework and data
specification
Can also be used post survey to interpret the data.
Time: 2 - 4 months, Cost: Salary and per diems
13
Copyright 2010, The World Bank Group. All Rights Reserved.
Participant Observation Research
Anthropological technique for gathering information
Field researchers live with the persons of interest.
Can be used to ensure questionnaire will provide enough
information and help phrase questions
Time: 6 months - 1 year, Cost: Support of researcher and
incidentals
14
Copyright 2010, The World Bank Group. All Rights Reserved.
Procedures for Testing the
Questionnaire Draft
1.
Informal Testing - subjective evaluations of the
questionnaire,
2.
Formal Testing - relies on statistical evaluations
a.
Pilot Studies
b.
Split sample tests
15
Copyright 2010, The World Bank Group. All Rights Reserved.
Informal Testing
A questionnaire field test with a small number of
interviews
Depends on subjective information provided by
interviewers and observers.
Not designed to be evaluated on a rigorous statistical
basis.
Usually done as a first test of the draft or as a final step
to ensure that revisions work well.
16
Copyright 2010, The World Bank Group. All Rights Reserved.
Informal Testing: personnel, cost and time
Coordinator.
Interviewers
Personnel to evaluate the results.
50 to 300 respondents.
Respondents are selected purposively rather than randomly.
Can evaluate entire questionnaire or a portion. More than one
questionnaire evaluated with split-sample testing.
Time: Dependent on questionnaire and sample but operational
aspects around 4 months. Cost: also variable but generally low.
17
Copyright 2010, The World Bank Group. All Rights Reserved.
Formal Testing
Relies on probability sampling and results are evaluated on a
rigorous statistical basis.
1. Pilot study - a prototype of the survey conducted to
observe all of the proposed survey operations working
together.
2. Split-Sample Test - conducted to evaluate two or more
alternate versions of the questionnaire.
18
Copyright 2010, The World Bank Group. All Rights Reserved.
Formal Testing - Pilot Study
Reduces time between final survey and results.
Often little time for a pilot test, evaluation and refinement of
survey tools and procedures.
Large, complicated, or repetitive surveys need a pilot study
19
Copyright 2010, The World Bank Group. All Rights Reserved.
Pilot Study – personnel, time and cost
Probability sample necessitates larger numbers of personnel,
including a sampling statistician, data processing staff and
analysts.
Different respondents should be used for pilot and survey.
Not finished until data analysis is completed.
Evaluation plan must be decided in advance.
Time and costs are variable but are likely greater than for an
informal test.
20
Copyright 2010, The World Bank Group. All Rights Reserved.
Formal Testing - Split-sample
Experimental design incorporated into the data collection
process.
Used as methodological studies to advance knowledge of
questionnaire design and the survey research process.
Personnel requirements are similar for a pilot test but require
more sophisticated statistical and analytical expertise.
Time and cost are similar to pilot test but the time may be
distributed differently for a split-sample with more planning
and less editing and coding.
21
Copyright 2010, The World Bank Group. All Rights Reserved.
Techniques for Evaluating a Questionnaire
1. Investigating Respondents Interpretations of Questions
2. Observation and Monitoring of Interviews
3. Learning from Interviewers
4. Using Record Checks
5. Response Analysis Surveys
22
Copyright 2010, The World Bank Group. All Rights Reserved.
Questionnaire Review Worksheet
Exercise
 Questionnaire organization
 Question wording
 Question grouping
 Skip patterns
 Coding
 Questionnaire length
23
Copyright 2010, The World Bank Group. All Rights Reserved.