Transcript Overview of Questionnaire Design Issues

Overview of Questionnaire Design
Issues
Content
Question Wording
Question Sequencing and Flow
Ease of Questionnaire Administration
Other Design Issues
1
Content
Decide what to include and exclude to meet your survey
objectives.





Analysts and data users should be consulted as early as possible
in the process of specifying the subject matter.
All aspects of the survey subject must be investigated to ensure
that robust information is gathered. (childcare).
You also need to be sure that only questions are asked which
respondents are knowledgeable enough to answer (floor space in
square meters).
Questions about past specific events may need to be
accompanied by a memory trigger, such as a calendar of events.
Designers must carefully sequence sensitive questions.
2
Question Wording
Questions must have the same meaning and provide comparable
data from every unit in the sample.





Vocabulary used must be familiar to the average respondent,
without regional or cultural variations in meaning.
The designer should take into consideration the variety of “frames
of reference” existing among respondents (neighborhood).
Response categories offered, type (open or closed), number, and
order.
Length of question - introduction or explanation, some respondents
may feel this gives them time to think while others become
inattentive or confused.
Different types of respondents may respond differently introducing
bias.
3
Ease of Questionnaire
Administration
The questionnaire must be easy for the interviewer and
respondent to use.
Placement of instructions
Reference to previous information, flipping back and forth
The harder it is for the interviewer to determine the flow of
the interview, the more chances for introducing interviewer
error, item nonresponse, and respondent frustration.
4
Other Design Issues
Procedural decisions and questionnaire format are also
important considerations.
Interview time may be limited due to cost or respondent
burden, forcing reduction in questions asked.
The survey respondent should be considered, questions for
each person may be worded differently than for those asked of
a responsible respondent for a household.
The designer should also keep in mind the method of data
collection, the unit of analysis, and how many times the
sample unit will be interviewed.
Format can affect how well interviewers/respondents are able
to follow instructions and answer questions.
5
Questionnaire Design Preparation
Method of data collection
1.
Self-enumeration
2.
Interviews
Method of data capture
1.
Keyed data entry
2.
Scanning (OMR and/or OCR)
6
What is a Questionnaire?
A questionnaire is a tool designed to transfer
specific data from the respondent to the
sponsor/user/analyst.
Questionnaires must be designed to yield
information of the highest quality possible with
special emphasis on relevance, timeliness and
accuracy.
An effective questionnaire is one that is
economical in human resources, produces data
which is wanted and is processable.
7
Major Components of the
Questionnaire:
Introduction
1.
2.
3.
4.
Identify the survey and who is responsible for
the survey.
Identify any laws that require participation in the
survey and any assurances of confidentiality
Provide a statement of purpose which explains
how the data are to be used and how it will
benefit the respondent.
Provide contact information for questions and
assistance
8
Major Components of the
Questionnaire:
Administrative data
1.
2.
3.
Survey identification codes, geographical codes
and a serial number for the questionnaire
Name of the respondent, address or location of
the housing unit, establishment, etc. for which
the data is being recorded
Certification of the enumerator and sometimes
the enumerator’s supervisor/crew leader
9
Major Components of the
Questionnaire:
Main body
1.
2.
3.
Contains substantive questions on the subjects of the
questionnaire.
OPENING QUESTIONS are used to put the respondent
at ease. These questions tend to be factual questions on
demographic data applicable to all respondents.
CLASSIFICATION QUESTIONS are the questions that
allow respondents to be grouped into different categories
of interest based on their answers.
10
Types of Questionnaires:
Verbatim:
•
The enumerator reads the questions off the
questionnaire word for word.
“What is …’s marital status?”
If single ask: “Has … ever been married?”
Schedule:
•
The questionnaire gives topics covered and response
categories without specific question wording.
Marital Status:
11
Method of Administration:
Self-enumerated:
•
Clear and detailed information on how to complete the
questionnaire is required


•
Appropriate writing instrument to use (pen/pencil)
Completion of responses (written, check-mark, oval)
Include detailed instructions on where to leave the
questionnaire and alternative actions if unable to comply with
the directions
Interview:
•
•
Have the advantage of training enumerators
Still provide clear guidance on completion of the
questionnaire but can use abbreviated instructions
12
Types of Questions
Open-ended questions
•
produces an unstructured response that is recorded
on the questionnaire in the words of the respondent
Example: What have you done to look for work during the
last 4 weeks? _____________________________
13
Types of Questions
Open-ended questions are good:







for openers or change of subject
where not enough is known of the likely answers to form a
response list
where responses are technical or complex
where respondent cooperation would be better than for a
close-ended question
as a source of hypotheses and of wording for close-ended
questions
to get more detail or elaboration of an answer
in avoiding leading respondents
14
Types of Questions
Closed-ended questions
•
requires respondent to choose from a small number
of choices, usually printed on the questionnaire
Is … male or female?
1 Male
2 Female
15
Types of Questions
Closed-ended questions are good because:







all respondents answer in the same frame of reference from
the same choice of answers
variations in the respondent’s articulateness are irrelevant
the response categories are known before enumeration which
simplifies data capture and analysis
it’s easier for interviewers and respondents to complete
it’s quicker and cheaper to administer
the sensitivity of some topics can be reduced
more complete information can be obtained through lists that
can be read to the respondent
16
Types of Questions
Filter or Screening questions
•
A two (or more) part question that is used to direct
respondents to relevant questions. The answer to the
filtering question determines which of two different
questions a respondent next receives.
Were you engaged in work for pay or profit the week
preceding the survey?
If yes, what was the name of your employer?
17
Skip Pattern
The logical organization of an interview so
questions are asked only of those who fit
certain criteria. Other respondents are directed
elsewhere in the questionnaire depending on
the criteria they meet.
•
Did you work last week? (yes/no)

If yes, what type of work did you do last week?

If no, what was the reason you did not work last week?
18
Types of Questions
Read-in question
•
Includes the answer categories as part of the
question. In an interview situation, the response
categories for these questions would be read to the
respondent.
At your job, are you:
1.
2.
3.
4.
5.
Employer
Employee for pay
Self-employed or own account worker
Apprentice
Unpaid family worker
19
Types of Questions
Sensitive questions
•
Questions which respondents may be reluctant to
answer for various reasons:

Fear of consequences (legal or political action)

Fear that their honest response might not be socially
acceptable

Question is considered too personal

Question is associated with painful memories
20
Translating Concepts into Questions
1.
2.
3.
Applicable - Ask only for information that cannot be obtained
elsewhere and are directly related to the objective of the
survey.
Specific - Avoid questions which are open to interpretation or
are too vague (sometimes, mostly, rarely). Always use the
simplest terms possible to phrase the question.
Accurate - Do not ask for information or precision that
respondent’s don’t know or are unwilling to provide. Avoid
bias created by “wishful thinking”, questions on past behavior
are more reliable than those on future behavior.
21
Translating Concepts into Questions
4.
5.
6.
Precise - Ask only one question at a time, avoid conjunctions
like “and” and “or”. Also avoid constructing negative
questions as they lead to confusion.
Neutral - Avoid emotionally charged language which may
offend respondents or cause them to answer in a certain way.
Unassuming - Questions should not assume that a certain
condition exists, unless it has been determined in a previous
question.
22
Methods of Recording Responses
Type of question:
•
•
•
•
Closed-ended questions are pre-coded
Open-ended questions are write-in answers
Questions asking for numeric information are
self-coded (age, income)
Some open-ended or administrative information
may be coded by the interviewer in the field.
23
Methods of Recording Responses
Data capture:
•
•
•
Manual coding required (timeliness, conversion
error)
Method of data capture (keyed data entry or
scanning)
Ease of keying, scanning in order to capture data
accurately and quickly
24
Structure and Format
Layout
•
Facilitate data collection, transformation and
capture
•
Differentiate between questions, responses,
sections and clarify skip patterns
•
Encourage better response rates
25
Structure and Format
1.
Abbreviations and punctuation
a.
b.
c.
d.
2.
Response categories
a.
b.
3.
Spell out abbreviations
Do not use commas or periods inappropriately
Abbreviated wording only with trained interviewers
Underline or bold critical words
Margin-justify pre-coded and self-coded responses
Margin-justify code boxes for write-in responses
Space
a.
b.
c.
Avoid cluttered appearance
Provide enough space for write-in responses
Avoid too many thick black lines - try thin gray lines
26
Structure and Format
4.
Answer spaces
a.
b.
c.
5.
Interviewer working space
a.
6.
Should be clearly identifiable for each question
Use columnar or tabular format for answer spaces to
avoid confusion
Ensure each response category box is clearly identifiable
(dotted lines)
Provide space and instructions on the questionnaire for
any calculations to be performed
Differentiation
a.
Differentiate between the types of answers required and
provide guides to skip patterns whenever possible.
27
Structure and Format
7.
Color
a.
b.
c.
8.
Drop-out colors for image data capture
Differentiate between sections and clarify skip patterns
Colorful graphics can help motivate respondents to
complete self-enumerated questionnaires
Question numbering
a.
b.
Each question should be numbered sequentially (1, 2, 3)
Each section should be numbered sequentially (I, II, III,
A, B, C…)
28
Structure and Format
9.
Instructions
a.
Should be placed (in order of preference):




b.
10.
Instructions should have a differentiated and standardized
font, location or separation (box)
Language
a.
b.
11.
Above the question
At the beginning of a questionnaire section
On the front of the questionnaire
On a separate sheet or in an instructional booklet (referred to on the
questionnaire)
Various languages may affect spacing and layout
Translations should be rigorously checked
Consistency
a.
Formatting decisions must be consistent throughout.
29
Grouping and Ordering of Questions
Facilitates gathering information from
respondents
Progresses logically from topic to topic
Only contains relevant questions
Interviewer can follow the questioning
sequence
30
Grouping Questions
1. Divide questionnaire into sections, each dealing
with a particular topic or subject.
2. Group together questions relating to the same time
interval, reference period, or frame of reference.
3. Group questions by type of respondent required
(proxy, individual)
4. Group questions by the unit of enumeration
(households, house members, enterprises, holdings)
31
Ordering Questions
a. After grouping, question and questionnaire
section order should be reviewed by
designers and users.
b. Prioritize questionnaire sections and
questions.
c. A questionnaire flowchart is useful for
deciding question order and can be helpful
during the development of edit specifications
and data entry programs.
32
Questionnaire Flowchart Sample
Yes
Q. 18a
Q. 8b
No
Yes
Q. 9-11a
Q. 11b-c
No
<5 yrs. old
Q. 12-15a
Yes & No,
outside US
No,
in US
Q. 33
Yes
Q. 16-18
No
Q. 15b
Yes
Q. 19a
Q. 19b
Yes
Q. 19c
No
No
Q. 20a
Burmese Questionnaire Flowchart
<5
Q. 1-8
Stop
5+>
< 10
Q. 9-11
Stop
10+>
Males & never
married women
Q. 12-16
Ever married
women
Q. 17-19
Stop
Processing Considerations
Keyed data entry
1.
2.
3.
4.
5.
6.
Appropriate number of code boxes per response
One box per character
Answer boxes margin justified
How many responses should be marked for each
multiple choice question?
How will “not stated” be marked?
How will questions not applicable to a particular
respondent be marked?
35
Processing Considerations
Scanned data capture
1.
2.
3.
4.
5.
6.
7.
Forms must be kept in good condition
If multiple pages, must be separated (bar codes,
cutters)
Paper color, printing clarity, drop out colors
Sufficient white space around answer boxes
Proper writing instrument used to complete forms
More rigorous testing of questionnaires, early enough
to make changes to software, equipment, and/or forms.
Compare against a keyed sample
36
Questionnaire Quality Issues with
Scanned Data Capture
Questionnaires still must be easy for interviewers and
respondents to use but must also be very machine
readable to maximize the advantages of speed and
reduced human error with scanned data capture.
Questions should be pre-coded as much as possible.
 The major areas requiring quality control are:
1.
2.
3.
Paper (color, weight and size)
Printing (color and clarity)
Format (page layout, response position(s), and clear zones)
37
Questionnaire Quality Issues with
Scanned Data Capture
Interviewers, and respondents for self-enumeration, must
be trained, instructed in how to complete the form
correctly.
1.
2.
OMR bubbles should be filled in completely with no stray
marks, using the appropriate writing instrument.
OCR characters should be printed in upper case in individual
character, spaced boxes. A standard should be developed for
numbers and characters and alphanumeric fields should
probably be avoided. Cursive alphanumeric fields can be
recognized but this usually involves teaching the scanner to
read local handwriting.
38
Approaches to Developing
Questionnaires
Some generally accepted rules exist that can guide the
questionnaire designer in constructing an initial
draft of a questionnaire.
The development of any particular questionnaire is
unique and refinement is necessary to ensure that a
questionnaire used in the field will produce
sufficiently accurate results.
39
Approaches to Developing
Questionnaires
There are a series of tools and tests which are
useful in the initial drafting and subsequent
refinement of a survey questionnaire.
 The maximum effort needs to be dedicated
to questionnaire development because once a
questionnaire is in use, problems are costly
or impossible to correct.
40
Tools for Developing Questionnaires
1. Unstructured Individual
Interviewing
2. Qualitative Group Interviews
3. Participant Observation
41
Unstructured Individual
Interviewing
A discussion of the proposed survey topics between
individual members of the group to be surveyed and
the questionnaire designer.
The discussion is guided by a topic outline rather than
a set of specific questions.
This technique is primarily used to gain insights into
the best way to structure the questionnaire.
Time required: 2 - 6 weeks, Cost: Salary and per
diems
42
Qualitative Group Interviews
Informal discussions of selected topics between
participants chosen from the population of interest and
someone who is knowledgeable about group
interviewing techniques and the purpose of the survey.
The information gathered can aid in developing the
conceptual framework and data specification for a
statistical survey and evaluating draft questionnaires.
Qualitative group interviews can also be used after a
survey is completed to help analysts interpret the data.
Time: 2 - 4 months, Cost: Salary and per diems
43
Participant Observation Research
Anthropological technique for gathering information about
persons with language, values, or experiences very different
from the designer, or about whom very little is known.
Field researchers, knowledgeable in the local language, live
with the persons of interest. They conduct key informant and
unstructured interviews.
Information obtained can be used to ensure that the content of
the questionnaire will provide enough information to satisfy the
survey’s objectives and help phrase questions that can be
understood by all respondents.
Time: 6 months - 1 year, Cost: Support of researcher and
incidentals
44
Procedures for Testing the Questionnaire
Draft
1. Informal Testing - subjective evaluations of the
questionnaire, less control is necessary in choosing
the sample and conducting interviews.
2. Formal Testing - relies on statistical evaluations,
this report describes two variations:
a.
b.
Pilot Studies
Split sample tests
45
Informal Testing
A questionnaire field test involving a small number of
interviews in the kind of setting chosen for the final
survey (home, work, etc.).
The detection and correction of errors or weaknesses in
the questionnaire depends mainly on subjective
information provided by interviewers and observers.
This test is not designed to be evaluated on a rigorous
statistical basis.
Usually done as a first test of the draft or as a final step to
ensure that revisions work well.
46
Informal Testing - personnel
Requires a coordinator to select the test site, sample, select and train
interviewers, structure a system to receive feedback and set up an
evaluation plan. Experience or knowledge about data collection is an
essential qualification.
Interviewers with varying experience and skill levels are desirable,
however all interviewers should know how to probe to obtain
information that will be used to refine the questionnaire. Observers
may also need to be selected and trained.
Knowledgeable personnel are required to carry out the evaluation of
the results. Skills include the ability to recognize problems in an
interview, during review of completed questionnaires or tabulations
and the implications of the test results for the questionnaire design.
47
Informal Testing - cost and time
Usually, adequate subjective information can be obtained from 50 to
300 respondents.
Respondents are generally selected purposively rather than
randomly to achieve the desired objectives of the test. The sample
may include either a broad range of sub-populations in the test or
questions may be directed to a specific sub-population.
Informal tests can be used to evaluate the entire questionnaire or
only a portion. More than one type of questionnaire may also be
evaluated (split-sample testing).
Time: variable depending on particulars of the questionnaire and
sample, however operational aspects may be around 4 months.
Cost: also variable but the relative speed of the test keeps them low.
48
Formal Testing
Field test which relies on probability sampling for respondent
selection and for which results can be evaluated on a rigorous
statistical basis.
1.
2.
Pilot study - a prototype of the survey conducted to observe all of
the proposed survey operations working together. It calls for
developing a design which duplicates the final proposed survey
from beginning to end, including plans for data processing and
analysis.
Split-Sample Test - conducted specifically to determine the
“best” of two or more alternate versions of the questionnaire, or
almost any aspect of survey operations.
49
Formal Testing - Pilot Study
Advantages of a pilot study include the reduction of time between
the final survey and availability of results, since the post-collection
procedures and analysis plan have been tested and refined.
Disadvantages are that there is often little time scheduled between
the formulation of the survey plan and the data collection effort for a
pilot test, evaluation and refinement of survey tools and procedures.
A pilot study is also costly as it encompasses all survey procedures.
It is the large, complicated, or repetitive survey that warrants the
pilot study investment, or situations where the efficient operation and
flow of each phase is crucial to the success and cost of the operation
50
Pilot Study - personnel
The size and use of a probability sample in a pilot study
necessitates the involvement of larger numbers of
personnel with a variety of skills. In addition to a
coordinator, field interviewers and supervisors, a sampling
statistician is required. Data processing expertise and
management and data analysts are also needed.
When selecting respondents the possibility of the same
respondents falling into both the pilot and the survey
sample should be avoided because of conditioning effects
and respondent burden (except for a census). Overlap can
be avoided if both samples are chosen at the same time.
51
Pilot Study - time and cost
The pilot study is not finished until the last phase planned for the
final survey is completed - data analysis. The more complete a pilot
study can be, the greater the potential for discovering and correcting
individual errors and deficient chose of both survey and analytical
methods.
Since pilot study analysis is overwhelming and must be
accomplished quickly, it should be decided in advance (evaluation
plan) what key indicators will be requested and what feedback
system to employ.
Time and costs are variable but are likely greater than for an
informal test.
52
Formal Testing - Split-sample
The main feature that distinguishes split sample testing from a pilot
test is the experimental design incorporated into the data collection
process.
Split samples are used as methodological studies designed to
advance basic knowledge of questionnaire design and the survey
research process. Because of interest in assessing trends over time
and comparing results across surveys, they also have an important
transitional objective in repetitive or recurrent surveys.
Personnel requirements are similar for a pilot test but require more
sophisticated statistical and analytical expertise.
Time and cost are similar to pilot test but the time may be
distributed differently for a split-sample with more planning and less
editing and coding.
53
Techniques for Evaluating the
Questionnaire Draft
1.
Investigating Respondents Interpretations of Questions
a.
2.
Observation and Monitoring of Interviews
a.
3.
Determines respondent’s frame of reference directly from respondent
Provides input for development of questionnaires or those used in
repetitive surveys
Learning from Interviewers
a.
b.
Interviewer debriefing
Structured post-interview evaluation questionnaire
4. Using Record Checks
a.
5.
Comparison of survey data with same person data from independent
records.
Response Analysis Surveys
a.
Follow-up visit to respondents after completion of survey
54
Questionnaire Review Worksheet
Questionnaire organization
 Question wording
 Question grouping
 Skip patterns
 Coding
 Questionnaire length

55