Transcript Overview of Questionnaire Design Issues

Questionnaire Design Issues
Section A
Disclaimer: The examples used are not necessarily good or recommended but
are used for illustrative purposes only.
1
Copyright 2010, The World Bank Group. All Rights Reserved.
Questionnaire Content
Decide what to include and exclude to meet your survey
objectives.
 Analysts and data users should be consulted early
 Complete information on survey subject must be investigated
 Only questions which respondents are knowledgeable enough
to answer are asked
 Questions about past events accompanied by a memory trigger.
 Must carefully sequence sensitive questions.
2
Copyright 2010, The World Bank Group. All Rights Reserved.
What is a Questionnaire?
A questionnaire is a tool designed to transfer specific
data from the respondent to the sponsor/user/analyst.
Questionnaires must be designed to yield information of
the highest quality possible with special emphasis on
relevance, timeliness and accuracy.
An effective questionnaire is one that is economical in
human resources, produces data which is wanted and is
processable.
3
Copyright 2010, The World Bank Group. All Rights Reserved.
Major Components of the Questionnaire:
Introduction
1.
Identify the survey and who is responsible for the survey.
2.
Identify any laws that require participation in the survey and
any assurances of confidentiality
3.
Provide a statement of purpose which explains how the data
are to be used and how it will benefit the respondent.
4.
Provide contact information for questions and assistance
4
Copyright 2010, The World Bank Group. All Rights Reserved.
Major Components of the Questionnaire:
Administrative data
1.
Survey identification codes, geographical codes and a serial
number for the questionnaire
2.
Name of the respondent, address or location of the housing
unit, establishment, etc. for which the data is being recorded
3.
Certification of the enumerator and sometimes the
enumerator’s supervisor/crew leader
5
Copyright 2010, The World Bank Group. All Rights Reserved.
Major Components of the Questionnaire:
Main body
1. Contains substantive questions on the subjects of the
questionnaire.
2. OPENING QUESTIONS are used to put the respondent at
ease. These questions tend to be factual questions on
demographic data applicable to all respondents.
3. CLASSIFICATION QUESTIONS are the questions that allow
respondents to be grouped into different categories of interest
based on their answers.
6
Copyright 2010, The World Bank Group. All Rights Reserved.
Types of Questionnaires:
Verbatim:
• The enumerator reads the questions exactly as written:
“What is …’s marital status?”
If single ask: “Has … ever been married?”
Schedule:
• The questionnaire gives topics covered and response categories
without specific question wording:
Marital Status:
7
Copyright 2010, The World Bank Group. All Rights Reserved.
Method of Administration:
Self-enumerated:
• Clear and detailed information on how to complete the
questionnaire is required
• Include detailed instructions on where to leave the questionnaire
and alternative actions if unable to comply with the directions
Interview:
• Have the advantage of training enumerators
• Still provide clear guidance on completion of the questionnaire but
can use abbreviated instructions
8
Copyright 2010, The World Bank Group. All Rights Reserved.
Procedural Decisions
Interview time may be limited due to cost or respondent burden,
forcing reduction in questions asked.
The survey respondent should be considered, questions for each
person may be worded differently than for those asked of a
responsible respondent for a household.
The questionnaire must be suitable for the method of data
collection, the unit of analysis, and how many times the sample unit
will be interviewed.
Questionnaire format can affect how well interviewers/respondents
are able to follow instructions and answer questions.
9
Copyright 2010, The World Bank Group. All Rights Reserved.
Data Capture
1.
Keyed data entry
2.
Scanning (OMR and/or OCR)
3.
Computer Assisted Personal Interview (CAPI) or
handheld/PDA interview
10
Copyright 2010, The World Bank Group. All Rights Reserved.
Ease of Use
The questionnaire must be easy for the interviewer and
respondent to use.
Placement of instructions
Simple questions and response categories
Reference to previous information, flipping back and forth
The harder it is for the interviewer to determine the flow of the
interview, the more chances for introducing interviewer error,
item nonresponse, and respondent frustration.
11
Copyright 2010, The World Bank Group. All Rights Reserved.
Question Wording
Questions must have the same meaning and provide
comparable data from every unit in the sample.
 Vocabulary used must be familiar to the average respondent.
 Take into consideration the respondents’ “frame of reference”
 Response categories offered, type (open or closed), number, and
order.
 Length of question
 Different respondents may respond differently introducing bias.
12
Copyright 2010, The World Bank Group. All Rights Reserved.
Open Ended Questions
Produces an unstructured response that is recorded on
the questionnaire in the words of the respondent
Example: What have you done to look for work during the last
4 weeks? _____________________________
13
Copyright 2010, The World Bank Group. All Rights Reserved.
Closed Ended Questions
Requires respondent to choose from a small number of
choices, usually listed on the questionnaire
Example: Is … male or female?
1 Male
2 Female
14
Copyright 2010, The World Bank Group. All Rights Reserved.
Filter or Screening Questions
A two (or more) part question that is used to direct
respondents to relevant questions. The answer to the
filtering question determines which of two different
questions a respondent next receives.
Example: Were you engaged in work for pay or profit the week
preceding the survey?
If yes, what was the name of your employer?
15
Copyright 2010, The World Bank Group. All Rights Reserved.
Read-in Questions
Includes the answer categories as part of the question. In
an interview situation, the response categories for
these questions would be read to the respondent.
Example: At your job, are you:
1. Employer
2. Employee for pay
3. Self-employed or own account worker
4. Apprentice
5. Unpaid family worker
16
Copyright 2010, The World Bank Group. All Rights Reserved.
Sensitive Questions
Questions which respondents may be reluctant to
answer for various reasons:
 Fear of consequences (legal or political action)
 Fear that their honest response might not be socially
acceptable
 Question is considered too personal
 Question is associated with painful memories
17
Copyright 2010, The World Bank Group. All Rights Reserved.
Skip Pattern
The logical organization of an interview so questions are
asked only of those who fit certain criteria. Other
respondents are directed elsewhere in the questionnaire
depending on their criteria.
Example: Did you work last week? (yes/no)
 If yes, what type of work did you do last week?
 If no, what was the reason you did not work last week?
18
Copyright 2010, The World Bank Group. All Rights Reserved.
Translating Concepts into Questions
1. Applicable - Ask only for information that cannot be obtained
elsewhere and are directly related to the objective of the
survey.
2. Specific - Avoid questions which are open to interpretation or
are too vague.
3. Accurate - Do not ask for information or precision that
respondent’s don’t know or are unwilling to provide.
19
Copyright 2010, The World Bank Group. All Rights Reserved.
Translating Concepts into Questions
4.
Precise - Ask only one question at a time – avoid
conjunctions, such as “and”, “or.”
5.
Neutral - Avoid emotionally charged language.
6.
Unassuming - Questions should not assume that a certain
condition exists.
20
Copyright 2010, The World Bank Group. All Rights Reserved.
Methods of Recording Responses
Type of question:
• Closed-ended questions are pre-coded
• Open-ended questions are write-in answers
• Questions asking for numeric information are self-coded
(age, income)
• Some open-ended or administrative information may be
coded by the interviewer in the field.
21
Copyright 2010, The World Bank Group. All Rights Reserved.
Methods of Recording Responses
Data capture considerations:
• Manual coding required (timeliness, conversion error)
• Method of data capture (keyed data entry, scanning, or keyed
entry during interview/CAPI)
• Ease of keying, scanning in order to capture data accurately and
quickly
22
Copyright 2010, The World Bank Group. All Rights Reserved.
Section A Quiz
1. When should you involve data users in the questionnaire
development process? (slide 2)
2. What are the 3 main components of a questionnaire? (slides 4-6)
3. List 3 methods of questionnaire data capture. (slide 10)
4. Name 4 types of questions. (slides 13-19)
5. What is a skip pattern? (slide 20)
6. List 3 rules to keep in mind when translating concepts into
questions. (slides 21-22)
7. Give 2 examples of self-coded questions. (slide 23)
8. How can you ensure data capture concerns are addressed? (slide
24)
23
Copyright 2010, The World Bank Group. All Rights Reserved.