Transcript Tabulation and processing

Questionnaire Design Part I
Disclaimer: The questions shown in this section are not necessarily good or
appropriate for a labour force survey. Some are shown as examples of what not to
do. Do not take questions shown in this sections as recommendations for a survey.
Copyright 2010, The World Bank Group. All Rights Reserved.
Design Phase
Define goals or purpose of survey
– What kind of information do you want available for analysis?
– What kind of analyses do you want to do?
Evaluate resources and barriers
–
–
–
–
–
–
–
–
–
Agency capabilities and experience
Highly trained interviewers?
Experienced data entry personnel
Telephones? cell phones? addresses?
Funding
Language, fluency, and literacy
Hard to reach areas or subpopulations
Urban/rural or other geographic differences
Internal displacement, refugees, seasonal migration
Copyright 2010, The World Bank Group. All Rights Reserved.
Design Phase
Survey format
– Common modes - in person, telephone, self-administered
– May be paper based or computer/handheld assisted
– Make sure the format is suitable for the literacy level and
technological access of respondents
– Labor force survey questions will almost always be about
attributes (what one is) and behavior (what one does) rather
than attitudes (what one wants or prefers) or beliefs (what one
thinks is true)
Copyright 2010, The World Bank Group. All Rights Reserved.
Design Phase
Technology can make up for human resources lacks and save
time and materials
– Using computers or handheld devices for interviewing:
• Removes the need for interviewers to follow skip patterns.
This may be very important if interviewers do not have a lot of
experience.
• Allows for much more complex skip patterns.
• Eliminates some data entry errors and legibility issues.
• Makes record keeping easier.
• Reduces the amount of paper interviewers must carry and
keep track of.
• Eliminates the need for the data to be hand entered into a
computer later, reducing staffing needs, leaving more time to
focus on data cleaning, preparation, and analysis, and
reducing time from field to publication.
Copyright 2010, The World Bank Group. All Rights Reserved.
Design Phase
Use one survey for everyone
– There should not be separate surveys for urban and rural areas
• For comparison reasons
• Areas may shift from rural to urban or be in an in between
phase.
– There should not be separate surveys for different regions or
ethnic groups
– The survey should be designed to fit and take note of any
special groups such as refugees, internally displaced people,
and households comprised of orphans
Copyright 2010, The World Bank Group. All Rights Reserved.
Design Phase
Language
– Would segments of the population have difficulty understanding,
responding to, or providing accurate information for the survey in English?
• The most accurate results can be obtained when respondents take the
survey in a language in which they are fluent.
– Survey translation is time consuming, difficult, and expensive.
• Best practice is to have several people independently translate from
English to the target language. They then discuss the differences in their
translations and agree on a common version that will work for all groups.
– It is more important that the same sense of the questions and concepts is
communicated than a literal word for word translation.
Copyright 2010, The World Bank Group. All Rights Reserved.
Design Phase
Language
– If there are many languages in use, it might not be possible to
translate the survey into all of them.
– Testing is required on the translation.
• Verification the translation is accurate
• Cognitive testing should be done
– Interviewers must be able to interview in the required languages.
Copyright 2010, The World Bank Group. All Rights Reserved.
Define the Concepts
Internationally accepted concepts
– Well defined
– Allow for comparability
– Questions exist to measure them
Country specific considerations
– Are there topics where internationally accepted concepts may
need to be modified?
• Household structures may vary for reasons such as polygamy,
orphans, and age at first marriage.
– May additional questions be needed to cover a topic?
• What employment arrangements are likely?
• Is subsistence farming common?
Copyright 2010, The World Bank Group. All Rights Reserved.
Define the Concepts
Country specific concepts (continued)
– Countries can define their own concepts tailored to the realities of
their economy and the needs of their government and private data
users
– This can be done while also maintaining the internationally accepted
concepts.
– Questions must be written and placed carefully so they do not
interfere with others.
– Concepts must be defined clearly so the effectiveness of the
questions and questionnaire can be evaluated.
Copyright 2010, The World Bank Group. All Rights Reserved.
Sourcing Questions
Use existing surveys and questions when possible
– If a survey is appropriate, use it whole.
• If changes must be made, all possible effects on the
questions, concepts, and skip pattern must be thought out.
• Mixing and matching questions between surveys is not a
good idea.
– Modular design allows sections to be added, removed, or
replaced while minimizing risk
• Skips are predominantly internal to the module so they are
less likely to be broken.
• Most of the impact of the question changes will be internal to
the section.
Copyright 2010, The World Bank Group. All Rights Reserved.
Constructing Questions
• Conceptualization
Process of defining, formulating, and clarifying concepts by
specifying what exactly should be measured and how
• Reliability
Repeated measurements should yield the same results?
• Validity
– What was supposed to be measured has to match with what is
actually measured.
– Is measurement bias – systematic error - low?
Copyright 2010, The World Bank Group. All Rights Reserved.
Constructing Questions: Example
Example: Health care use
– What reasons for use should be included? Only major or all?
– What qualifies as a health care provider? Hospital? clinic ?
doctor ? traditional healer ? nurse ? family member ?
– Self only or all family members? What if parent takes child?
– Measured by time? Visits? Illnesses? Treatments?
– What is the reference period?
12
Copyright 2010, The World Bank Group. All Rights Reserved.
Constructing Questions
•
•
•
•
Use simple vocabulary
Don’t try to fit too much in a single question
Group questions on similar topics together
Provide “don’t know” and “refused” answer categories when they
will likely be needed.
• Use lead-ins to provide instructions, explain terms, and make
segues between different sets of questions.
Copyright 2010, The World Bank Group. All Rights Reserved.
Question-Answer Process
• Comprehension
– Understanding words and phrases
– What the respondent thinks you want
• Retrieval of information
– Did the respondent ever store the information in his mind?
– Salience – the type of thing that a person remembers
– How long since the occurrence?
• Judgment
– Generate internal answer, then assess it
• Response Formation
– Edit (possibly censor)
– Communicate
Copyright 2010, The World Bank Group. All Rights Reserved.
Example: How frequent is your tea
consumption?
• Comprehension
– Unusual/difficult wording - “How often do you drink tea?” is more
common
– Presupposition – assumes the person drinks tea
• Retrieval
– Might do it so often they don’t really think about it or pay
attention
– Might have different patterns on work days and non-work days
and not think of this.
– How long is the reference period? Recall. Guessing.
Computational error.
Copyright 2010, The World Bank Group. All Rights Reserved.
Example: How frequent is your tea
consumption?
• Judgment
– Supplement memories with guesses
• Response
– Formulate a response
– Edit for interviewer. Might report more or less if they think their
answer is strange.
16
Copyright 2010, The World Bank Group. All Rights Reserved.