Transcript Developing a High Quality Baseline Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006

Developing a High
Quality Baseline
Salimah Samji & Mona Sur
World Bank, New Delhi
June 21, 2006
Overview
What is a baseline?
 Why you should care
 The phases of conducting a baseline

 The
errors to avoid in each phase
How to manage the common errors
 Elements of a baseline survey (TOR)

What is a baseline?
Fixing the time at the base – a benchmark
from which you measure progress
 Snapshot of indicators at a time
 Instrument used to:

 Test
hypotheses of project (assess results)
 Planning (refine targeting, indicators to
monitor)
Why you should care …

To identify whether there were any benefits for
the investments made
 Were
objectives met?
 What factors explain the result?
 How can the program be improved?



Compare alternative models to get the biggest
bang for your buck
To inform next generation projects
Evidence-based policy making – demonstration
effect for government
The phases of conducting a baseline




Design
Implementation – actual survey
Data entry and analysis
Report writing
Phase 1: Design Phase
P1: Check-list
Clear objectives (what is the problem?)
 Clear idea of how you will achieve the
objectives (causal chain or hypotheses)
 Clear and measurable indicators

 Clear
(precise and unambiguous)
 Relevant (to objectives)
 Monitorable
Example of a causal chain
Example: APDPIP
Impact
Outcomes
Higher income levels
Increased use of credit for income generation, Loan
repayment rates
Outputs
Number of SHGs, decreased input prices
Activities
Forming, federating and organizing SHGs
Inputs
Facilitators, Revolving fund (credit)
P1: Check-list (cont’d)




Design survey instrument – keep it simple and
related to the objectives and hypotheses you
want to test
Link surveys to GIS – use consistent units
Select controls/counterfactuals to attribute
change (causality)
Timing of baseline



Before project (what if project never materializes)?
2 years into the project (intervention has begun)?
Other factors (seasonality)
P1: Check-list (cont’d)

Sampling strategy

Random allows inferences
about a population



Random
Stratified random (include groups
which could be excluded)
Non-random – use a group
smaller than the population

Introduces selection bias. Note:
every stratification introduces a
level of bias.
Population
Size
Sample
Size
10
10
50
44
100
80
500
217
1,000
278
3,000
341
50,000
381
1,00,000
385
P1: Check-list (cont’d)



Over sample for attrition
Design the database system for data entry
Translate the questionnaire


Back translate to verify
Provide adequate training on

The objectives and importance of the study
 How the sections are linked and what the questions
mean

Attend the training if possible (stay involved)
P1: Check-list (cont’d)

Test the questionnaire

Ask yourself, can you answer these
questions?
 Are they relevant to the outcomes?
 Will they be understood?
 Field test: To test both how the surveyor
administers the instrument AND how the
respondent understands the question.
P1: Why it is important to field test (i)
When asking a question about the level of
awareness, the surveyor used a word that could
mean awareness or knowledge – the
respondent understood it to mean education. :
The question was: “Ram/ Gita knows about everything that happens
(vikas) in the village. For instance, they know [the name of the
sarpanch, when and where the Gram Panchayat meets, nature and
type of development work in village, etc.]”
P1: Why it is important to field test (ii)
CASE STUDY: Domestic Violence Study in India
“In studying domestic violence, a question in the survey instrument asked if
female respondents had ever been beaten by their husbands in the
course of their marriage. Only 22 per cent of the women responded
positively to this question – a domestic violence rate much lower than
studies in Britain and the US had shown. In probing the issue with indepth interviews we discovered that the women had interpreted the word
‘beating’ to mean extremely severe beating – when they had lost
consciousness or were bleeding profusely and needed to be taken to the
hospital. Hair pulling, ear twisting, etc, which were thought to be more
everyday occurrences, did not qualify as beating. Reponses to a broader
version of the abuse question, comparable to the questions asked in the
US and UK surveys, elicited a 70 per cent positive response.”
Source: Vijayendra Rao (1998) – “Wife-Abuse, Its Causes and Its Impact
on Intra-Household Resource Allocation in Rural Karnataka”
Phase 2: Implementation
What would you do if you …





Were a city person who didn’t speak the local
language very well
Had to travel to several villages and spend
hours asking people questions that have no
relevance to you.
Were paid a small sum per questionnaire
Not monitored by supervisors
This is not your full time job
The answer is simple
Sit at home or in a bar and fill
out the questionnaires!!
P2: Providing incentives and motivation
Sub-contracting surveyors from the state
who speak the language
 Include women surveyors
 Include a supervisor who conducts data
scrutiny
 If possible pay reasonable wages
 Randomly verify questionnaires to reduce
the likelihood of false responses (inform
them beforehand - during the training)

Phase 3: Data Entry and Analysis
P3: Check-list
Data Entry




Make the data entry system as fool proof as
possible - has unique identifiers to link both
household, village and GIS data
Ensure database allows for merging of data
Do not change/erase data on questionnaires
Raw data should always be input as is, changes
can then be made in the database software
(programatically) with documentation
P3: Check-list (cont’d)

Often data entry is contracted out.







Name variables corresponding to the question and section in the
questionnaire – include a dictionary
Code descriptive answers (to facilitate analysis)
All fields should be filled (NA or NR)
Units should be uniform by district
Totals calculated by formula not from summary column
Consistency checks – check for missing entries, wrong
entries, sample statistics, patterns (queries should be
inbuilt)
Validity checks – similar questions in different places on
the questionnaire (RCH example)
P3: Check-list (cont’d)
Data analysis
 Common mistakes in interpreting data

No analysis!


No correlations, crosstabs, statistical significance levels or
regressions
Over generalizing the results
 Mis-reporting statistics
 Using % when the numbers are small
 Attributing causality when it is not demonstrated
Phase 4: Report Writing
P4: What the report should be …






Simple, Clear and Relevant
State limitations (attribution, causality)
Major findings should be upfront
Focus on quality rather than quantity
Technical details in an appendix
Should always




include the questionnaire in the appendix
ask for electronic copy of data
Request copies of filled out surveys
Essential if you change consultants at midterm
or want to conduct internal analysis to compare
modes of delivery (data lost example).
How to manage the common errors

Phase 1: Design
objectives and hypotheses – know what you
want to test
 Identify a person in your unit who will manage this
process
 Write a good TOR, remember the baseline determines
the quality of your panel
 You can add questions as project evolves but cannot
change questionnaire – threat to internal validity
 Identify consultants
 Clear


Procurement – focus on quality not the cheapest bid “if you
throw peanuts you’ll attract monkeys”
Ideally you should have a black-list of organizations
How to manage the common errors

Phase 2: Implementation
 Organize
an impact evaluation workshop if
necessary
 Randomly verify questionnaires to reduce the
likelihood of false responses (no filling it in a
bar)
 Pay reasonable wages to surveyors (if
possible)
 Show the client and firm that you care
How to manage the common errors

Phase 3: Data entry and analysis
 Double-data
entry (2 separate organizations and
verify. Payment based on quality of data entry)
 Select 15 questionnaires at random and check data
entry – person in your unit managing
 Check data quality (consistency and validity checks)
 Hold an IE workshop to build data analysis capacity (if
necessary)
How to manage the common errors

Phase 4: Report writing
 Agree
on an outline beforehand
 Dedicate a chapter on indicators you are
tracking
 Focus on quality not quantity
 Think “Big Picture”
Elements of a Baseline Survey
Terms of References
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
Background: Project objectives and components
Survey design: Consult a sampling expert!!!
Survey instruments
Guidance on survey implementation
Data processing and analysis
Staffing
Duration and time schedule
Submission of reports and datasets
Support to the firm
Budget & Payment Schedule
Annexes: Draft questionnaires, Results Framework
Baseline Survey Design:
Typical Tasks for Consultants





Recommend the methodology for sampling
Calculate the optimal sample size
Develop the sample frame and select the
sample
The final sample and details of the statistical
methodology used to select the sample need to
be cleared by the project
Construct the sample weights and provide
documentation on the methodology used to
construct the weights
Survey Instruments:
Questionnaires






Design or refinement and adaptation of the data
collection instruments
Specify levels of data collection
Length of questionnaires
Prepare all support documentation including
coding guides, interviewer and supervisor
manuals and the data entry manual
Translation and back-translation
Skip patterns, coding open ended questions
Guidance on Survey
Implementation





Implementation plan
Selection and training of field workers: specify
minimum duration of training
Pilot testing should be explicitly specified in ToR
Responsibility for all field operations, including
logistical arrangements for data collection and
obtaining household consent lies with
Consultants.
Ask for field-work progress reports (biweekly/monthly)
Staffing






Sampling expert/statistician
Technical specialists as relevant
Economist
Sociologist.
Core survey staff: the survey manager, the
field manager, the data manager
Enumerators, supervisors and data entry
staff
Baseline Report & Data
Explicitly request final electronic
datasets—with complete documentation.
 Agree on outline of baseline report upfront.

Managing a Baseline Survey




Consult the experts—survey specialist and
sampling specialist and develop the ToR in
consultation.
Selection committee should include a survey
expert and social scientists in addition to
technical experts.
You can never over-supervise!!! Hire third-party
supervision consultant if needed.
Question the data and the findings.
Lets recap what you have learned
The devil lies in the detail
 Be watchful
 No pain, No gain
