Transcript Combining different methods in impact evaluation Presentation by Steen Folke, Senior researcher at the Danish Institute for International Studies International Workshop on Impact Evaluation Paris, November.

Combining different methods in
impact evaluation
Presentation by Steen Folke,
Senior researcher at the Danish Institute for
International Studies
International Workshop on Impact Evaluation
Paris, November 15 2006
Important banalities
• Impact evaluation is not an exact science
– and it is dangerous to pretend that it is
• The use of a combination of different –
quantitative and qualitative – methods can
highlight different dimensions of impact
• Impact is the result of development
processes that depend on the context as
well as the intervention
Definition of impact
• ”Impact assessment is the systematic
analysis of the lasting or significant
changes – positive or negative, intended
or not – in people’s lives brought about by
a given action or series of actions”
(Chris Roche: Impact Assessment for
Development Agencies, Oxford 1999)
Classical effect evaluations
• Quasi-experimental survey design
• Ceteris paribus assumptions
• Before/after and/or with/without
• Quantitative methods
• Attempted objectivity
• Dubious assumptions
Participatory impact assessments
• Involvement of beneficiaries
• External facilitators
• Participatory techniques
• Qualitative methods
• Subjective
• Problems of reliability
Wider impact studies
• Heterogeneous category of in-depth studies
• Contextualised, tailor-made approach
• Unintended consequences as well as stated
•
•
•
objectives
Extensive fieldwork
Combination of quantitative and qualitative
methods
Development interventions and societal
processes
Model of development intervention,
context, process and impact
Noakhali Rural Development
Project, Bangladesh
• Danida funding 1978-92 390 m. DKK (> $50 m.)
•
•
•
•
– flagship in Danish aid
2 phases, 15 components: irrigation,
cooperatives, rural poor prog., mass education
etc.
>60 expatriate advisers, staff >1000
Waning enthusiasm, many implementation
problems (complexity)
3rd phase planned, but aborted
Ex-post impact evaluation
• Did the flagship float or sink?
• No terminal evaluation
• Ex-post impact study 9 years after
• Contextualised, tailor-made approach
• Combination of quantitative and
qualitative methods
• 8 researchers, 15 assistants, 4 months
fieldwork
Methods I: Project focus
• Documentary study (project documents)
• Archival work in Danish embassy, Dhaka
• Questionnaire survey with former advisers
and Danida staff
• Stakeholder interviews (Danida staff,
former advisers, Bangladeshi staff)
• Quantitative analysis of project monitoring
data
Methods II: Study of context
• The national context: books, articles, statistics
•
•
•
•
(economy, policies)
The local context: census data, other sources
(environment, population, socio-economic
development)
Institutional mapping (esp. NGOs)
Extensive village studies (12 villages, 9 with and
3 without NRDP)
Intensive village studies (4 NRDP villages,
restudy 20 years later)
Methods III: Quantitative
• Surveys of 5 important project components:
•
•
•
irrigation, infrastructure, fisheries, cooperatives
for rural poor, mass education
Random sampling (beneficiaries)
Questionnaire-based interviews with
beneficiaries
Some interviews with non-beneficiaries (’control
group’)
Methods IV: Qualitative
• Assessment of roads, buildings and
irrigation canals (function, maintenance)
• Key informant interviews
• Focus group interviews
• Observation
• In-depth interviews (issue-based and life
stories)
Example of findings: Irrigation I
Economic situation of household
Better
Same
Worse
Total
Marginal 26
2
10
38
Small
27
1
6
34
Larger
6
1
1
8
Total No. 59
4
17
80
Total %
5%
21%
100%
74%
Example of findings: Irrigation II
Most important reason for improvement in
economic situation compared to 15 years
ago (total: 59 respondents):
Increased production (irrigation): 20 resp.
Remittances:
20 resp.
Petty trade/business:
8 resp.
Other reasons:
11 resp.
Most important reasons for deterioration (17 resp):
Illness, large family, loss in business
Conclusion: The project
• The Danida flagship did not sink – the
evaluation team found substantial impact
in many areas 9 years after
• The impact was primarily in the form of
marginal improvements for the rural
population, mainly the poor and women
• The socio-economic inequalities and the
local power structure were not challenged
Conclusion: The methods
• The combination of a range of different methods
•
•
•
contributed to unravel different dimensions of
impact from a complex project
Study of the context was essential to understand
impact of the project versus other factors
Quantitative and qualitative methods were
complementary
Attempts to assess the counterfactual brought
limited insight