Transcript A fresh look at non-experimental evaluation designs

IPDET 2011 Lunchtime Presentations
A fresh look at
non-experimental
evaluation designs
Michael Bamberger
June 28, 2011
Non-Experimental Designs [NEDs]
• NEDs are impact evaluation designs that do
not include a matched comparison group
• Outcomes and impacts assessed without a
conventional counterfactual to address the
question
– “what would have been the situation of the target
population if the project had not taken place?”
2
NEDs not considered methodologically
sound by many evaluators
• Many evaluators argue NEDs are not sound
impact evaluation designs as they cannot
control for effects of external factors that
might have caused observed changes
• Many evaluation texts treat NEDs as the
default option that is only used when budget,
time, data or other constraints do not permit
the use of a “rigorous” evaluation design
3
Situations in which an NED may be the
best design option
• Complex programs
• Not possible to define a comparison group
• When the project involves complex processes of
behavioral change
• outcomes not known in advance
• Many outcomes are qualitative
• Projects operate in different local settings
• When it is important to study implementation
• Project evolves slowly over a long period of time
4
Simple projects, complicated programs and
complex development interventions
Large,
complex
Complex
interventions
Complicated
programs
Small,
simple
Simple projects
• country-led planning and evaluation
• Non linear
• Many components or services
• Often covers whole country
• Multiple and broad objectives
• May provide budget support with no clear
definition of scope or services
• multiple donors and agencies
• context is critical
• May include a number of projects and wider scope
• Often involves several blueprint approaches
• Defined objectives but often broader and less
precise and harder to measure
• Often not time-bound
• Context important
• multiple donors and national agencies
•
•
•
•
•
•
“blue print” producing standardized product
relatively linear
Limited number of services
Time-bound
defined and often small target population
Defined objectives
5
The limited applicability of conventional evaluation
designs in many development contexts
• An increasing proportion of development
interventions fall into one or more of the
categories where NEDs are the best option
So
• conventional impact evaluation designs are of
limited utility for many evaluations
• We need to take a closer look at what NEDs
can offer……
6
Some potentially strong NEDs
A.
B.
C.
D.
E.
Interrupted time series
Single case evaluation designs
Longitudinal designs
Mixed method case study designs
Analysis of causality through program theory
models
F. Concept mapping
7
A. Interrupted time series
Alcohol-related driving accidents
Antidrinking law
Monthly reports of driving accidents
8
B. Single case designs
Phase 1
Baseline
Observation and
rating
Treatment
Phase 2
Phase 3
Post-test
Observation and
rating
Baseline
Observation and
rating
Treatment
Post-test
Observation and
rating
Baseline
Observation and
rating
Treatment
Post-test
Observation and
rating
9
Single case designs
The same subject or group may receive the treatment 3
times under carefully controlled conditions
or
•
• different groups may be treated each time.
• The baseline and posttest are rated by a team of
experts – usually based on observation
• If there is a significant change in each phase the
treatment is considered to have produced an effect
10
A Mixed-Method Case Study [Non-Experimental] Design
Inputs
Implementation
Outputs
Contextual analysis
Process analysis
Using program theory to test
causal analysis
Program
theory +
theory of
change
National
household
survey
Qualitative
data collection
Defining a
typology of
individuals,
households,
groups or
communities
Selecting a
representative
sample of
cases:
Random or
purposive
Ensuring
sample is large
enough to
generalize
Preparation
and analysis
of case
studies
11
Conclusions
• Conventional evaluation designs have limited applicability in
many development contexts
• NEDs are often regarded as the default option when a
“rigorous” design cannot be used
• Strengthening NEDs has received very little attention in the
evaluation literature
• There are many cases where NEDs are the strongest option
• There are a number of potentially powerful NEDs
SO ………………..
12
• It is time to take a fresh look at NEDs
– Not as a second best alternative but
– As the potentially best design in many real-world
evaluation settings.
•
13