Transcript Document

Daniel Stein (DIME)
Using Randomized Evaluations to
Improve Policy
Development Impact Evaluation Initiative
innovations & solutions
in infrastructure, agriculture & environment
naivasha, april 23-27, 2011
in collaboration with Africa region, SD network, GAFSP and AGRA
1
What we will learn...




Impact Evaluation looks to asses the causal
impact of a project
To assess causality of a project, we must
gather data from a control group
Randomly selecting people into a treatment
and control group is the “gold standard” for
causal inference
Opportunities for randomization abound,
even in large infrastructure projects!
Randomization? That’s Not
For Me


There are opportunities for randomization in
almost every project
Maybe you can’t randomize the placement of
water connections, but you might be able
randomize:





Access price
Maintenance Contracts
Monitoring
Etc
Randomization allows clear answers to YOUR
questions!
Overview of the
Presentation




Control groups and Causality
Selection Bias and Randomization
Opportunities for Randomization
Sampling
Room for improvement:
before-after comparison



Historically, many projects measure “impact”
by looking at project indicators before and
after the project implementation
This is not good enough! Many things change
over time naturally
Impact evaluation seeks to improve on this
strategy
Using monitoring For impact
Before-After Comparison
Treatment Group
14
Is this the impact of the program?
12
10
8
6
4
2
0
Before
After
6
Weather Shock!!
What if there is a drought? Is this the
project’s fault?
9
8
7
6
Farm 5
Income 4
3
2
1
0
Is this the impact of
the program?
Before
After
Before-After Comparison
In most cases, comparing populations before and
after a project is not a good measure of impact!
 Lots of things can affect indicators over time that
have nothing to do with the project

 General economic growth/recession
 Weather
 World prices of commodities

Need a measurement of the counterfactual: what
would have happened in the absence of the project,
with everything else the same
Impact Evaluation and
Causality

Impact evaluation seeks to understand the
causal effect of a program
 Separate the impact of the program from other
factors

Need to find out what would have happened
without the program, or with an alternative
strategy
9
What is Impact Evaluation?
•
Counterfactual analysis
 Compare same individual with & without subsidy,
information etc. at the same point in time to
measure the difference
 This is impossible!
•
The solution: Use a control group
 Need to identify people that represent what the
treatment group would have been like if there was
no project
The Value of a Control Group
Control Group
Treatment Group
14
(+) Impact of the
program
12
10
(+) Impact of other
(external) factors
8
6
4
2
0
Before
After
11
Differences in Differences

Instead of comparing before after, compare
the difference in before-after between the
treatment and control groups
Treatment
Control
Difference
Before
8
8
0
After
14
10
4
Difference
6
2
4
Before-After
Comparison
Diff-inDiff
Difference-in-Difference
Control Group
Treatment Group
14
(+) Difference-inDifference
12
10
Before-after
comparison
8
6
4
2
0
Before
After
13
Diff-in-Diff


Major assumption for diff-in-diff: In the
absence of the project, treatment and control
would have to be the same.
So you need a treatment and control group
that are as similar as possible from the
outset!
Control Group Quality

Control Group has to be good!
 Projects started at specific times and places for particular
reasons

What is a good control group?
 By design treatment and comparison have the same
characteristics (observed and unobserved), on average
 Only difference is treatment

Control group represents what would have
happened to the treatment population if the project
has not occurred
15
Selection Bias

Can we just compare people who received the project to
anyone who didn’t receive the project?
 Danger of Selection Bias
 What was the reason that some people received it and others didn’t?

Selection bias a major issue for impact evaluation
 Projects started at specific times and places for particular reasons
 Participants may select into programs (eligibility criteria)
 First farmers to adopt a new technology are likely to be very different
from the average farmer, looking at their yields will give you a
misleading impression of the benefits of a new technology
Danger of Selection Bias
1)
Village Electrification
Higher Income
OR
Home Electrification
2)
Village is
Politically
Influential
Higher Income
from Other
Sources
How to create Control
Group?



Need to find a group of non-treated people
who can proxy for people who received
treatment
This is hard: there is normally some reason
why some people received treatment and
others not, meaning any differences might
not be due to the project
Unless…
Randomized Experimental
Design
Randomization is the best way to create a good
control group
 Randomly assign potential beneficiaries to be in the
treatment or comparison group


By design treatment and comparison have the same
characteristics (observed and unobserved), on
average
 Only difference is treatment

With large sample, all characteristics average out
19
Can we Randomize?


Randomization does not mean denying
people the benefits of the project
Usually there are constraints within project
implementation that allow randomization
Opportunities for
Randomization

Budget constraints  prevent full coverage


Limited implementation capacity


Randomized phase-in gives all the same chance to go
first
No evidence on which alternative is best


Random assignment (lottery) is fair and transparent
Random variation in treatment with equal ex ante
chance of success
Take up of existing program is not complete

Encouragement design: Randomly provide information
or incentive for some to sign up
21
Example: Irrigation Canal
Project



The government of Umbastan wants to
undertake a project creating irrigation canals
to farming communities
It has identified 100 villages where the project
is feasible and the community would likely
benefit
What types of randomized designs might be
possible?
Example: Irrigation Canal
Project


Possible Constraint: The government only has
money to fund 50 villages
Opportunity for Randomized Assignment:
50 villages to receive project could be
randomly selected from 100 eligible
 This is a fair way to select beneficiaries
 Other 50 serve as control group
Example: Irrigation Canal
Project


Possible Constraint: The government can
fund 100 villages eventually, but only has
time to build the canals the first year in 50
villages
Opportunity for Randomized Phase-In: 50
villages to receive project in the first year
 This is a fair way to select who gets project in first
year
 Other 50 serve as control group for first year
 Drawback is that it would be difficult to measure
long term effects
Example: Irrigation Canal
Project


Possible Constraint: There are worries that
there will not be equitable distribution of
water within a village
Opportunity for randomized Variation in
Treatment: 50 villages could receive water
meters, and 50 could be organized into water
user groups
 If the best system is unknown ex-ante,
randomization can provide evidence for which is
best
Example: Irrigation Canal
Project


Possible Constraint: Worried people will not
connect their fields to main canal
Opportunity for Encouragement Design:
Within villages that receive a canal, farmers
are randomly given a course on how to use
and benefit from irrigation
 This show the effect of the training session
 Also allows us to identify the effect of the
irrigation project
Steps to Randomizaion
Choose sample for impact evaluation
1.


These are people who are eligible for project and
can be in treatment or control
Selection of sample affects external validity
only
Randomize into treatment and control
group
2.

This step affects internal validity, allowing you
to assess the impact of your project
Start with sample of all possible
program beneficiaries
Choose who will be part of
impact evaluation
IE Sample: Could be part of
treatment of control group
Out of IE sample:
Ineligible for program,
or must be treated
Randomize IE sample into
treatment and control
IE Sample: Randomly selected
treatment AND control group
Out of IE sample:
Ineligible for program,
or must be treated
Actual Randomization
Example: Coin Flip
Village A
B
Treatment Group
Control Group
Thank you