Transcript Improving Anti-Poverty Policies: The role of Creative

The why and the how of randomized
experiments
Abhijit Vinayak Banerjee
Santiago, December 2009
Why evaluate?
Theory is a very limited guide to policy
• It is very good at alerting us to various
possible mechanisms
• It just does not tell us which of those
mechanisms actually work and which
don’t
• Because theory is always necessarily
incomplete.
– It is based on concepts, which come out of our
need to limit the complexity of reality
– It is based on behavioral assumptions and
assumptions about the environment that are
untested and often untestable
An example of a challenging concept
• Education
• Let us accept the evidence that education promotes
growth
• How to promote education?
• Concepts often used in the growth literature are
things like “fraction of population completing
primary school”
– These are not policy levers
• Policy levers are things like:
– Teacher-student ratio, textbooks, school uniforms,
school meals, etc.
• Theory tells us nothing about which of works best
• Can we assume that the policy makers know?
An example of unexpected behavior
• The demand for health goods
– Like immunization, insecticide treated bed-nets,
chlorine for purifying drinking water
• Standard economic theory
– These goods are valuable to some people (people who
live in malarial zones, say) less to others
– Pricing at marginal social cost correctly discriminates
between those who want them more than the social costs
and those who don’t
– However there are externalities: infectious diseases,
intra-family issues
– So some subsidy may be justified.
– But no argument for giving them away
– Sunk cost fallacy argues for charging a positive price.
The pricing of health goods
• Several studies are exploring
the impact of charging or
subsidizing people for health
behavior
• First example: Lentils for
vaccine.
– Immunization is really low in
Rajasthan (less than 5%)
– What is the price elasticity of
vaccine demand?
Abdul Latif Jameel Poverty Action Lab
Testing Demand and Supply
• One first possibility is that the supply channel is the
problem:
– Conducted monthly immunization camps in 60 villages:
regular camps held rain or shine
• One second possibility is that there is a problem of demand:
– People not interested in immunization, scared, etc.
– Can demand be affected?
– Extra incentive: in 30 of the camps, provided a kg of
lentils for each immunization
• 60 camps’ villages remained the control group.
Immunization rates were followed in treatment villages,
control villages, and one neighboring village of each of the
treatment villages
Abdul Latif Jameel Poverty Action Lab
High price elasticity
• Surprisingly high price elasticity, in light of the
expected benefits. Why might that be so:
– Information?
– Discount rate?
– Something else?
• Smokeless stoves
• Chlorine for water: same effect
• Bed-Nets:
– Dupas and Cohen : randomize the price at which
bednets are offered in pre-natal clinic
– Dupas: similar experiment among households.
• Deworming drugs: take up collapses with
(small) positive price
Should People Pay for
Health Goods?
• Conventional economic theory:
– If the health good has a high externality and is
expensive, you want to subsidize its use
(possibly even subsidize people to do it, like in
the case of the immunization program), in
particular when the price elasticity is high
• “New” conventional wisdom (social marketing)
– Health goods you give away may not end up in
the hands of those who need it the most
– The fact of paying may encourage people to use
the good (sunk cost fallacy)
Abdul Latif Jameel Poverty Action Lab
Evidence from Bed nets
• Do people who get a free bed
net use it less than those who
have to pay?
• Do people who get a free bed
net need it less than those who
have to pay?
Abdul Latif Jameel Poverty Action Lab
Effective Coverage: Share of Prenatal Clients
Sleeping Under ITN, by Price
Number Sleeping under ITN
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0
10
20
40
50
ITN Price (in Ksh)
Abdul Latif Jameel Poverty Action Lab
Selection Effects of Price on Health
• Do higher insecticide treated
net prices induce selection of
more vulnerable women (i.e.
sickest)?
• Result: No. People who buy
nets at current cost-sharing
price are healthier than the
average prenatal client in the
region
– Ability to pay seems to be
the binding factor
Abdul Latif Jameel Poverty Action Lab
Lessons
• Find no evidence that cost-sharing reduces wastage on
those who will not use the product
• Also find no evidence that cost-sharing induces selection
of those who need net more
• Cost-sharing does considerably dampen demand
– As the price goes from zero to the prevailing costshare price, uptake drops by 75%
• Another example where the cost benefit analysis is not
what you thought!
– Free distribution seems at least as cost-effective as
cost-sharing (because of externalities)
• The debate generated by the paper: does this generalize?
– Proponents of cost sharing proposed various theories
to explain why the results may not generalize.
– But they had to formulate new arguments, which can
then be tested: this moved the debate beyond where it
was… “cost sharing is good”.
Abdul Latif Jameel Poverty Action Lab
Why experiment?
Policy has to be based on learning
from experience
• History is a set of experiments…
• Why do we need Experiments?
Why experiment: First answer
• Selection bias in observational data
• Can sometimes be solved using natural
experiments or other non-experimental
methods
• But that limits the set of programs that
can evaluated.
• And can be less than transparent
Why Experiment: Second answer
• You typically know exactly who was treated or
something close
• And you control data collection
• Therefore you can target your data collection.
• As against in an observational study where the quasirandomization emerges opportunistically.
– You study a policy and typically decide ex post who was treated
and who was not. Unless you were directly involved in the policy
making—in which case why not try to randomize?
• And especially with Instrumental Variables methods,
you don’t always know who you are comparing
• And even when you do, the actual event may be many
years past
• Therefore you cannot rely on data collected for you
• Hence detail/creativity in data collection is lower
Example: Decoy visits in the Evaluation of
Police Reform in Rajasthan
• Can the police be taught to investigate better, treat
people better and not to try to fob off complainants?
• In collaboration with Rajasthan Police Department
• How do you measure the willingess to register crimes
• Decoy visits: Surveyors act as real complainants.
– Revealed themselves if the case was about to be registered
– Only 54% of complaints were registered
• Neither training nor more vacations for the police
nor community observers in the police station had
any effect on registration
• However the (random) number of previous revealed
decoy visits matters: each visit increased the
probability of FIR registration by 4%
Why experiment: Third answer
• Experiments can be designed to test very
specific hypotheses
• Whereas observational studies, because
they rely on policy variation that exists in
the world, can rarely be mapped to
something quite so narrow.
• Therefore we can test individual elements
of a policy rather than the policy as a
whole
Technology adoption: fertilizer in
Kenya (Duflo-Kremer-Robinson)
• Current hike in food price underscore the needs to
improve the productivity of agriculture in developing
countries.
• The Green revolution has changed the face of Asian
Agriculture since the 1970s. Need a green revolution for
Africa
• Part of the solution is to develop new technologies, and
part is to make sure the ones we have get used.
• Big puzzle: why do so few farmers adopt fertilizer in
Western Kenya
– The technology is well known
– It has high returns (which we confirmed in experiments)
– It can be done in small quantity
Possible reasons
• Development economists have worked on
this topic for decades. Hypotheses:
– Low actual returns
– Lack of information/knowledge
– Financing difficulties
• Duflo-Kremer-Robinson set up a series of
field experiments over several years in
Western Kenya to investigate these issues.
– Controlled trial on (randomly selected) farmers’
plot: fertilizer seem profitable
Information
• Several experiments to understand the role and
the diffusion of information:
– Experiments on farmers plot were done with randomly
selected farmers: we follow their adoption and that of
the control farmers
– Other ways to disseminate information:
• Starter kits (Malawi)
• Demonstration farms
• Neighbors
– Invited
– Non invited (natural diffusion)
• Bottom line:
– information plays a role (10% increase in adoption
among pilot farmer)
– However it does not diffuse naturally (no effect of
friends network)
– Financial barriers remain a concern for farmers
Present-bias
• Farmers seem to make plans to use fertilizer at harvest
time, but not carry them out
• By the time of planting they have no money left.
• Possibly it is because they postpone the purchase until
they need the fertilizer, but then they consume too much
in the intervening period (may be because of family
demands).
• If this is the case, a program that encourages them to
purchase fertilizer right after harvest, rather than later,
should be effective.
• With an NGO (ICS), they offered free delivery of
fertilizer right after harvest (the “SAFI” program)
Results
• SAFI is taken up by 40% of farmer and
led to a 14 percentage point increase in
fertilizer use (a 60% increase).
• As large as the impact of a 50% subsidy
• Offering free delivery at the time they
need it is not sufficient (it led to a 7
percentage point increase in use).
• When farmer are asked to chose the
timing, they chose the early delivery.
What can’t experiments do?
• Tell you what exchange rate to set
• But it is not clear that regressions of any
kind are the right methodology for
answering such questions
• May better to be build simulation models
based on reliable parameter estimates.
• From micro regressions.
• Experiments may play an important role
here
The question of external validity
• How do we know that the result from an experiment
will generalize to other places
• Problem for all forms of empirical research.
• However it might seem that it is less of a problem for
cross-country comparisons
– Because the result is an average covering a wide range of
environments
• Not exactly right
– Different results when you include different controls
– Could be because the set of countries being compared are
different
• One advantage of experiments is you are much
more likely to know the exact population that is
being compared.
However
• There is no dismissing this issue.
• Replications help build some confidence
in the results
– Theory helps give us identify the kind of
replications that make sense.
• The problem remains that as of now
organizations that participate in
randomized trials tend to be “strong”
organizations
• We need more randomized evaluations in
weak organizations
Most importantly
• There is an infinity of policy questions out
there.
– Textbooks, uniforms, meals, teachers,
blackboards, toilets, scholarships, computers..
• Theory is the only reliable guide to what
are the right questions to answer
• Unfortunately in many cases we do not
have a useable theory: that tells us, for
example, textbooks are more like
computers or more like teachers.
On balance however
• We have learnt things from experiments we
could have not learnt otherwise
• The benefits from the experimental approach go
beyond the specific learnings (with their
limitations).
– Have helped popularize a culture of emphasizing
experimentation and learning from experiments
– And have imposed some discipline on what can and
cannot be claimed as known
• Does not mean that we learn only from
Randomized Experiments
• But experiments set a standard of evidence and
a culture of being willing to embrace failures.
• THANK YOU!