IMPACT EVALUATION WORKSHOP

24 February 2011

Workshop objectives

Our goals for this workshop: 1.

You know when and how an impact assessment can help you to evaluate and improve your work.

2.

You can be part of a team conducting an impact assessment with the support of experts.

3.

You can apply knowledge gained from other impact assessment in Africa and the world.

AGENDA

I.

Introduction to IPA II.

Impact evaluation III.

Lessons from evaluations: Africa & beyond IV.

Discussion and exercise: designing an evaluation for your sector

AGENDA

I.

Introduction to IPA II.

Impact evaluation III.

Lessons from evaluations: Africa & beyond IV.

Discussion and exercise: designing an evaluation for your sector

I.

INTRODUCTION TO IPA

24 February 2011

Who is IPA?

   Innovations for Poverty Action (IPA) is a nonprofit organization that creates, evaluates, and replicates innovative approaches to solving real-world problems faced by the poor in developing countries. IPA disseminates proven findings about what works, and what does not work, to policymakers, practitioners, and donors around the world.

IPA works with a network of researchers from major universities around the world who are interested in issues related to the fight against poverty.

What do we do?

  Uses rigorous evaluations to determine what works and what does not work      Surprisingly little is known about what works in the development field Sometimes conventional wisdom needs to be rethought Estimate magnitude of effects  Most cost-effective solutions; can be surprising Know how to improve programs Where to spend limited resources IPA was created in 2002 to answer questions about what poverty alleviation programs work best

IPA’s approach

We generate insights on what works and what does not through randomized evaluations, and ensure that those findings will be useful to, and used by practitioners and policy makers

Innovate Evaluate Replicate Communi cate Scale - Understand market failures -Develop innovative solutions to poverty - Use frontier knowledge from economics, and psychology Randomized Controlled Trials -Impact evaluations - Comparing variations of an intervention - Experiment with product designs Replicate evaluations in various settings to : - Generalize research findings - Tell practitioners what works (and not), when Effectively communicate to practitioners: -Conferences -Workshops with policy makers and practitioners -Policy memos and focus notes Facilitate scale up of effective solutions : - Active policy outreach -Practitioners’ toolkits - Hands-on technical assistance

Innovations for Poverty Action

IPA works on 300 projects in 42 countries

IPA in Ghana

       Here since 2005 16 active projects, 40 full time staff, up to 250 part time, 2 permanent offices Partnerships with: Ghana Education Service, Sinapi Aba Trust, Freedom from Hunger, Mumuadu Rural Bank, Zenith Bank, Presbyterian Agricultural Service Implementation and Evaluation Teams Dissemination of results to local policy makers and practitioners Providing education and support on RCTs Scaling ups and Replications

Impact evaluation warm-up

  Suppose there is a doctor in Ghana who wants to test the effectiveness of a new drug that she has developed to prevent diabetes. Diabetes is a disease marked by high levels of sugar in the blood; obesity, poor eating and exercising habits, high cholesterol, and genetics can be risk factors for developing diabetes. The doctor wants to know how the drug will affect the ordinary Ghanaian.

A few NGOs wanted to test the drug and proposed the evaluation ideas listed below. Consider these questions as you read through:    What is the evaluation measuring?

Who is it measuring the impacts of the drug on?

What could be problems with this evaluation?

Scenario 1: Clinic

   A medical clinic agrees to test the drug. The clinic is an old a prestigious organization and their offices are located on the top floor of a very tall storey-building. The doctor is excited to partner with such a well-renowned clinic (her only complaint is that there is no elevator and she has to climb the stairs every time she meets with their staff!) The staff of the medical clinic want to go to the market and give a coupon for a free treatment to the first 500 people that they meet. Their staff says that they will take the first 250 people who come to their office to collect the drug and will provide the drug for free until the study ends. Several years later, they will find all the 500 people and assess their health. The clinic staff says that this way they can show a better comparison: how the group receiving the drug responds to treatment by showing how the people who do not receive the drug compare

Scenario 2: NGO

  A health-focused NGO is also interested in how nutrition and exercise affect whether one develops diabetes. They propose to include the drug in a study they are running. In their study, 500 participants are given advice and help in preparing healthy meals as well as a free membership to a gym. The director of the NGO says that they will give half of these people the pill and see how it affects whether they develop diabetes.

AGENDA

I.

Introduction to IPA II.

Impact evaluation III.

Lessons from evaluations in Africa and around the world IV.

Discussion and exercise: designing an evaluation for your sector

II. IMPACT EVALUATION

• • • What is an impact assessment?

What makes an impact assessment?

The keys to a good impact assessment 24 February 2011

II. IMPACT EVALUATION

• • • What is an impact assessment?

Why randomize?

The keys to a good impact assessment February 16, 2011

What goes into policymaking?

Why focus on impact evaluation?

 Surprisingly little hard evidence on what works  Can do more with given budget with better evidence  If people knew money was going to programs that worked, could help increase pot for anti-poverty programs  Instead of asking “do aid/development programs work?” should be asking:  Which work best, why and when?

 How can we scale up what works?

18

Lecture Overview

 Why evaluate? What makes a good evaluation?

 What do we hope to learn?

 What do different type of evaluation do?  Understanding the program  Impact evaluation  Defining impact (preview)  When to do a randomized evaluation?

Objective of evaluation



Lesson learning

    Beneficiaries Program Organization World 

Accountability



Reduced poverty through more effective programs



Different types of evaluation contribute to these different objectives of evaluation

What makes a good evaluation?

 Ask the right questions   For accountability For lesson learning  Answers those questions in unbiased and definitive way  To do that you need a model: logical framework/theory of change     Who is the target?

What are their needs?

What is the program seeking to change?

What is the precise program or part of program being evaluated?

21

Example of Evaluation Types

Problem: Children are not learning to read at young ages, and they get left behind as they get older Solution: An NGO provides after school classes What is the problem?

Needs Assessment Program Evaluation

What are ways we can address the problem?

Are they being implemented?

Process Evaluation

Literacy improves among those students What are the program results?

Impact Evaluation How can we improve going forward?

The different types of evaluation

Evaluation (M&E) Program Evaluation Impact Evaluation Randomize d Evaluation

Different levels of program evaluation

 Needs Assessment  Program Theory Assessment  Process evaluation  Impact evaluation  Cost-benefit/Cost-effectiveness analysis 24

Program Evaluation

Evaluation (M&E) Program Evaluation

Program Evaluation: community mobilization in education

Needs Assessment

  Clear sense of target population Clear sense of need program will fill • • Clear articulation of program benefits Clear sense of alternatives 27

Program Theory Assessment

 How will the program address the needs put forth in your needs assessment?

 What are the prerequisites to meet the needs?

 How and why are those requirements currently lacking or failing?

 How does the program intend to target or circumvent shortcomings?  What services will be offered?

28

Process Evaluation

 Are basic tasks being completed?

 Are the services being delivered?

 Is the intervention reaching the target population?

 Is the intervention being completed well or efficiently and to the beneficiaries’ satisfaction?

Impact Evaluation

Evaluation (M&E) Program Evaluation Impact Evaluation

Impact Evaluation

 The program happened, how did it change lives?

 What does Theory of Change say we might expect to change?

 Primary outcome (impact): did community monitoring cause children to learn more (e.g. measured by test scores)?

 Also distributional questions such as, did the program impact some children more than others?

31

How to measure impact?

 What would have happened in the absence of the program?

 Take the difference between what happened (with the program) …and

- what would have happened (without the program)

= IMPACT of the program

Impact: What is it?

Intervention Impact Time

The counterfactual

     Ideally, we would want to observe how the exact same subject would

react having not received and received the program.

This is the counterfactual – what otherwise would have occurred to the same group of people if the program had not been given to them.

Unfortunately, this is impossible to observe Thus, we must try to invent an approximation of the counterfactual – comparison groups that are as similar as possible; where the only difference between them is whether they receive the program or not

Discussion Point: Why is this important?

 If we don’t have two groups that are similar, then if we see a difference between them at the end of the trial, that difference could be the result of a million other factors, rather than the program, and we cannot say whether the program or something else caused the difference.

Constructing the counterfactual



Counterfactual is often constructed by selecting a group not affected by the program



Randomized:

 Use random assignment of the program to create a control group which mimics the counterfactual.



Non-randomized:

 Argue that a certain excluded group mimics the counterfactual. 35

How impact differs from process?

 When we answer a process question, we need to describe what happened.  When we answer an impact question, we need to compare what happened to what would have happened without the program 36

Methodologies in impact evaluation



Experimental:

 Randomized Evaluations 

Quasi-experimental

  Instrumental Variables Regression Discontinuity Design 

Non-experimental

    Pre-post Difference in differences Regression Matching 37

Questions ?

II. IMPACT EVALUATION

• • • What is an impact assessment?

Why randomize?

The keys to a good impact assessment February 24, 2011

Preview: why randomize?

Evaluation (M&E) Program Evaluation Impact Evaluation Randomize d Evaluation

The Four W’s and How To of Randomization



What



Why



When



Where



How To

What is a randomized evaluation? Think clinical test!

 EX: Say we want to test a new diet pill  We draw lots among the 200 volunteers: 100 receive the pill, 100 do not. This is key between we get two groups of 100 similar people in all respects (same initial weight, same motivation, etc.).

 The clinical test allows us to say something about the effect of the intervention by comparing the average outcome for each group after treatment. 

Avg Treatment Weight < Avg Control Weight



The pill!

Now, let’s apply an RE to a program

Suppose you have the ability to reach 250 new beneficiaries in your program. Before starting the program :

1.

Make a list of 500 eligible individuals.

2.

Perform a lottery and the 250 who are selected form the treatment group.

* We get two groups that are similar in all respects*

3.

Rollout the program to the 250 selected individuals only.

4.

After you feel that the program has been in operation long enough to observe an effect, interview both groups – the 250 (treatment) and 250 (comparison)

Why? -- Let’s Review! Objectives for evaluation

 Accountability  Lesson learning  Reduced poverty through more effective programs 

Solid reliable impact evaluations are the building blocks for more general lesson learning

44

Why randomize ? Non-randomized methods are often biased

Example: Pratham’s Read India program Method of impact assessment Before / After Beneficiaries / Non beneficiaries Double difference Randomized evaluation Impact on literacy (measured through cards) 0.60

-0.90

0.31

0.88

Clear cost-benefit analysis

 Needs assessment gives you the metric for measuring impact  Process evaluation gives you the costs of all the inputs  Impact evaluation gives you the quantified benefits  Identifying alternatives allows for comparative cost benefit 46

Eg: Comparative cost effectiveness

47

When ? Learning from impact evaluations

 Learning from a single study  Learning from an accumulation of studies  We learn more if we place each evaluation inside a general learning strategy, helps us be explicit about  questions we want to answer  hypotheses we want to test 48

When to do a randomized evaluation?

 When there is an important question you want/need to know the answer to  Timing--not too early and not too late  Program is representative not gold plated  Or tests an basic concept you need tested  Time, expertise, and money to do it right  Develop an evaluation plan to prioritize 49

When NOT to do an RE

 When the program is premature and still requires considerable “tinkering” to work well  When the project is on too small a scale to randomize into two “representative groups”  If a positive impact has been proven using rigorous methodology and resources are sufficient to cover everyone  After the program has already begun and you are not expanding elsewhere

Where?



YOUR PROGRAM

How? Developing an evaluation strategy

 Determine key questions for your organization  How many can you already answer?  Select top priority questions for impact evaluation  Establish plan for answering them 52

How? Developing an evaluation strategy



Things to Keep in Mind :

 With key questions answered from impact evaluations, process evaluation can give your overall impact  A few high quality impact studies are worth more than many poor quality ones  If you ask the right question, more likely to use it 53

The steps for a randomized evaluation

Hypothesis: The NGO has the capacity to reach 100 new villages with fertilizer distribution Compile a list of 200 eligible villages Complete a survey based on a sample of the households in these villages Randomize who receives and not 100 treatment villages 100 control villages Intervention: fertilizer distribution Complete another survey for all the villages Impact = T-C

Random sampling and random assignment

Randomly

sample

from area of interest

Random sampling and random assignment

Randomly

sample

from area of interest Randomly

assign

to

treatment

and

control

Randomly

sample

from both treatment and control

Different levels of randomization are possible depending on the context

 The randomization can be both at the individual level of the level of villages (or neighborhoods), classes, schools, or community health center, etc.

 Examples: • • • Micro-credit in a city  individual level Awareness campaigns  village level Encourage to vaccinate  community health center

Questions ?

II. IMPACT EVALUATION

• • • What is a good impact assessment?

Why randomize?

The keys to a good impact assessment 24 February 2011

Why measure the impact?

 The intuition about a program’s effect is often misleading  Measuring the impact of several alternative programs allows for the optimal allocation of resources  Impact assessments increase the confidence of donors.

  If you design the evaluation in advance, stick to the plan and are open about the results, it shows you’re willing to be accountable, to learn and improve you projects The impact assessment can also be carried out by an independent organization  Much thought goes into an impact assessment, beyond the simple question of effect. It can tell us:    The best way to implement the program Who to target Several questions at the same time

Sample size

 The number of units measured in a study is also important  The more people or households or schools we can collect data from, the better we can estimate the broader effects of a program.

Sample Size Exercise

 Take a coin and flip it twice  How many heads did each person have?

 Flip it 10 times  Flip it 20 times  As the number of flips increases, the data we collect gets closer to the real principle operating behind the question we are trying to answer: what is the probability that the coin lands heads?

Sample size, continued

 How large we make our sample will depend on the statistics behind choosing enough units to allow us to see if there is a change because of the program  I.e., if we chose two people, gave one the program and the other not, could we say that any change we see in the “treated” person is reliable?

 Sample size may also tradeoff with funding and monitoring constraints

Bias that may influence a study

 Selection bias  Attrition  Spillover  Hawthorne effect

Selection bias

   Selection bias is present when the sample does not represent the larger population from which it was drawn Sometimes even random samples can be influenced by the type of people who are chosen to participate For example:  Suppose a pork seller wants to see how if a discount would increase his sales    So you decide to test whether a coupon for a 50% discount off a first purchase will influence if people are willing to buy pork or not So you go to Nima and find 100 people on the street who agree to be interviewed and you randomly give 50 a coupon for 50% off their first purchase of pork.

What could go wrong?

 Selection bias will influence the outcome of the study positively or negatively in a way that does not accurately reflect how the study would have faired within the population you are hoping design a program for

Attrition

   Attrition is when people dropout of your sample in a way that influences the outcome you hope to estimate.

 People can attrit by refusing to continue to participate, moving without follow-up contact information, dying For example:  The University of Cape Coast wants to see how an internship will affect the future careers of its business students  So 100 students agree to participate and 50 are randomly selected to be an intern over the summer with an investment firm in Accra; 5 years later, researchers return to find 60 to interview. What could be a source of bias?

Attrition affects the comparison one makes at the end of analysis. It is important to think about  who “dropped out” during the evaluation,   what those people were like, and if the “drop out” rate was different between the treatment and the control groups

Spillover

 Spillover influences the “treatment effect” estimated at the end of an evaluation – some of the control group receiving the treatment influences the comparison we make at the end  For example:  Imagine that a bank wants to increase use of its savings program among market women:  100 in one market, 50 get the offer. What could go wrong?

 Spillover affects the comparison you can make at the end of the evaluation; there are two kinds

Spillover, continued

  Intention to treat – the effect of the program on those you intended to be affected will be a simple difference between treatment and control Treatment on the treated – the effect of the program on those who received the treatment (being given, directly, a pamphlet) by taking the intention to treat estimate and dividing by the differences in treatment receipt rates between treatment and control  E.g., if 95% of the women who received the pamphlet started an account and 85% of women who did not receive the pamphlet started an account, assuming that the intention to treat estimate effect was 0.1 percentage point difference between treatment and control, then the treatment on the treated estimate would be 0.1/(0.95 – 0.85) or 1%, significantly higher!



Discussion Point – Which do you feel is more important to informing policy – Treatment on the treated or intention to treat?

Hawthorne effect

    The fact of being part of a study may influence the outcomes that are observed This effect was observed in the United States in a factory  A researcher changed certain things about the factory – the lighting, length of breaks, providing food  Each time a change was made, productivity increased because the workers knew they were being monitored E.g., think of a weight-loss medicine study in which participants know that their weight will be monitored – people in the control group may exercise and eat healthier because they are more aware of their weight Measuring this effect still happens today and is important to consider when conducting an evaluation

Questions ?

AGENDA

I.

Introduction to IPA II.

Impact evaluation III.

Lessons from evaluations: Africa & beyond IV.

Discussion and exercise: designing an evaluation for your sector

Lessons from our impact assessments

    

Education Health Microfinance Agriculture/Food security Governance

Education: Deworming in schools

 

Hundreds of millions of children worldwide are infected with intestinal parasites. These parasites affect their health, cognitive development, schooling, etc.

An assessment conducted by IPA in Kenya and in India revealed the immense impact of deworming campaigns in schools

For only $0.50 per child per year: you can reduce student absence by 25%!

 IPA initiatives worldwide: in 2009, 20 million now encourages children deworming worldwide have benefited from the program

Health (1/2): Distribute free malaria nets

  Nets can reduce child mortality by 20%.

Reducing malaria also has an impact on learning and economic conditions.

However, few pregnant women and children use them   How can we increase their use?

 Our study aimed to inform debate about the best price http://www.povertyactionlab.org/policy-lessons/health/pricing-health products •

Free distribution does not increase use but

•

Charging does not imply that those who need it will buy more

•

Demand reduced by 60%, from free to 3 GHC

•

Free distribution can save more lives, and has a lower cost per life saved (through indirect effects)

Health (2/2): Place chlorine dispensers at community water sources

   Diarrhea accounts for 20% of infant mortality in the world. They are related in large part to contamination of drinking water In western Kenya, three quarters of household are aware that treatment with chlorine reduces risk of contamination, but only 5% use it.

http://www.povertyactionlab.org/policy lessons/health/child-diarrhea

Establish chlorine dispenser at water sources in communities is the best strategy to improve water quality.

Microfinance: Taking account of psychology in the design of savings products

   Nearly everywhere, men and women say that they do not save as much as they want to IPA has tested many elements that take into account psychological aspects to save, thereby building a safer future http://poverty-action.org/project/0071

Some simple interventions can increase deposits

•

Ghana: the opportunity to give the account a name and a savings target has increased deposits by as much as 50%

•

Peru, Bolivia, Philippines: SMS reminders to save

•

Philippines: to enable beneficiaries to embark on savings (deposit accounts, savings schemes, etc.) increased deposits may even help stop smoking!

Agriculture and Food security: allow farmers to commit to use fertilizers

 Fertilizers can increased yields by 70% in western Kenya, but only 40% of farmers use them  http://www.povertyactionlab.

org/evaluation/nudging farmers-use-fertilizer experimental-evidence-kenya

Coming to sell them at the time of harvest for fertilizer to be delivered (using vouchers) in the next planting season increased fertilizer use, with a subsidy, by 50%.

Decentralization / Governance: still no clear answer to the question of community control

 There is a great debate on the issue of community involvement in monitoring corruption.

Our results differ  • Indonesia: audits by an external official authority were more effective than community controls to limit corruption in the construction of local roads • Kenya: temporary teachers employed and supervised by a local committee were more efficient than teaching staff.

• India: no impact for local committees to control schools.

• Uganda: involvement of citizens in the functioning of health facilities improved attendance, visits, and has an impact on health indicators.

http://www.povertyactionlab.org/policy-lessons/governance/community-participation

We need more studies in this area!

Thank you for your attention!

AGENDA

I.

Introduction to IPA II.

Impact evaluation III.

Lessons from evaluations: Africa & beyond IV.

Discussion and exercise: designing an evaluation for your sector

Contact information

www.poverty-action.org

Email : [email protected]

Tel : (233) 026 806 8213 IPA Ghana Ako Adjei, Osu PMB 57 Accra 0302 745 570