Transcript Impact Evaluation in Ethiopia Arianna Legovini Africa Impact Evaluation Initiative www.worldbank.org/africa/impact

Impact Evaluation in Ethiopia
Arianna Legovini
Africa Impact Evaluation Initiative
www.worldbank.org/africa/impact
Addis Ababa
June 16, 2006
The Africa Impact Evaluation Initiative
• OBJECTIVES
– Build government capacity for evidence-based
policy-making, and
– Mainstream rigorous impact evaluation in Banksupported operations
• APPROACH
– THEMATIC : to learn how similar innovations
perform in different contexts and apply these
lessons to scale up successful experiences
– COUNTRY : to build in-country capacity to
measure the cost-effectiveness of different
interventions and use this information to
improve programs and policies
– SECTOR/PROJECT: to test innovations in the
short run and provide early feedback into
program design and roll-out
Education in Kenya: addressing basic health
problems
Evaluations of cost-effectiveness require knowledge of a
program’s costs as well as its impact.
Because the Kenyan programs were conducted in similar
environments, cost-effectiveness estimates from these
randomized evaluations can be readily compared.
• Deworming was found to be extraordinarily cost-effective at
only $3.50 per additional year of schooling.
• Provision of free uniforms would cost $99 per additional
year of school participation induced.
• The school meals program, which targeted preschoolers,
cost $36 per additional year of schooling induced.
• In other words, $10000 of public resources put in:
– Deworming will keep 2800 children in school for an additional
year
– Uniforms or school feeding will keep only 100-277 children in
school for an additional year
• Kenya adopted de-worming treatment as part of its
Universal Primary Education strategy
Africa Impact Evaluation Initiative
THEMES & COUNTRIES
• Infrastructure for Africa
– Ethiopia, Malawi, Mozambique, South Africa, Tanzania
• Strengthening Safety-nets
– Angola, Ethiopia, Gambia, Kenya, Madagascar, Malawi, Sierra
Leone, Tanzania, Zambia
• Promoting Inclusion and Access to Justice
– Ethiopia, Mozambique, Niger, Uganda
• Education: what works and how
– Kenya, Madagascar, Rwanda, Sierra Leone, Uganda
• Health: what works and how
– Ethiopia, Mali, Mozambique, Rwanda, Uganda
• Roll-Back Malaria: improving take-up
– Kenya, Congo, Eritrea, Ethiopia, Nigeria, Senegal, Sudan
• Agriculture & environment: strengthen sustainable dev’t
– Kenya
Ethiopia Impact Evaluation Program
OBJECTIVE
• Support government to evaluate progress of
the PASDEP
BY
• Generating sound evidence on what works to
achieve the targets in the PASDEP,
• Building capacity in the country and among
implementing institutions for evidence-based
programs and policies, and thus,
• Securing longer-term effectiveness of
programs in achieving PASDEP outcomes.
How does Impact Evaluation fit within the
M&E strategy of the government?
• Monitoring is the best tool to track
input to output changes
(implementation efficiency)
• Monitoring is not adequate to establish
output to outcome linkages
• Impact evaluation help identify the
causal link between outputs and
outcomes, the size of the effect and its
cost (policy effectiveness)
• Monitoring and evaluation are
complementary tools
Alignment with PASDEP
1. Agriculture and rural development
2. Food Security
3. Infrastructure
4. Health
5. Education
6. Governance
7. Environment
8. Private sector and exportations
development
9. Tourism and culture
10. Mining
Ethiopia Impact Evaluation Program
• Food security
→ Evaluation of the national food security program
• Infrastructure
→ Evaluation of Electricity Access Expansion
→ Evaluation of ICT Expansion
→ Poverty Impact of Transport Operations on Road
Corridors
2
2
1
2
• Health
→ Evaluation of the Community-Based Child Growth
Promotion
→ Evaluation of the health extension package and
health extension workers (Basic Services Project
component)
→ Evaluation of Malaria Booster Program
• Governance
→ Basic Services Project
→ Women Development Initiative project
→ Gender & Law program
2
1
1
1
3
1
Health
Evaluation of the Community-Based Child
Growth Promotion
•
•
•
•
•
In 2005, the Government of Ethiopia initiated the
Child Growth Promotion Component (CGPC) of
the broader Food Security Project (FSP).
The CGPC seeks to improve the nutritional
status of children below 2 years of age through
growth monitoring and advice to the caregivers.
A baseline survey was conducted in July and
August 2005 of 5706 children in 7 woredas/70
kebeles of SNNPR.
The sample is divided into two groups, both of
which receive the core FSP services. One of the
groups will also receive CGPC.
The baseline provides background information
on the nutritional status and practices prior to the
CGPC as well as some information on general
FSP service delivery.
Health
Protection of Basic Services (PBS)
The PBS supports Ethiopia to make continued
progress toward its MDG targets for health,
education, and water, by:
•
Scaling up financial resource available at the
local government level for the delivery of
basic services, and
•
Increasing the distribution of commodities
(such as immunization, family planning and
malaria control) to health providers.
Health
The evaluation will seek to establish:
•
Whether the PBS improves the allocation of funds
across sectors and government levels, and the
availability of resources to front-line services
(before-after comparison using PETS/QSDS at
municipal and facility level).
•
The extent to which approaches to strengthen
voice of citizens and civil society groups bring
about positive change in service delivery (before
and after and relative to control using household
surveys of perceptions/satisfaction).
•
The extent to which procurement of commodities
improve front line availability and use of preventive
health (before and after).
Malaria
• Malaria Booster Programs scale-up
malaria control efforts through the
delivery of
– Insecticide treated nets (ITNs)
– Effective anti-malarial medication (ACTs),
and
– Intermittent preventive treatment (IPT)
Malaria
• While the causal link between use of
ITNs and effective anti-malarial
medication and improved malaria
outcomes IS well established,
• The relative effectiveness of alternative
delivery mechanisms and intervention
packages to induce a change in
people’s health behavior for the
adoption and use of ITNs and effective
anti-malarial medication IS NOT
Malaria
• The evaluation will test alternatives for their
impact on adoption behavior across several
country contexts.
• Alternative delivery modes include:
–
–
–
–
House to house visits
Mass ITN campaigns
ITN distribution through existing prime care services
Health vs school system-based delivery
• And different emphasis on:
– Improving human capital
– Improving physical capital
– Strengthening social capital
Malaria
• Impact would be measured on:
– Ownership of ITNs by household
– Use of ITNs by children under five,
pregnant women, and households
– Children under five with fever receiving
treatment with effective anti-malarial
medication within 24 hours of the onset of
symptoms
– Percentage of pregnant women receiving
intermittent preventive treatment (IPT)
– All cause under-five mortality
– Household income and consumption
Access to Justice
Gender & Law
This regional program builds capacity for
governments and civil society partnerships to
support:
• gender-responsive legal reform and
institutional and policy environment;
• gender-responsive legal education and
literacy activities at the grassroots level;
and
• improved access to justice and legal
services for the poor and socially
vulnerable
Access to Justice
Gender & Law
The program has been implemented in 13
SSA countries and is planned for expansion.
During this new phase the program plans to
develop a rigorous framework of evaluation to
measure to what extent the program is an
effective mechanism for expanding women’s
access to justice and to learn what features of
the program are most successful.
The evaluations are planned to take place in a
number of countries (Niger, Uganda and
proposed Ethiopia) from which a series of
Access to Justice
Gender & Law
Different communities will be randomly
assigned to alternative :
• Training and capacity building packages
for legal service providers and
practitioners, and
• Direct provision of legal assistance (new)
Impact will be measured on indicators of
access to justice, outcomes of legal
disputes, and economic outcomes.
Governance
Women Development Initiatives
• The project, in operation since 2001,
facilitates the formation and
strengthening of self-help women’s
groups, to enhance women’s capacity
to act collectively and increase the
social and economic welfare of their
households.
Women Development Initiatives
• When the evaluation started, the project was already
under implementation, and therefore we could not take
a baseline
• There were doubts that a rigorous counterfactual could
be established
• But, during the first year of implementation, the project
had rationed participation (one group of max 30
women per kebele)
• The officials stated that to keep the number of
participants under 30, they had carried out a lottery.
• This offered an opportunity
• We tracked down the women who had been rationed
out (from official records and participant recollection) &
• Administered a 1000 household survey to a random
sample of participants and non-participants
Some findings from WDIP mid-term
evaluation: Participatory stage
• Economic:
– Exclusion from many sectors of economic
activity
– Restricted freedom of movement
– Wage gap and low bargaining power
– Poor access to education, transport, financing
– No access to water, electricity
– Time diverted to household chores--in rural
areas women report spending 3-4 hours a day
collecting water and grinding grains
• Judicial:
– Traditional court system upholding tradition
rather than national law
– No access to formal courts
– Limited recourse
Some findings from WDIP mid-term
evaluation: Participatory stage
• Political:
– Political representation and government male
dominated
– Women involved in social activities (iqub, idir),
seldom comfortable to participate in male
dominated political gatherings
• Household & social:
– Traditional roles and mores
– Prevalence of harmful practices (FGM, early
marriage,..)
– Domestic violence common and accepted
Some findings from WDIP mid-term
evaluation: Quantitative analysis
• Economic √
– WDIP participants earn 100 birr more on
average per month than non participants
– Income, in turn, significantly increases
women’s say on household purchases
• Project cost-effectiveness
– Assuming the gains can be sustained for 5
years, average group size of 12, a transfer of
$4,000 per group, and 40% project overhead,
the Internal Rate of Return for this project
would be about 15%.
Some findings from WDIP mid-term
evaluation: Quantitative analysis
• Mental health √
– On average women report being neither happy
nor sad (3 on a scale of 1-5). 40% of women
show signs of depression.
– WDIP participants are happier. WDIP adds 0.7
to the mean, when controlling for several
variables including employment (+1.1), income
(no effect) and social capital (+.3 per additional
person they can rely on))
– WDIP effect on happiness is not economic
Some findings from WDIP mid-term
evaluation: Quantitative analysis
• Social customs
– Fewer participants (about 6 percentage
points) uphold traditional rules that
women should not
• Farm (plough)
• Ride animals or carts
• Walk alone at night
– Yet more participants uphold their
husband right to beat them
Some findings from WDIP mid-term
evaluation: Quantitative analysis
• Traditional harmful practices: spill over?
– Even though many more WDIP participants
received training on THP, more of them are
likely to practice them
• Domestic violence
– WDIP participants are as likely to suffer from
domestic violence. The women who chose
their husband suffer less domestic violence.
• Political awareness
– WDIP participants are as politically aware as
non-participants (on political representation,
civil rights, etc). Rural women are more aware.
Some findings from WDIP mid-term
evaluation: Quantitative analysis
• Participant recommendations to
improve project:
– Speed up delivery of funds
– Relax minimum group size condition
– Strengthen technology transfer and
transfer of ideas
Follow-up WDIP evaluation
• Collect follow up survey data and
evaluate impact at project end
• Come up with detailed project costs
to estimate returns to the project
accurately
THANK YOU