Transcript Document

Institute for International Programs
ASADI V
Accra, November 2009
Jennifer Bryce
Institute for International Programs
The Johns Hopkins University
Outline – What have we learned?
1. From the evaluation of the ACCELERATING
CHILD SURVIVAL AND DEVELOPMENT (ACSD)
Program?
2. From prospective evaluations of the
CATALYTIC INITIATIVE TO SAVE ONE MIIILION LIVES
(CI) to date?
3. The way forward
ACSD, 2002-2005
 11 countries in Africa
 Support from CIDA and other
partners
 Implemented through UNICEF
 Aim: To reduce mortality among
children less than 5 years of age
Accelerated Child Survival and Development
CIDA funded project
Mauritania
Mali
Cape Verde
Niger
Senegal
Gambia
Guinea Bissau
Chad
Burkina Faso
Benin
Guinea
Togo
Sierra Leone
Liberia
Nigeria
Côte d’Ivoire
Ghana
Cameroon
Central African
Republic
Equatorial Guinea
 Strategy: Accelerate coverage
with three packages of highimpact interventions, with a
special focus on communitybased delivery
Sao Tome &
Principe
High Impact Package
EPI + Expans ion
Gabon
Congo
Congo - Democratic Republic
ACSD Program:
Intervention packages
EPI+
 Vaccinations
 Vitamin A supplementation
 ITNs for U5s & pregnant women
 De-worming
ANC+
– Malaria prevention in pregnant
women (IPTp)
– Tetanus Toxoid
– Iron/folic acid supplementation
– Vitamin A post-partum
– PMTCT
IMCI+
 Facility IMCI
 Community case management
(CCM) of childhood illnesses



Diarrhea: oral rehydration therapy
(ORT)
Malaria: based on current policy
Pneumonia: referral to facility
 Promotion of timely initiation of
breastfeeding, exclusive
breastfeeding to 6 months,
timely complementary feeding
 Promotion of household
consumption of iodized salt
The retrospective independent
evaluation of ACSD
 High-impact districts in
Benin, Ghana, Mali
 Standard indicators
 Existing DHS/MICS with
oversampling
 National comparison areas
 Documentation of program
implementation & contextual
factors
 No cost component
 Stepwise design
ACSD Implementation: GHANA
EPI
+
ITN
Activities
IPTp
Facility &
Community
IMCI
Facility
Community
2001
2002
Limited
Coverage
2003
2004
2005
Lmt’d ITNs Available
Partial
Coverage
2006
2007
Key: Bars represent districts in the following order: Builsa, Bawku East, Kasena-Nankana,
Bolgatanga, Bawku West, Bongo
Coverage for EPI+ interventions
before and after ACSD, in HIDs
Benin
ITNs
10
6
*
*
*
61*
26*
After ACSD
*
63
60
DPT3
Before ACSD
Mali
51
49
Measles
Vitamin A
Ghana
Key
*
*
*
*
Increases in coverage across the board in Ghana and Mali;
Benin achieved increases for vitamin A and ITNs.
*Change was significant at p < 0.05.
Coverage for IMCI+ interventions
before and after ACSD, in HIDs
Benin
Ghana
Key
Before ACSD
After ACSD
Mali
*
*
*
*
*
*
*
No coverage gains, and some significant losses, in sick child care.
Exclusive breastfeeding increased in Ghana, declined in Mali.
*Change was significant at p < 0.05.
Coverage for ANC+ interventions
before and after ACSD, in HIDs
Benin
Ghana
3+ antenatal care
visits
IPTp with SP
**
7
*
Skilled attendant
at delivery
Postnatal vit A
5
*
38
*
*
*
*
76
74
After ACSD
*
0
44
55
Before ACSD
Mali
71
64
Tetanus Toxoid
Key
*
*
*
*
Ghana and Mali improved care for childbearing women; delivery of TT
and postnatal vit A benefited from EPI system in Mali.
*Change was significant at p ≤ 0.05.
** Measured level was 28%, but country team reported this was incorrect as IPTp had not been implemented in 2001.
Under-five mortality in the ACSD HIDs
19% (p=0.10)
Under-five mortality in the ACSD HIDs
and national comparison areas
Declines in U5M in ACSD focus districts,
but not greater than national comparison areas.
Under-five mortality (per 1000 live births)
280
Jul
1999Jun
Jan 2004Dec 2006
Jul 1998Dec 2001
Jan 2004Jul 2007
No changes in child
nutritional
status
Jul 1998Jul 2003Dec 2001
Dec 2006
attributable
to ACSD.
240
260
200
Comparison area
Comparison area
248
160
197
172
120
141
145
123
80
109
107
86
40
0
Benin
Ghana
Mali
Did ACSD implementation contribute to
reducing inequities?
Baseline sample sizes
too small to support
analysis of equity trends
in Benin or Ghana.
Socioeconomic inequalities, showing
breakdown by wealth quintiles of ANC 3+
coverage in ACSD “high-impact” zones and
the comparison area, Mali, 2006-7.
3+ antenatal visits
100%
HID (before)
Comp (before)
HID (after)
Comp (after)
80%
Coverage (%)
Yes, in Mali, where
socioeconomic and
urban/rural inequities
decreased more in the
ACSD HIDs than in the
comparison area.
60%
40%
20%
0%
Poorest
2nd
3rd
Wealth quintiles
4th
Richest
Conclusions & implications
1.
Intervention coverage CAN be accelerated if there is
adequate funding & human resources.
2.
Acceleration of mortality declines require:
a)
Focus on interventions that have a large and rapid impact on major
causes of child death
b)
Sufficient time to fully implement approach and for coverage to
translate into declines in mortality
c)
Reasonable expectations, given level of resources
► Work for closer match between program resources &
cause of death
► Be realistic about what can be accomplished
► Level of funding matters
Conclusions & implications
3.
Policy barriers prevented key ACSD interventions
directed at pneumonia and malaria from being fully
implemented.
4.
Breakdowns in commodities and gaps in funding stall
progress toward impact.
5.
More attention and operations research needed on
incentives and supports for community-based workers
► Work for policy reform as first step, where needed
► Pay attention to health systems supports such as
commodities, supervision, & incentives
Contributors & acknowledgements
Contributors
Jennifer Bryce
Kate Gilroy
Elizabeth Hazel
Gareth Jones
Robert Black
Cesar Victora
Acknowledgements
Ministries of Health, National
Statistics Offices, UNICEF
country staff, Collaborators in
documentation
UNICEF regional and global
staff Genevieve Begkoyian,
Mark Young, Sam Bickel
Technical consultants Trevor
Croft, Macro International
UNICEF leadership For their
commitment to learning and
change
Part 2
EVALUATING THE
CATALYTIC INITIATIVE
TO SAVE A MILLION LIVES
Two Linked Evaluations
The Catalytic Initiative
Independent Evaluation of the
MNCH Rapid Scale-Up
“Real-time”
Mortality Monitoring (RMM)
 Overall objective: Provide
“proof of concept” that proven
interventions can be scaled up
rapidly to reduce newborn and
child mortality.
 Supported by: BMGF
 Implementing partners:
Governments and UNICEF,
WHO, UNFPA
 Overall objective: To monitor
changes in under-five mortality
in real-time.
 Countries: Burkina Faso,
Malawi, Mozambique
 Countries: Ghana, Malawi,
Mali, Mozambique
 Supported by: CIDA
 Implementing partner:
Governments and UNICEF
Process of evaluation design
Assessment visits
• Learn about CI implementation plans
• Identify in-country evaluation counterpart
• Assess current monitoring
Estimation of program impact
• Standard template for assessment of plans
• Application of Lives Saved Tool (LiST)
• Feedback to implementers to strengthen program plan & supports
Design of Evaluation
• Comparison areas?
• Sufficient time for scale-up to population coverage?
• Cost and feasibility?
Prepare
• Stakeholders’ meetings
• Ethical clearances
• Formative research
Progress: Malawi
In-country partners: Centre for Social Research
and National Statistics Office
Implementation
Evaluation (full)
 Features of accelerated approach:
 Mortality monitored by:
 Government-paid CHWs trained
to deliver CCM for pneumonia,
malaria, diarrhea (including zinc)
 Strengthening district health
management
 Implementation status:
 In 10 intervention districts, 5-15%
of CHWs trained by June 2009
Having CHWs report vital
events
Calibrating facility deaths
against community deaths
Two rapid survey approaches
 Full documentation of program &
contextual factors
 Quality of care assessments at 1stlevel facilities and for CHWs
 Costs & equity tracked
 12 districts: 6 “accelerated” and 6
routine
 National platform approach under
discussion
Progress: Mozambique
In-country partner: Eduardo Mondlane University
Implementation
Evaluation (full)
 Features of accelerated approach:
 Mortality monitoring by calibrating
facility to community deaths.
 Increased access to quality child
health care in facilities
 Quality of immunization services
improved
 Stepped-wedge design based on
scheduled cohorts for Rapid Scale Up
 Long-lasting insecticide-treated
nets (ITNs) distributed and used
 Vitamin A supplementation
 Breastfeeding promotion
 BMGF funds used to fill gaps in
maternal health
 Implementation status:
 CI planned to be implemented in
33 districts each year from 2008
to 2012
 Documentation of baseline health &
nutrition, inputs & contextual factors and
coverage for each cohort using national
evaluation platform approach
 Will support dose (program intensity) –
response (coverage & modeled impact)
analyses
Progress: Burkina Faso
In-Country Partner: Institut Supérieur des Sciences de la Population (ISSP)
Implementation
Evaluation (no RMM)
 Features of accelerated approach:
 National platform approach in 9
intervention and 2-3 comparison districts
 Volunteer community health workers
providing:
• CCM for diarrhea (ORT + zinc) and
malaria (ACT) in 9 districts
• CCM for pneumonia in 2 districts
 Strengthened district planning and
supervision
 Implementation status:
 Materials ready; cascade training of all
CHWs in 9 districts to be completed
before end 2009
 New “LiST” survey to collect districtlevel estimates of coverage for proven
MNCH interventions
 Modeled mortality using LiST
 Analysis using pre-/post-intervention
with comparison and/or dose response
Lesson 1:
Existing plans include high-impact interventions…
CCM
(Pn/Mal/
Dia)
Facility
IMCI
Delivery
Care
IPT
Burkina
Faso



Ghana

Malawi


Water/
Sanitation
Modeled
 in
U5M
(LiST)*
ITNs
Vitamin
A
Excl. BF
Vaccines
(Measles/
Hib)












26%







36%

28%
Mali





Mozambique





*If targets fully achieved at adequate service quality.
24%
26%
…but if feasibility and speed are issues,
just 4 or 5 interventions can achieve ≥ 20%
reduction in U5M by 2015
Number of interventions included in
national plan
Number to ≥ 20% reduction in U5M
Malawi
Burkina Faso
Ghana
18
13
20
4
5
5



(29; 67)
(30; 50)
(33; 60)



(55; 85)
(41; 60)
(42; 60)



(23; 69)
(10; 70)
(40; 55)



(27; 69)
(48; 57)
(65; 70)
Interventions (current and target coverage levels)
Pneumonia treatment with antibiotics
Diarrhea treatment with ORS and zinc
Malaria prevention with insecticidetreated nets
Malaria treatment with ACTs
Vitamin A supplementation

(67; 90)
Improved sanitation

(18; 70)
Pre-publication results; not for citation or distribution
Lesson 2:
Implementation takes time
Implementation status of functional village health clinics
with CHW trained in CCM, Malawi CI districts, June 2009
(18 months after project start-up)
CHW
CI
Total
training
District
CHWs
target*
Karonga
273
60
Mzimba
725
406
Kasungu
616
198
Dedza
624
120
Ntcheu
474
120
Lilongwe
1228
250
Chiradzulu
291
40
Balaka
317
61
Nsanje
238
66
Phalombe
313
61
*1 trained CHW per village health clinic
Trained
to date
37
48
72
45
42
40
37
44
66
39
Drug kit
available
37
46
54
23
39
39
27
39
36
36
% CHWs
trained & with
drugs
available
14%
6%
9%
4%
8%
3%
9%
12%
15%
11%
…especially when policy reform is
needed.
In Mali, the MoH scheduled a “forum” to decide
on CCM for childhood pneumonia & malaria.
Original
date
(cancelled)
Planned
(cancelled)
July
2008
Months
Forum held;
agreed
“YES” on
Planned
CCM
(cancelled)
November
2008
4
+
March
2009
February
2009
3
+
Discussions
about how to
implement are
still under way
1
=
7 months
in a 3-year
CI project
November
2009
Lesson 3:
“Virgin” comparison areas do not exist
Simultaneous implementation of
multiple programs
Separate, uncoordinated, inefficient
evaluations, if any
Mozambique
Lesson 4:
There are no shortcuts for mortality
measurement (at least not yet)
 Capturing a 25% difference-in-difference for rates of
child mortality in a two-year period requires a survey of
≈ 12,300 households in each group*
 Promises of measuring declines in 1 year using survival
analysis or other techniques still require these
prohibitively large sample sizes, plus detailed info on
age of death
 CI work on “real-time” mortality monitoring will assess
the validity of alternative methods, but in first trials
require validation against a gold standard
*based on Malawi; sample sizes will increase as mortality rates decrease, e.g. in Ghana
RMM Options by country
Options for RMM methods
Data collection at community level
Paid
Gov’t
health
workers
Mali
Mozambique
Lay
volunteers

Ghana
Malawi
Unpaid
Gov’t
health
workers
Deaths
recorded
in
facilities
vs. community
survey



Vital
registration
program
New
methods
for using
surveys


Vital
events
reporting
at Child
Health
Days





Contributors
In-country partners
Burkina Faso: ISSP, INSP
Ghana: Noguchi Institute,
University of Ghana
Malawi: NSO, CSR, Department
of Economics, University of
Malawi
Mali: CREDOS
Mozambique: Eduardo Mondlane
University
IIP-JHU
Agbessi Amouzou, Abdullah
Baqui, Robert Black, Jennifer
Bryce, Kate Gilroy, Elizabeth
Hazel, Gareth Jones, Marjorie
Opuni, Jeremy Schiefen, Cesar
Victora, Damian Walker
Part 3
THE WAY FORWARD:
NATIONAL EVALUATION
PLATFORMS (NEPS)
What is a national evaluation platform
(NEP)?
 District-level databases covering the entire country
 Containing standard information on:






Inputs (partners, programs, budget allocations, infrastructure)
Processes/outputs (DHMT plans, ongoing training,
supervision, campaigns, community participation, financing
schemes such as conditional cash transfers)
Outcomes (availability of commodities, quality of care
measures, human resources, coverage)
Impact (mortality, nutritional status)
Contextual factors (demographics, poverty, migration)
Permits national-level evaluations of multiple
simultaneous programs
NEPs: A common evaluation framework
 Common principles
(with IHP+, Countdown, etc.)
 Standard indicators
 Broad acceptance
NEPs: Sound evaluation principles
 In-country evaluation counterparts


Local expertise, able to provide continuing evaluation research
support to the MOH
Continuity of inputs from evaluation team; cross-country
network of investigators
 Linked “independence”



Investigators not involved in implementation of MNCH activities
Regular exchange with in-country implementation team
Ongoing activity; not one-off approach
 Attribution by approach


Documentation of all contributions
Comparison of accelerated approach with “routine” approach
What types of questions
can an NEP answer?
? Are programs being deployed where need is greatest?
? Is implementation strong enough to have an impact?
? Did programs increase coverage?
? Was coverage associated with impact?
? How equitable are the programs?
? How much did programs cost?
How can the MOH and partners
use the platform?
To learn from well-performing districts
and guide those doing less well
? Which approaches or combinations are contributing to
rapid scale-up?
? Are some districts more efficient than others? Why?
? Are changes in epidemiology (e.g., due to IRS)
reflected in reallocation of resources in district plans?
Why should you consider a national
platform approach (or not) ?
Advantages
Limitations
 Adapted
Observational design (but no
other alternative may be
possible)
to current reality of
multiple simultaneous
programs/interventions and
partners
 Flexible
design allows for
changes in implementation
 Can
be used to evaluate
multiple programs (child
survival, HIV, malaria,
maternal health, etc.)
 Supports
country ownership
and capacity building
Cost, particularly due to large
size of surveys (!But cheaper
than many standalone surveys!)
Requires transparency and
collaboration by multiple
programs and agencies
Thank you
Further details at www.jhsph.edu/iip
and
www.cherg.org