Transcript Sanitation and Child Health in Urban Slums

Are Disasters Any Different?
Challenges and Opportunities
for Post-Disaster Impact Evaluation
Alison Buttenheim, Princeton University
Howard White, 3ie
Rizwana Siddiqui, PIDE
Katie Hsih, Princeton University
April 1, 2009
Cairo
3ie post-disaster impact evaluation
(PDIE) study
Motivation:
•
Frequency and severity of natural disasters
•
Quantity of assistance provided in postdisaster settings
•
Recent interest from humanitarian and
development sectors in more and better
impact evaluation
•
Opportunity to use Pakistan ERRA experience
2
as case study
3ie post-disaster impact evaluation
(PDIE) study
Goals:
•
Review existing approaches to PDIE
•
Develop a framework for rigorous PDIE
•
Apply framework to the 2005 Pakistan
earthquake case
•
Identify a set of principles to guide PDIE
3
Disasters
Natural events:
414 reported in 2007
(CRED criteria)
4
Disasters
Natural events:
414 reported in 2007
(CRED criteria)
Human consequences:
211 million affected
16,847 lives lost
USD 100+ billion damages
5
Disasters
Natural events:
414 reported in 2007
(CRED criteria)
Human consequences:
211 million affected
16,847 lives lost
USD 100+ billion damages
Institutional
responses
6
Post-disaster relief and recovery efforts
•
USD 5.9 billion (pledged) for 2005 Pakistan
earthquake
•
USD 13.5 billion (pledged) for 2004 Indian
Ocean tsunami
•
Actors: Diverse mix of governments,
funders, IFIs, aid agencies, humanitarian
agencies, int’l/local NGOs.
7
How does PD assistance get evaluated?
•
Extensive process evaluations
•
Multiple levels of analysis (project, agency,
sector, disaster)
•
Some joint evaluations (e.g. TEC)
•
Review of ALNAP database, etc. suggests
few examples of “rigorous” impact evaluation
8
Why so little focus on IE in PD settings?
9
Why so little focus on IE in PD settings?
“Disasters “Disasters
“Disasters
are “Disasters
are“Disasters
are
are
are
different” different”
different”
different”
different”
“Disasters
are
different”
10
Are disasters any different?
1. Unpredictable, rapid-onset event
2. Proven life-saving measures cannot be
randomized or withheld
3. Mismatch between resources and need
(sometimes)
4. Absence of baseline data (usually)
5. Which counterfactual is the right one?
11
Are disasters any different? Maybe not…
1. Nonrandom exposure to disaster event and
consequences
2. Nonrandom assignment of interventions
3. Fragile states/vulnerable populations
4. Multiple concurrent interventions
5. Which counterfactual is the right one?
12
Lessons learned from other PDIE
experiences
•
Bangladesh floods, 1998
•
Hurricane Mitch, 1998
•
Indian Ocean tsunami, 2004
•
Hurricane Katrina, 2005
13
Disaster-related time periods
Emergency
DISASTER
Relief
Pre-disaster
t-1
Recovery/Reconstruction
Immediate
postdisaster
t0
Postintervention
(1)
t1
Postintervention
(2)
t2
14
Disaster-related populations
A
Disaster-affected households* that
receive assistance or interventions
B
Disaster-affected households that do
not receive assistance or
interventions‡
C
Non-affected households† that were
similar to A before the disaster
* or communities (or other unit of analysis)
‡ or receive them later, or receive different ones
† or less-affected households/communities
15
Within treatment group, single-difference
over time
Time
Description
Disaster-affected
households, treated
t0-t-1
Disaster-related losses
A0-A-1
t1-t-1
Restoration to baseline
A1-A-1
t1-t0
Recovery from disaster
A1-A0
t2-t-1
Sustained restoration to baseline
A2-A-1
t2-t0
Sustained recovery from disaster
A2-A0
t2-t1
Persistence of recovery
A2-A1
16
Within treatment group, single-difference
over time
Time
Description
Disaster-affected
households, treated
t0-t-1
Disaster-related losses
A0-A-1
t1-t-1
Restoration to baseline
A1-A-1
t1-t0
Recovery from disaster
A1-A0
t2-t-1
Sustained restoration to baseline
A2-A-1
t2-t0
Sustained recovery from disaster
A2-A0
t2-t1
Persistence of recovery
A2-A1
ERRA:
“Build
Back
Better”
17
Within treatment group, single-difference
over time
Time
Description
Disaster-affected
households, treated
t0-t-1
Disaster-related losses
A0-A-1
t1-t-1
Restoration to baseline
A1-A-1
t1-t0
Recovery from disaster
A1-A0
t2-t-1
Sustained restoration to baseline
A2-A-1
t2-t0
Sustained recovery from disaster
A2-A0
t2-t1
Persistence of recovery
A2-A1
Problems: Recall bias if no baseline; attribution?
18
Cross-sectional, single-difference over
treatment groups (A vs. C)
Time
Description
Affected treatedNon-affected
t-1
Baseline (pre-disaster)
A-1-C-1
t0
Emergency (immediate post-disaster)
A0-C0
t1
Relief/Reconstruction (post-intervention #1)
A1-C1
t2
Recovery (post-intervention #2)
A2-C2
19
Cross-sectional, single-difference over
treatment groups (A vs. C)
Time
Description
Affected treatedNon-affected
t-1
Baseline (pre-disaster)
A-1-C-1
t0
Emergency (immediate post-disaster)
A0-C0
t1
Relief/Reconstruction (post-intervention #1)
A1-C1
t2
Recovery (post-intervention #2)
A2-C2
20
Cross-sectional, single-difference over
treatment groups (A vs. C)
Time
Description
Affected treatedNon-affected
t-1
Baseline (pre-disaster)
A-1-C-1
t0
Emergency (immediate post-disaster)
A0-C0
t1
Relief/Reconstruction (post-intervention #1)
A1-C1
t2
Recovery (post-intervention #2)
A2-C2
Implied counterfactual: What would “A”
households look like if there had been no disaster?
21
Cross-sectional, single-difference over
treatment groups (A vs. C)
Time
Description
Affected treatedNon-affected
t-1
Baseline (pre-disaster)
A-1-C-1
t0
Emergency (immediate post-disaster)
A0-C0
t1
Relief/Reconstruction (post-intervention #1)
A1-C1
t2
Recovery (post-intervention #2)
A2-C2
Problems: Is there an appropriate “C” group? If so,
were they observed? Attribution?
22
Difference-in-difference (A vs. C)
Affected — Unaffected
Time
Description
t0-t-1
Disaster-related losses
(A0-A-
— (C0-C-1)
t1-t-1
Restoration to baseline
1)
— (C1-C-1)
(A1-A1)
t1-t0
Recovery from disaster
(A1-A0) — (C1-C0)
t2-t-1
Sustained restoration to baseline
(A2-A-
— (C2-C-1)
t2-t0
Sustained recovery from disaster
1)
— (C2-C0)
t2-t1
Persistence of recovery
(A2-A0) — (C2-C1)
(A2-A1)
23
Difference-in-difference (A vs. C)
Affected — Unaffected
Time
Description
t0-t-1
Disaster-related losses
(A0-A-
— (C0-C-1)
t1-t-1
Restoration to baseline
1)
— (C1-C-1)
(A1-A1)
t1-t0
Recovery from disaster
(A1-A0) — (C1-C0)
t2-t-1
Sustained restoration to baseline
(A2-A-
— (C2-C-1)
t2-t0
Sustained recovery from disaster
1)
— (C2-C0)
t2Controls
-t1
Persistence
of recovery
2-A0) — (C2-C1)
time-variant
factors that are the(Asame
(A2-A1)
between A & C
24
Cross-sectional, single-difference
over treatment groups (A vs. B)
Time
Description
Affected treatedAffected control
t-1
Baseline (pre-disaster)
A-1-B-1
t0
Emergency (immediate post-disaster)
A0-B0
t1
Relief/Reconstruction (post-intervention #1)
A1-B1
t2
Recovery (post-intervention #2)
A2-B2
25
Cross-sectional, single-difference
over treatment groups (A vs. B)
Time
Description
Affected treatedAffected control
t-1
Baseline (pre-disaster)
A-1-B-1
t0
Emergency (immediate post-disaster)
A0-B0
t1
Relief/Reconstruction (post-intervention #1)
A1-B1
t2
Recovery (post-intervention #2)
A2-B2
26
Cross-sectional, single-difference
over treatment groups (A vs. B)
Time
Description
Affected treatedAffected control
t-1
Baseline (pre-disaster)
A-1-B-1
t0
Emergency (immediate post-disaster)
A0-B0
t1
Relief/Reconstruction (post-intervention #1)
A1-B1
t2
Recovery (post-intervention #2)
A2-B2
Implied counterfactual: What would “A” households look
like if there had been no intervention?
27
Cross-sectional, single-difference
over treatment groups (A vs. B)
Time
Description
Affected treatedAffected control
t-1
Baseline (pre-disaster)
A-1-B-1
t0
Emergency (immediate post-disaster)
A0-B0
t1
Relief/Reconstruction (post-intervention #1)
A1-B1
t2
Recovery (post-intervention #2)
A2-B2
Problems: How were interventions assigned to A but not
to B?
28
Difference-in-difference (A vs. B)
Time
Description
Disaster-affected
households
“Treated”
— “Control”
t0-t-1
Disaster-related losses
(A0-A-
— (B0-B-1)
t1-t-1
Restoration to baseline
1)
— (B1-B-1)
(A1-A1)
t1-t0
Recovery from disaster
(A1-A0) — (B1-B0)
t2-t-1
Sustained restoration to baseline
(A2-A-
— (B2-B-1)
t2-t0
Sustained recovery from disaster
1)
— (B2-B0)
29
Difference-in-difference (A vs. B)
Time
Description
Disaster-affected
households
“Treated”
— “Control”
t0-t-1
Disaster-related losses
(A0-A-
— (B0-B-1)
t1-t-1
Restoration to baseline
1)
— (B1-B-1)
(A1-A1)
t1-t0
Recovery from disaster
(A1-A0) — (B1-B0)
t2-t-1
Sustained restoration to baseline
(A2-A-
— (B2-B-1)
Sustained recovery from disaster
)
— (B2-B0)
Controls
time-variant factors that are the 1same
30
t2between
-t1
Persistence
(A2-A0) — (B2-B1)
A & B of recovery
t2-t0
World Bank impact evaluation of housing and
livelihood grants
31
World Bank impact evaluation of housing and
livelihood grants
•
Instrumental variable approach to disaster
impact:
–
–
Villages at same distance from epicenter, at
same elevation and slope had comparable predisaster SES
Villages at different distance from fault line
experienced different earthquake severity.
32
World Bank impact evaluation of housing and
livelihood grants
•
Instrumental variable approach to disaster
impact:
–
–
Villages at same distance from epicenter, at
same elevation and slope had comparable predisaster SES
Villages at different distance from fault line
experienced different earthquake severity.
A1-C1
33
World Bank impact evaluation of housing and
livelihood grants
•
Instrumental variable approach to disaster
impact:
–
–
•
Villages at same distance from epicenter, at
same elevation and slope had comparable predisaster SES
Villages at different distance from fault line
experienced different earthquake severity.
A1-C1
Variation in receipt of relief and recovery
funds:
–
–
Between-district variation in implementing
agency for housing grant
Threshold eligibility for livelihoods grant of 5
dependents/households: regression continuity
design.
34
World Bank impact evaluation of housing and
livelihood grants
•
Instrumental variable approach to disaster
impact:
–
–
•
Villages at same distance from epicenter, at
same elevation and slope had comparable predisaster SES
Villages at different distance from fault line
experienced different earthquake severity.
A1-C1
Variation in receipt of relief and recovery
funds:
–
–
Between-district variation in implementing
agency for housing grant
Threshold eligibility for livelihoods grant of 5
dependents/households: regression continuity
design.
A1-B1
35
ERRA impact evaluation case study
1.
Evaluation opportunities using existing data & HH
sample
–
Household data collection at t2
–
Retrospective household reports of t0
–
Use of ongoing government household surveys (e.g., HIES)
as baseline
–
Randomization of some interventions from 2009 
36
ERRA impact evaluation case study
2.
Evaluation opportunities in a future disaster
–
Maintain surveillance sample in disaster-prone regions
–
Household-level data collection at t0
–
Randomized interventions, e.g,
•
Timing of interventions:
–
Group 1: Housing grant first, followed by livelihood cash grant
–
Group 2: Livelihood cash grant first, followed by housing grant
•
Conditionality of grants
•
Types of interventions, e.g, different formats or recipients of
livelihoods cash grant
37
PDIE Guiding Principles
1.
PDIE is necessary to ensure that relief and recovery
funds are appropriately targeted, effective, and
efficient.
2.
Each phase of a disaster (emergency, relief,
recovery/reconstruction) presents distinct evaluation
challenges and therefore may require a different
evaluation approach or methodology.
3.
“Evaluation preparedness” is an important part of
disaster preparedness.
38
PDIE Guiding Principles
4.
PDIE should incorporate evaluation of (pre-disaster)
investments in disaster mitigation, prevention, and
resilience.
5.
Rigorous PDIE requires the tools and perspectives of
multiple disciplines and sectors.
6.
Quantitative PDIE can benefit from the qualitative and
mixed-methods approaches.
39
PDIE Guiding Principles
7.
Proportionate changes in outcomes over time and
over groups can be as instructive as changes in
levels.
8.
Change-over-time impact evaluations should
recognize two distinct baselines: pre-disaster, and
immediately post-disaster.
40
PDIE Guiding Principles (ct’d)
9.
PDIE will be most successful when the goals of the
intervention are clearly defined through a logical
framework or similar model; when the interventions
are appropriately targeted, and when the purpose/use
of the evaluation is clear.
10. Experimental and quasi-experimental approaches are
feasible in PDIE if ethical, logistical and “fit” issues are
adequately addressed.
41