Transcript Aid - Faculty of Health Sciences
Evaluating Humanitarian Aid:
November 2013 Harry Shannon McMaster University
Working with
• David Sanderson Director, Centre for Development and Emergency Practice (CENDEP), Oxford Brookes University • Charles Parrack, CENDEP
Outline of talk
• • • • • What is (humanitarian) aid Issues in evaluation of aid Shelter after disaster Proposed research agenda for evaluation Discussion Work in Progress!
Humanitarian imperative
• • Helping fellow human beings in (desperate) need “How we think about each other, our relationships, our place in the ecosystem … are the most basic questions … We must recognise humility; trying and doing; governance … given them, don’t wait for the perfect answer.” – James Orbinski, former President of MSF
Maimonides: hierarchy of charitable giving
1. The lowest: Giving begrudgingly and making the recipient feel disgraced or embarrassed.
2. Giving cheerfully but giving too little.
3. Giving cheerfully and adequately but only after being asked.
4. Giving before being asked.
5. Giving when you do not know who is the individual benefiting, but the recipient knows your identity.
6. Giving when you know who is the individual benefiting, but the recipient does not know your identity.
7. Giving when neither the donor nor the recipient is aware of the other's identity.
8. The Highest: Giving money, a loan, your time or whatever else it takes to enable an individual to be self-reliant. charitywatch.org
Definitions
(though boundaries fuzzy) • (Overseas Development) Aid: flows to developing countries and institutions whose aim is to promote development and welfare of the recipient countries.
• Humanitarian aid/action (HA): aid that follows a disaster (complex emergency), typically on a more urgent basis than regular ODA • Contrast with … Humanitarian intervention: the use of military force by a state to end human rights violations by a state (R2P)
Millennium Development Goals, 2000
By 2015: 1. Eradicating extreme poverty and hunger 2. Achieving universal primary education 3. Promoting gender equality and empowering women 4. Reducing child mortality rates 5. Improving maternal health 6. Combating HIV/AIDS, malaria, and other diseases 7. Ensuring environmental sustainability 8. Developing a global partnership for development
Millennium Development Goals
• • • • Specific targets set, e.g., between 1990 and 2015, halve the proportion of people living on less than $1.25
Ban Ki Moon: ‘The Millennium Development Goals (MDGs) have been the most successful global anti poverty push in history.’ However, progress has been uneven Being replaced in 2015 by Sustainable Development Goals
Development aid
• • • • • $140 billion / year Dambisa Moyo, William Easterly - sceptics Jeffrey Sachs – proponent Paul Collier - ?intermediate
Esther Duflo, Abhijit Banerjee – RCTs (Randomized Controlled Trials) • All support humanitarian aid (HA)
Duncan Green is Senior Strategic Adviser at Oxfam GB
Duncan Green, From Poverty to Power
Sceptical view of Canada’s NGOs
Humanitarian aid ‘system’ - some numbers
• • • Estimated 4,400 NGOs worldwide doing some HA (Haiti sometimes called a ‘Republic of NGOs’) 2010: estimated 274,000 humanitarian workers worldwide Approx $14 billion / year
Humanitarian aid ‘system’
• • • • UN humanitarian agencies International Movement of the Red Cross and Red Crescent Five largest International NGOs Médecins Sans Frontières (MSF) Catholic Relief Services (CRS) Oxfam International International Save the Children alliance World Vision International Increasing attention to ‘accountability’
Cluster approach – 2005 Inter-Agency Standing Committee (IASC)
CODE OF CONDUCT for the International Red Cross and Red Crescent Movement and NGOs doing humanitarian aid activities 1) The humanitarian imperative comes first. 2) Aid is given regardless of the race, creed or nationality of the recipients and without adverse distinction of any kind. Aid priorities are calculated on the basis of need alone. 3) Aid will not be used to further a particular political or religious standpoint. (Independence) 4) We shall endeavour not to act as instruments of government foreign policy. (Neutrality) 5) We shall respect culture and custom. 6) We shall attempt to build disaster response on local capacities.
CODE OF CONDUCT for the International Red Cross and Red Crescent Movement and NGOs doing humanitarian aid activities 7) Ways shall be found to involve programme beneficiaries in the management of relief programmes.
8) Relief aid must strive to reduce future vulnerabilities to disaster as well as meeting basic needs. 9) We hold ourselves accountable to both those we seek to assist and those from whom we accept resources. 10) In our information, publicity and advertising activities, we shall recognise disaster victims as dignified human beings, not hopeless objects.
Emphases added
What should humanitarian groups aspire to?
Some ethical dilemmas
• • • • Do we respond in Japan, where it’s expensive, or in the Congo where we can do more for less?
If we work with, e.g., the Syrian government, do we risk being co-opted into their strategy?
If we bring food, do we drive down market prices and upset local agriculture?
If we give out food at distribution points, do we put people at risk of attack when they walk home?
Ctd …
• • •
Some ethical dilemmas (ctd)
What is the balance between paternalism and autonomy?
Do we speak out about terrible things (only) we see, or stay quiet, do our work and save lives, and let others speak out?
Relations with military?
These decisions are made under (severe) time pressure, and possibly armed force and persuasion/coercion.
Source: Hugo Slim
The Guardian, 28 August 2013
On 14 August, Médecins Sans Frontières (MSF) announced that, after 22 years, it was closing all its activities in Somalia as “the situation in the country has created an untenable imbalance between the risks and compromises our staff must make, and our ability to provide assistance to the Somali people.” … Of the 160 aid workers killed, 82 wounded and 99 kidnapped in Somalia since 1997, the majority have taken place since 2007 (106 killed, 53 wounded and 57 kidnapped), when the overall security environment deteriorated for both humanitarians and Somali civilians.
The Guardian, 28 August 2013
Report on aid worker security October 2013
Some practical concerns with HA
• • • • • • • Little argument about the principle Questions about how well HA is provided Speed of response Coverage of those in need Duplication Unnecessary help Doing more harm than good (cholera in Haiti)
Critique of humanitarian aid
• • • • • • Use of HA for political purposes has been going on for a long time British soldiers in Afghanistan – giving food in exchange for information on Taliban hiding places Colin Powell to NGOs: You are our force multipliers In Darfur, aid workers have provided actionable information for ICC Secret services infiltrate NGOs Embassies keep an eye on what ‘their’ NGOs are doing Source: Antonio Donini
Other possible problems with aid
• • • • • • • • NGOs attract talent from local governments Effect of internationals on local prices NGOs supplant government roles Plethora of NGOs; potential competition Government corruption Qualifications to provide help (Non) Involvement in politics Promised money may not arrive
Port-au-Prince, October 2011
Humanitarian/development aid or conscience money?
Amira Hass in Israeli newspaper Haaretz, 6 Feb 2013
7 Golden Rules of Recovery Management
1. Trust survivors and avoid paternalism 2. Trust survivors and enable survivors to assess their own needs 3. Trust survivors and provide cash rather than kind 4. Think locally 5. Give priority attention to vulnerable groups 6. Think ‘process’ not product, ‘sheltering’ not shelters, ‘housing’ not houses 7. Adopt a long-term perspective while addressing short-term needs Ian Davis + Anshu Sharma, 14 Nov 2013
Toby Ord: “Aid Works (on Average)”
Talk in Oxford, 13 February 2013 • Cited Moyo: – Over 60 years, more than US$1 trillion in aid and not much good to show for it – $1 trillion is nearly $1,000 per person on the planet today • Ord: – Nonsense, actually 1 billion people in Africa – $1 tn / 440 mn (average number of people in Sub-Saharan Africa over period) = $2,263 per person = $38 per person per year
Toby Ord: “Aid Works (on Average)”
(ctd) Successes include: Smallpox eradication Oral rehydration therapy for diarrheal diseases Immunization Malaria Lives saved since eradication: 60-120 million Annual deaths down from 4.6 million to 1.6 million Measles annual deaths down from 5 million to 1.2 million Annual deaths down from 3.8 million to 0.7 million
Why evaluating ‘aid’ is difficult
• • • • • • • • • Reason aid is given Reverse causation Non-developmental aid How/where aid is actually spent Fungibility Multiple donors Remittances Unpredictability of aid Short- vs. long-term impact of projects
Aid or …?
The Guardian, 21 February 2013
Critique of methods in evaluating aid
Stuckler, McKee, Basu 2012 • • • • • • Reviews are narrative not systematic Evaluations don’t consider bias, confounding Multiple outcome measures Time lags Data limitations Power, Type II errors • • • They advocate: Experimentation Evaluation Replication
Evaluating aid - Definitions
• • • Process evaluation: how the intervention was done Impact evaluation: longer-term, intended or unintended Indicator – measure of the effect
Experiments
• • Small-scale RCTs Examples: – paying parents if child attends school regularly – giving food when children vaccinated – microcredit
Increasing attention to accountability
• • • ALNAP - Active Learning Network for Accountability and Performance in humanitarian action Jim Kim, President of World Bank: “Science of delivery [of aid]” Various documents produced by different NGOs on how to evaluate
Evaluating the humanitarian system
• • ALNAP Collates evaluations and produces summary reports State of the Humanitarian System: • ‘assesses progress and performance in six areas: coverage/sufficiency, relevance/appropriateness, effectiveness, connectedness, efficiency and coherence.’ Interactive online evaluation training materials
Difficulties in evaluating HA impact
• • • • • • No baseline / control (or poor data) Need for rapid information Insecure / volatile environments Surveys in settings with little information on the target population Impact of single intervention vs broader impact of programme Lack of capacity
Qualitative and quantitative methods
• • • “… much of the guidance for the evaluation of humanitarian action suggests that evaluations are more likely to provide robust evidence where they use ‘mixed methods’ approaches.” “However, in practice, humanitarian evaluation ‘uses mainly qualitative methods’ (Buchanan-Smith and Cosgrave 2012).” “… evaluations still tend to undervalue the experience of affected populations as a source of evidence …” James Darcy and Paul Knox Clarke, 2013
Focus on Shelter
• • • • • • Seen as human right One of Clusters – incorporates settlement and non food items Housing used for purposes other than purely shelter Relatively little on shelter evaluation ‘Providing adequate shelter is one of the most intractable problems in international humanitarian response’ ‘(Humanitarian Emergency Response Review, 2011) ‘Building Back Better’
Housing and health
Shaw, Ann Rev Pub Health, 2004
Right to adequate housing
• • Article 25 of Universal Declaration of Human Rights Adequacy includes – – legal security of tenure availability of services – – affordability habitability – – accessibility location – cultural adequacy UNHCR, 1991
Section in Sphere Handbook
Proposed timing of shelter after disaster
Relief Emergency shelter 0-2 weeks Recovery Temporary / transitional shelter 2 weeks – 2 years Reconstruction Permanent shelter / reconstruction Charles Parrack 2 years +
Shelter damage in Port-au-Prince after 2010 Haiti earthquake Damage to housing Completely destroyed Partly damaged No visible damage % 24.4
41.5
34.1
Kolbe, Hutson, Shannon et al., Medicine, Conflict and Survival, 2010 95% CI 22.1-26.9
38.9-44.1
31.7-36.6
Shelter for displaced - refugees
Proposed agenda for evaluating shelter after disaster
• • • • Step 1a: Systematically identify indicators Step 1b: Improve list by consulting past and potential beneficiaries Step 2: Develop (short) questionnaire Step 3: Conduct longitudinal study (RCT?) after disaster
Housing trade-off: durability vs comfort
© David Sanderson; with permission
Step 3: Conduct longitudinal study/RCT after disaster
• • • • Timing: – – start 1-2 weeks after disaster follow up at 6 months, 1 year and 2 years Allocate areas/communities/villages to different approaches to housing, i.e., cluster allocation/randomization E.g., ‘T-shelters’ vs. housing in camps vs. building materials and tools Mixed methods – quantitative and qualitative
Feasibility of RCTs?
Example of randomization at the individual level
Technical problems with RCTs include the need to deliver the treatment in a standardised way, the difficulties of maintaining stable membership in the treatment and control groups, and the expense and effort needed. These constraints led a review on evaluations of health promotion by the World Health Organization in Europe to find that the use of RCTs for this was ‘in most cases, inappropriate, misleading and unnecessarily expensive’ (WHO Regional Office for Europe, 1998, p. 5). RCTs also require that the evaluation be designed from the start of the project with random selection of control and treatment groups. For all of these reasons, RCTs are rare in humanitarian evaluation.
(emphasis added) ALNAP EHA Pilot Guide p.113