Two recent systematic reviews for development

Download Report

Transcript Two recent systematic reviews for development

Annette N. Brown, Deputy Director for AIES

TWO RECENT SYSTEMATIC REVIEWS FOR DEVELOPMENT

International Initiative for Impact Evaluation www.3ieimpact.org

Outline

• Review questions • Inclusion criteria • Theory of change • Search pipeline • Results • Fun methods finding

International Initiative for Impact Evaluation www.3ieimpact.org

“Interventions in Developing Nations for Improving Primary and Secondary School Enrollment of Children: A Systematic Review” By Anthony Petrosino, Claire Morgan, Trevor A. Fronius, Emily E. Tanner-Smith, Robert F. Boruch, November 2012

International Initiative for Impact Evaluation www.3ieimpact.org

Review questions

• What are the effects of interventions implemented in developing countries on measures of students’ enrollment, attendance, graduation, and progression?

• Within those studies that report the effects of an intervention on [the above measures], what are the ancillary effects on learning outcomes?

International Initiative for Impact Evaluation www.3ieimpact.org

Inclusion criteria

Population/participants: Primary and secondary school aged children in LMICs

Interventions: Intended to affect one of the four primary outcomes

Comparison: No specific program comparison Outcomes: Enrollment, attendance, graduation, and progression Studies: RCTs and QEDs

International Initiative for Impact Evaluation www.3ieimpact.org

Typology of education interventions Demand Supply Reducing costs Providing information CCTs, scholarships and non-fee subsidies Increasing preparedness Early child development Buildings Teachers Vouchers Health/ nutrition Materials Abolishing school fees and capitation grants

International Initiative for Impact Evaluation

Management School feeding

www.3ieimpact.org

International Initiative for Impact Evaluation www.3ieimpact.org

“Farmer field schools for improving farming practices and farmer outcomes in low- and middle-income countries: a systematic review” By Hugh Waddington, Birte Snilstveit, Jorge Hombrados, Martina Vojtkova, Daniel Phillips, and Howard White, December 2012

International Initiative for Impact Evaluation www.3ieimpact.org

Review questions

• What is the impact of farmer field schools on their objectives in terms of ‘endpoint’ outcomes such as increased yields, net revenues and farmer empowerment, and intermediate outcomes such as capacity building and adoption of improved practices?

• Under which circumstances and why: what are the facilitators and barriers to FFS effectiveness and sustainability?

International Initiative for Impact Evaluation www.3ieimpact.org

Inclusion criteria

• Population/participants: Farm households in low and middle income countries • Intervention: Programs explicitly referred to as ‘farmer field school ’ • Comparison: No specific program comparison • Outcomes: effectiveness across the causal chain – Knowledge → adoption → – Impact on yields, revenues, environment, health, empowerment • Studies: – Effects: experimental, quasi-experimental with controlled comparison – Barriers/facilitators: qualitative (Critical Appraisal Skills Programme checklist 2006)

International Initiative for Impact Evaluation www.3ieimpact.org

Input 1 Training of trainers Input 2 Field school

T of Change

Capacity building (FFS participants) Capacity building (FFS neighbours) Adoption (FFS participants) Adoption (FFS neighbours) Measured impacts:

Yield, input-output ratio, income, empowerment, environmental

International Initiative for Impact Evaluation

outcomes, health

www.3ieimpact.org

Input 1 Training of trainers Input 2 Field school - Facilitators adequately trained - Farmers and facilitators attend sufficient meetings - FFS synchronised with planting season Capacity building (FFS participants) Capacity building (neighbours) - Field days/follow-up - High degree of social cohesion - Geographical proximity to other farmers (observation) or market (communication) - Curriculum relevant to problems facing farmers

Farmer attitudes changed Adoption (FFS participants) Adoption (neighbours) (convinced message appropriate) Measured impacts:

Relative

Yield, input-output ratio,

advantage over old

income, empowerment, environmental

International Initiative for Impact Evaluation

outcomes. health

- New technology appropriate - Market access - Favorable prices - Environmental factors including weather, soil

Effectiveness

27,866 titles screened 1453 abstracts screened 1,112 abstracts screened

Causal Chain Analysis

751 excluded 126 no access 369 full text obtained 312 full text sought 49 no access 186 excluded:

128 on relevance 58 on design (no comparison)

183 Extension impact papers: 134 FFS 49 non-FFS 134 FFS impact papers

Qualitative Synthesis

25 qualitative papers 257 excluded

BB+ Synthesis

30 IE and sister papers 80 individual FFS studies International Initiative for Impact Evaluation 20 individual FFS studies 11 individual FFS studies www.3ieimpact.org

Study ID FFS participants Huan et al., 1999 (Vietnam) Endalew, 2009 (Ethiopia) Price et al., 2001 (Philippines) Rao et al., 2012 (India) Reddy & Suryamani, 2005 (India) Kelemework, 2005 (Ethiopia) Mutandwa & Mpangwa, 2004 (Zimbabwe) Dinpanah et al., 2010 (Iran) Khan et al., 2007 (Pakistan) Bunyatta et al., 2006 (Kenya) Erbaugh, 2010 (Uganda) Rebaudo & Dangles, 2011 (Ecuador) .

Subtotal (I-squared = 93.3%, p = 0.000) FFS neighbours Khan et al., 2007 (Pakistan) Reddy & Suryamani, 2005 (India) Ricker-Gilbert et al, 2008 (Bangladesh) Rebaudo & Dangles, 2011 (Ecuador) Subtotal (I-squared = 0.0%, p = 0.610) .

NOTE: Weights are from random effects analysis Positive impacts on knowledge among participants -.5

1 Favours intervention

International Initiative for Impact Evaluation

www.3ieimpact.org

Study ID FFS neighbours Pananurak, 2010 (India) Khan et al., 2007 (Pakistan) Feder et al, 2004 (Indonesia) Labarta, 2005 (Nicaragua) Pananurak, 2010 (China) Wu Lifeng, 2010 (China) Pananurak, 2010 (Pakistan) .

Yamazaki & Resosudarmo, 2007 (Indonesia) Subtotal (I-squared = 49.5%, p = 0.054) FFS participants Pananurak, 2010 (India) Van Rijn, 2010 (Peru) Naik et al., 2008 (India) Huan et al., 1999 (Vietnam) Labarta, 2005 (Nicaragua) Rejesus et al, 2010 (Vietnam) Feder et al, 2004 (Indonesia) Wu Lifeng, 2010 (China) Ali & Sharif, 2011 (Pakistan) Pananurak, 2010 (China) Gockowski et al., 2010 (Ghana) Yang et al., 2005 (China) Hiller et al., 2009 (Kenya) Khan et al., 2007 (Pakistan) Cavatassi et al., 2011 (Ecuador) Davis et al, 2012 (Tanzania) Birthal et al., 2000 (India) Pananurak, 2010 (Pakistan) Dinpanah et al., 2010 (Iran) Wandji et al., 2007 (Cameroon) Mutandwa & Mpangwa, 2004 (Zimbabwe) Palis, 1998 (Philippines) Zuger 2004 (Peru) Carlberg et al., 2012 (Ghana) Yamazaki & Resosudarmo, 2007 (Indonesia) Van den Berg et al., 2002 (Sri Lanka) Davis et al, 2012 (Kenya) Pande et al., 2009 (Nepal) Dinpanah et al., 2010 (Iran) Orozco Cirilo et al., 2008 b) (Mexico) Todo & Takahashi, 2011 (Ethiopia) Subtotal (I-squared = 93.0%, p = 0.000) .

NOTE: Weights are from random effects analysis .5

2 3 0.79 (0.63, 1.00) 0.97 (0.74, 1.26) 0.99 (0.97, 1.01) 1.00 (0.99, 1.01) 1.02 (0.98, 1.07) 1.03 (0.99, 1.08) 1.03 (0.86, 1.25) 1.43 (1.05, 1.96) 1.01 (0.98, 1.03) 0.80 (0.61, 1.05) 0.86 (0.63, 1.18) 0.89 (0.83, 0.96) 0.95 (0.92, 0.98) 0.97 (0.92, 1.02) 0.97 (0.72, 1.31) 0.98 (0.96, 1.01) 1.08 (1.03, 1.14) 1.09 (1.03, 1.15) 1.09 (1.04, 1.14) 1.14 (1.03, 1.25) 1.15 (0.94, 1.41) 1.17 (0.53, 2.56) 1.17 (0.97, 1.42) 1.22 (0.97, 1.53) 1.23 (1.00, 1.51) 1.24 (1.13, 1.36) 1.24 (1.01, 1.54) 1.32 (1.22, 1.42) 1.32 (1.07, 1.63) 1.36 (1.06, 1.73) 1.36 (0.97, 1.92) 1.44 (1.09, 1.92) 1.58 (1.19, 2.10) 1.67 (1.23, 2.26) 1.68 (1.30, 2.18) 1.81 (1.15, 2.84) 2.11 (1.25, 3.56) 2.52 (2.05, 3.11) 2.62 (2.23, 3.08) 2.71 (1.11, 6.60) 1.23 (1.16, 1.32) Increased yields among FFS beneficiaries not neighbours

www.3ieimpact.org

Reduced environmental risk factors

Study ID FFS participants Pananurak, 2010 (India) Praneetvatakul & Waibel, 2006 (Thailand) Pananurak, 2010 (Pakistan) Cavatassi et al., 2011 (Ecuador) .

Subtotal (I-squared = 8.0%, p = 0.353) FFS neighbours Pananurak, 2010 (India) Pananurak, 2010 (Pakistan) Cavatassi et al., 2011 (Ecuador) Praneetvatakul & Waibel, 2006 (Thailand) Subtotal (I-squared = 0.0%, p = 0.878) .

NOTE: Weights are from random effects analysis .1

Favours intervention

International Initiative for Impact Evaluation

www.3ieimpact.org

Summary of quantitative findings

• FFS increase knowledge and improve adoption of the FFS practices • On average increasing yields and/or incomes • Suggestions of farmers feeling empowered • Limited, if any, spillovers • Neighbours do not adopt the practices consistently

International Initiative for Impact Evaluation www.3ieimpact.org

Study ID High risk of bias Naik et al., 2008 (India) Huan et al., 1999 (Vietnam) Labarta, 2005 (Nicaragua) Gockowski et al., 2010 (Ghana) Yang et al., 2005 (China) Hiller et al., 2009 (Kenya) Khan et al., 2007 (Pakistan) Birthal et al., 2000 (India) Dinpanah et al., 2010 (Iran) Wandji et al., 2007 (Cameroon) Mutandwa & Mpangwa, 2004 (Zimbabwe) Palis, 1998 (Philippines) Zuger 2004 (Peru) Carlberg et al., 2012 (Ghana) Van den Berg et al., 2002 (Sri Lanka) Pande et al., 2009 (Nepal) Dinpanah et al., 2010 (Iran) Orozco Cirilo et al., 2008 b) (Mexico) Subtotal (I-squared = 95.4%, p = 0.000) .

Medium risk of bias Pananurak, 2010 (India) Van Rijn, 2010 (Peru) Rejesus et al, 2010 (Vietnam) Feder et al, 2004 (Indonesia) Wu Lifeng, 2010 (China) Ali & Sharif, 2011 (Pakistan) Pananurak, 2010 (China) Cavatassi et al., 2011 (Ecuador) Davis et al, 2012 (Tanzania) Pananurak, 2010 (Pakistan) Yamazaki & Resosudarmo, 2007 (Indonesia) Davis et al, 2012 (Kenya) Todo & Takahashi, 2011 (Ethiopia) Subtotal (I-squared = 81.0%, p = 0.000) .

Overall (I-squared = 93.0%, p = 0.000) NOTE: Weights are from random effects analysis 0.97 (0.92, 1.02) 1.14 (1.03, 1.25) 1.17 (0.53, 2.56) 1.17 (0.97, 1.42) 1.32 (1.22, 1.42) 1.32 (1.07, 1.63) 1.36 (1.06, 1.73) 1.36 (0.97, 1.92) 1.58 (1.19, 2.10) 1.68 (1.30, 2.18) 2.52 (2.05, 3.11) 2.62 (2.23, 3.08) 0.80 (0.61, 1.05) 0.86 (0.63, 1.18) 0.98 (0.96, 1.01) 1.08 (1.03, 1.14) 1.09 (1.03, 1.15) 1.09 (1.04, 1.14) 1.22 (0.97, 1.53) 1.24 (1.01, 1.54) 1.67 (1.23, 2.26) 2.71 (1.11, 6.60) 1.10 (1.03, 1.17) Sensitivity analysis: Yields by risk of bias status High risk of bias studies over estimate impacts .5

2 3 Favours intervention

International Initiative for Impact Evaluation www.3ieimpact.org

www.3ieimpact.org

THANK YOU!

International Initiative for Impact Evaluation www.3ieimpact.org

Two recent systematic reviews for development

Transcript Two recent systematic reviews for development