Transcript Slide 1
8th Evaluation Conference, Evaluation in the System of Public Policies Warsaw, Poland, 12-13 November 2012 The Challenges Ahead for Impact Evaluation Studies in Poland Using Mixed Methods of Evaluation for Policy Making Purposes Philip Davies International Initiative for Impact Evaluation [3ie] Philip Davies www.3ieimpact.org What Is To Be Evaluated? • Intervention effectiveness - what works? • Implementation effectiveness – how/why it works? • Diversity of effectiveness across different groups – what works for whom and when? • Experiential effectiveness - users’ views • Resource effectiveness - at what cost/benefit? Philip Davies www.3ieimpact.org Which translates into: • Theory Based Evaluation How is a policy/intervention supposed to work? • Impact (or summative) evaluation Does the policy (programme, intervention) work? • Process (or formative) evaluation How, why, and under what conditions does the policy (programme, intervention) work? • Developmental evaluation Supporting innovation and adaptation. Providing feedback, generating learning, supporting direction or affirming changes in direction in real time. Building the evidence base over time. Philip Davies www.3ieimpact.org The key to the challenges ahead is: What’s the Question? Philip Davies www.3ieimpact.org How is the policy Logic Model supposed to work? Theories of Change Evidence for Policy Philip Davies www.3ieimpact.org Evaluation: Theory of Change/Logic Model • How is a policy/programme supposed to work? • What activities, mechanisms, people have to be in place? • And in what sequence – what is the causal chain? • What resources are required – and are available? • What data are required – and are available? • Is the policy/programme feasible/achievable? Philip Davies www.3ieimpact.org How is the policy Logic Model supposed to work? Theories of Change What is already known about this issue? Harness Existing Evidence Evidence for Policy Philip Davies www.3ieimpact.org Research Synthesis Types of Research Synthesis • Statistical Meta-Analyses (6-18 months) • Narrative Systematic Reviews (6-12 Months) • Rapid Evidence Assessments (1-3 Months) • Evidence Maps and Gap Maps (1 Month) • Meta-Ethnography /Qualitative Synthesis (6-12 Months) Philip Davies www.3ieimpact.org How is the policy supposed to work? Logic Model Theories of Change Evidence for Policy Philip Davies www.3ieimpact.org What is already known about this issue? Harness Existing Evidence Descriptive and Experiential Evidence Research Synthesis What is the nature, size and dynamics of the problem? Administrative Data Surveys Census Data Qualitative Research How is the policy supposed to work? Logic Model Theories of Change Evidence for Policy Evidence of Proven Effectiveness What is already known about this issue? Harness Existing Evidence Descriptive and Experiential Evidence Experimental and QuasiExperimental Evidence Philip Davies www.3ieimpact.org Research Synthesis What is the nature, size and dynamics of the problem? Administrative Data Surveys Census Data Qualitative Research What has been shown to work elsewhere? Experimental and Quasi-Experimental Evidence • Randomised Controlled Trials • Matched Comparison Designs - Propensity Score Matching • Difference-in-Differences • Regression Discontinuity Designs • Interrupted Time Series Designs • Instrumental Variable Analysis Philip Davies www.3ieimpact.org How is the policy supposed to work? Logic Model Theories of Change Evidence for Policy Case Studies Interviews Focus Groups Ethnography Operations Research Philip Davies What is already known about the problem/policy? Harness Existing Evidence Descriptive Analysis and Implementation Experiential Evidence Evidence of Evidence Proven Effectiveness Experimental and QuasiExperimental Evidence How do we make the policy work? www.3ieimpact.org Research Synthesis What is the nature, size and dynamics of the problem? Administrative Data Surveys Census Data Qualitative Research What has been shown to work elsewhere? Impact Evaluations Evaluations of Outcome Attainment (Have targets been met?) Policy Delivery: trajectories Long Term Strategic Goal Mid term Delivery Contract Goal 95 90 Intermediate progress indicators or milestones 85 80 75 70 65 Priority Review Historical performance 60 Data Point 1 55 Data Point 2 Indicator DataDelivery Low Trajectory (policy has a lagged impact) Point Mid 3 trajectory High Trajectory (policy has an immediate impact) 50 1996 1997 Philip Davies 1998 1999 2000 2001 2002 2003 2004 www.3ieimpact.org Theory of Change Case Studies Interviews Focus Groups Ethnography Admin data Surveys 2010Operations Analysis 2005 2006 2007 2008 2009 How is the policy supposed to work? Logic Model Theories of Change Cost-Benefit/ Effectiveness/ Utility Analysis What are the costs and benefits of the policy ] Case Studies Interviews Focus Groups Ethnography Operations Research Philip Davies Economic and Econometric Evidence Evidence for Policy Implementation Evidence/ Evidence of What is already known about the problem/policy? Harness Existing Evidence Descriptive Analysis and Experiential Evidence Proven Effectiveness Experimental and QuasiHow do we make The policy work? Experimental Evidence www.3ieimpact.org Research Synthesis What is the nature, size and dynamics of the problem? Administrative Data Surveys Census Data Qualitative Research What has been shown to work elsewhere? Economic Appraisal No policy programme or project should be adopted without first having the answer to these questions: • Are there better ways to achieve this objective [Cost-Effectiveness Analysis] • Are there better uses for these resources? [Cost-Benefit Analysis] Philip Davies www.3ieimpact.org Social Ethics Public Consultation What are the ethical implications of the policy? Cost-Benefit/ Effectiveness/ Utility Analysis What are the costs and benefits of the policy ] Case Studies Interviews Focus Groups Ethnography Operations Research Philip Davies How is the policy supposed to work? Ethical Evidence Logic Model Theories of Change Economic and Evidence Econometric for Policy Evidence Implementation Evidence/ Evidence of Proven Effectiveness What is already known about the problem/policy? Harness Existing Evidence Descriptive and Experiential Evidence Experimental and QuasiHow do we make The policy work? Experimental Evidence www.3ieimpact.org Research Synthesis What is the nature, size and dynamics of the problem? Administrative Data Surveys Census Data Qualitative Research What has been shown to work elsewhere? The Evaluation Process Theorise Specify Theory of Change Activities Mechanisms People Resources Impact and Process Evaluations Systematic Review of All Evaluations Refine Re-Test Accumulate Refine Philip Davies Test www.3ieimpact.org Review An Example: Conditional Cash Transfers Specify Test Theorise Review Theory of Change Activities Mechanisms People Resources Impact and Process Evaluations •CCTS incentivise •By enforcing the •>100 impact conditionality of parents to send evaluations children to school schooling •Many process •Changing the •Improve evaluations behaviour of parents enrolment and •Testing CCTs in and children attendance •Altering the relative Latin America, •Improve learning costs and benefits of Asia, Africa outcomes schooling versus Philip Davies other uses of children’s time www.3ieimpact.org Systematic Review of All Evaluations •Petrosino et al, (2012) Systematic Review of 23 studies •Quality Education For All Children •CCTs increase enrolments and attendance •No overall impact on learning outcomes. Theorise Specify Theory of Change Activities Mechanisms People Resources Refine Refine •Deadweight issues •Measure learning outcomes •Levels of payment •Supply-side •Payment recipient interventions needed •Reward performance •Teacher quality a mediating factor •Phased transfers? Philip Davies Test Impact and Process Evaluations Review Systematic Review of All Evaluations Re-Test Accumulate •Need to re-test refinements/mec hanisms •Focus on contextual specificity •Build an evidence base over time •Establish generalisability •And context specificity •Avoid Type I/II errors www.3ieimpact.org Thank you Philip Davies Email: [email protected] +44 (0)207 958 8350 Visit www.3ieimpact.org Philip Davies www.3ieimpact.org