Transcript Joint Evaluation of the Paris Declaration, Phase 2 Evaluation Framework & Workplan Presentation to International Reference Group 30 Nov 09 Joint Evaluation of the Paris.

Joint Evaluation of the Paris
Declaration, Phase 2
Evaluation Framework & Workplan
Presentation to International Reference Group
30 Nov 09
Joint Evaluation of the Paris Declaration, Phase 2
Core Team
Umbrella Framework for All Elements in
Phase 2
• Overview
• Evaluation Profile: Reasons for the Evaluation, Objectives,
Audiences and Stakeholders, Approach
• Approaches to Methodology: Basis, Principles, The
Evaluation Framework and Core Questions, Core
components of the methodology, Anticipated methods,
Ensuring comparability, Ensuring validity and reliability.
• Accountabilities and Responsibilities
• Workplan and Schedule
A.The Overarching Evaluation Matrix for Phase 2
B.Draft Generic TORs for Country Evaluations
C.Draft Generic TORs for Donor/Agency HQ Evaluations
D.First Draft Outline for Evaluation Synthesis Report
E.Two framework diagrams from preparatory studies
Joint Evaluation of the Paris Declaration, Phase 2
Core Team
Building blocks of the synthesis
SYNTHESIS
EVALUATION QUESTIONS
4. Alternatives
3. Development outcomes
2. Process and intermediate outcomes
1. Context
DONOR STUDIES
COUNTRY STUDIES
SUPPLEMENTARY
STUDIES
PHASE 1 RESULTS
Joint Evaluation of the Paris Declaration, Phase 2
Core Team
Relationship to ToRs
• The Framework is the umbrella that sets out all
substantive elements of the Evaluation, its governance
& broad approaches to methodology;
• Generic ToRs for Country and Donor/Agency HQ
evaluations will operationalize the main components,
with direct guidance to teams;
• The whole structure and synthesis is to be based on a
single, clear set of evaluation questions and a simple
but rigorous approach to ensure comparative findings
that are credible, relevant and useable by its key
audiences.
Joint Evaluation of the Paris Declaration, Phase 2
Core Team
Overall evaluation approach
1.
2.
3.
4.
5.
6.
7.
8.
Accepts special methodological challenges;
Process and results designed for accountability (Seoul & at home)
and learning (to strengthen policies & practices);
Fully joint, relying primarily in Phase 2 on the 20+ country
evaluations, including donors on the ground ;
Seven donor/agency HQ-level studies, plus possible updating of
Phase I. A small number of “supplementary studies”.
Evaluating a) to what extent the PD has been implemented, and b)
in so far as it has been implemented, what the results have been in
terms of aid effectiveness and contribution to development results;
Focus on the workings of country-donor partnerships and their
development outcomes at country-level;
Implementation or “process;” and
“Outcomes” or “Results” in terms of aid effectiveness and
contributions to development results.
Joint Evaluation of the Paris Declaration, Phase 2
Core Team
Approaches to Evaluation Methodology
1. Summative and formative;
2. A clear, simple and straightforward common
approach, language, and methods for the
evaluations;
3. Apply existing data sources, avoid duplicative and
unnecessary demands;
4. Take account of early preparatory work on the
many complex factors and relationships at work
(see diagrams in Appendix E). Matrix to Mgt.
Group in Sept/Oct 2009.
Joint Evaluation of the Paris Declaration, Phase 2
Core Team
The Evaluation Framework and Core
Questions
1. A manageable set of agreed common Core Evaluation
Questions and sub-questions, with common methods, for
robust comparative findings;
2. Focus on the most important, results-oriented questions of
most interest to most users;
3. If needed, partner country and Donor/Agency HQ
evaluations to supplement their answers to the common
Core Questions with questions of special interest/priority to
them;
4. Allow for the integration of Donor/Agency HQ Evaluations;
key cross-cutting assessments, e.g. on adherence to the
five PD principles and the Accra Agenda for Action
priorities; and the results of supplementary studies to fill
gaps.
Joint Evaluation of the Paris Declaration, Phase 2
Core Team
The Paris Declaration in Perspective
Other international &
national influences &
forces
Q1: PD in context
Q1
Q2: Effects of PD on aid
effectiveness
Q3
Q2
Q3: Effects of PD on
development results
Q4
Conclusions: Compared
against pre-PD or
alternative approaches
Overall development processes
The Aid Partnership
Aid influenced
by PDDeclaration,
commitments
Joint Evaluation
of the Paris
Phase 2
Core Team
The Core Questions
(Refined through regional workshops and inputs from other IRG
members)
1. “What are the important factors that have shaped and limited
Paris Declaration implementation and its potential effects on
aid effectiveness and development results?” (The Paris
Declaration in context)
2. “To what extent and how has the implementation of the PD
led to an improvement in the efficiency of aid delivery, the
management and use of aid and better partnerships?”
(Process and intermediate outcomes)
3. “Has the implementation of PD strengthened the contribution
of aid to sustainable development results? How?”
(Development outcomes)
Joint Evaluation of the Paris Declaration, Phase 2
Core Team
Framework for conclusions
1.
2.
3.
4.
5.
To what extent have the 5 principles of the PD been observed and
implemented and the Accra Agenda priorities reflected? Why? Have
there been conflicts or trade-offs between them?
What has the Paris Declaration achieved for aid effectiveness and
development results? How significant are these contributions? How
sustainable? Is there evidence of better ways to make aid more
effective and contribute more to development results?
What has Paris Declaration-style development cooperation added
compared with the pre-PD situation and alongside other drivers of
development in the country, other sources of development finance
and other development partners not so far endorsing the
Declaration?
What are the key messages for a) national stakeholders, and b)
donor countries and agencies?
What are the key implications for aid effectiveness in the future,
taking account of new challenges and opportunities (e.g. climate
change) and new actors and relationships?
Joint Evaluation of the Paris Declaration, Phase 2
Core Team
Core components of the methodology
1. Questions, and the framework for conclusions,
follow the criteria of relevance, efficiency,
effectiveness and sustainability;
2. Include assessment of “upstream” or precursor
steps to Paris in 2005;
3. Not expect clear-cut or one-dimensional causality,
seek “plausible contributions”;
4. Searching analysis of (vital) context;
5. Sub-questions: descriptive, analytical, normative
and evaluative.
Joint Evaluation of the Paris Declaration, Phase 2
Core Team
Common specifications and suggestions for
country evaluations
On:
• Types of evidence and, where applicable, indicators
• Anticipated availability and (probable) reliability of
data sources
• Proposed sources, methods & techniques for data
collection, analysis, triangulation and validation.
Joint Evaluation of the Paris Declaration, Phase 2
Core Team
Anticipated methods
(To be finalized at Inception Report stage)
1. Syntheses and meta-analyses of existing evidence;
2. The normal arsenal of evaluation tools and techniques (see
p. 12);
3. Comparative studies, esp. on one “tracer” and other
sectors;
4. Backward tracking to past Paris Declaration-like initiatives
and their results;
5. Analysis of time-series data, and synthesis studies;
6. ‘Theory based’ (longitudinal) studies that are forward
looking, incl. “direction of travel” and “distance travelled”;
7. Examine “mechanisms of change”: help explain results
8. Ensuring comparability (See p.12)
9. Ensuring validity and reliability (See p.13)
Joint Evaluation of the Paris Declaration, Phase 2
Core Team
Accountabilities and Responsibilities
•
•
•
•
•
•
•
International Reference Group
National Evaluation Coordinator
National Reference/Advisory Group
Country Evaluation Team
Donor Headquarters Evaluation Team
Core Evaluation Team
Evaluation Management Group and Secretariat
Joint Evaluation of the Paris Declaration, Phase 2
Core Team
Workplan and Schedule
• See “Critical Milestones” Document
Joint Evaluation of the Paris Declaration, Phase 2
Core Team
Appendix A. The Overarching Evaluation
Matrix for Phase 2
• Proposed Core Evaluation
Questions & Sub-questions
• Overview of sources of
evidence for answering the
Evaluation questions
(elaborated in Generic
TORs)
Joint Evaluation of the Paris Declaration, Phase 2
Core Team