Transcript JOINT EVALUATIONS: RECENT EXPERIENCES, LESSONS LEARNED, OPTIONS FOR THE FUTURE Study commissioned by the DAC Network on Development Evaluation June 2005 Horst Breier 2005

JOINT EVALUATIONS:
RECENT EXPERIENCES,
LESSONS LEARNED,
OPTIONS FOR THE FUTURE
Study commissioned by the DAC Network
on Development Evaluation
June 2005
1
Horst Breier 2005
Background
 2000: Publication of Effective Practices in Conducting
a Joint Multi-Donor Evaluation in the DAC Evaluation
and Effectiveness Series
 2004: DAC Network on Development Evaluation
commissions a new study on joint evaluations:
To review past experience,
To distil lessons learned, and
To explore options for the future
2
Horst Breier 2005
Background
 The report is based on
An extensive review of literature and written
material, including official documents, manuals,
evaluation reports, etc.
More than 100 interviews with evaluators and aid
officials, representatives of the research and NGO
community, consultants, etc.
The Nairobi workshop with representatives of
partner country governments, NGOs and
consultants
3
Horst Breier 2005
Background
 The report consists of three parts:
First, an analysis of key aspects of joint evaluation
work carried out between 1990 and today
Second, a review of recent experience with joint
evaluations, and a presentation of lessons learned
and recommendations
Third, a chapter on options for the future and
issues for discussion, primarily addressed to
the DAC Evaluation Network
4
Horst Breier 2005
Background
 Annex 1 contains an annotated table of about 50
joint evaluations that were identified during the
research phase. This overview provides a fascinating
picture of the variety of joint evaluations that DAC
members and others have implemented since 1990
 However, this summary table is unlikely to be fully
complete. Members are invited to provide any
additional information on joint evaluations of which
they are aware
5
Horst Breier 2005
Findings
 The report contains a large number, and a wide range,
of findings. This presentation highlights a selection of
the most important ones:
Joint evaluations are a dynamic area of
development co-operation, with an increasing
number of joint evaluations undertaken during the
last 4-5 years
The DAC Evaluation Network has been
instrumental in leading and forming the debate on
this important aspect of evaluation work
6
Horst Breier 2005
Findings
However, DAC members are displaying different
levels of involvement in joint evaluation work.
Denmark, the Netherlands and Norway are clearly
in the lead, followed by a large middle group and
then others who are demonstrating less
commitment to joint working
There is no evidence of a clear pattern that would
explain why particular joint evaluations are
undertaken and others are not
7
Horst Breier 2005
Findings
The wide variety of configurations of actors,
activities, focus, etc, in different joint evaluation
work means it is difficult to develop a typology of
joint evaluations that remains simple and workable.
However, the following categorization, based on the
mode of how actors work together, is proposed:
8
Horst Breier 2005
Findings
TYPE OF EVALUATION
Classic multi-partner
Qualified multi-partner
Hybrid multi-partner
MODE OF WORKING TOGETHER
Open to all stakeholders. All
evaluation partners participate and
contribute equally
Open to those who meet certain
requirements, such as membership
in a particular group of countries,
participation in a SWAp, etc.
A wide range of mostly complex
arrangements for joint work with a
general inclination to keep the
number of partners down, usually
to two or three
9
Horst Breier 2005
Findings
There is a trend over the last 4-5 years towards fuller
partner country participation. However, partner
countries are too often invited to participate at a
relatively advanced stage of the process; when key
decisions have already been taken
A clear message came out of the Nairobi workshop
that partner countries want to be involved in joint
evaluation work more prominently, upstream and
with stronger ownership
10
Horst Breier 2005
Findings
 Chapter 1 explores the wide range of advantages of
joint working, along with the challenges and costs
 Chapter 2 explores more detailed experiences,
challenges and possible solutions in joint evaluation
work. A wide range of lessons learned and practical
recommendations for the future are put forward.
11
Horst Breier 2005
Findings
 Strengths of joint working include:
A high degree of flexibility in forming and
organising evaluation partnerships
Knowledge sharing and mutual learning and
capacity building
High quality of products, usually due to the
pooling of resources and approaches
Enhanced credibility, legitimacy and impact
12
Horst Breier 2005
Findings
 Reduction of transaction costs for partner
countries
 Reduction of number of conflicting messages
emanating from a multitude of donor evaluations
 Shared financing of large evaluations - which
individual donors might not be able to afford
 Promotion of donor harmonisation and alignment
 Response to the evaluation challenges emanating
from the new modes of harmonised aid delivery;
such as GBS, SWAps, basket financing, etc.
13
Horst Breier 2005
Findings
 Potential weaknesses and challenges include:
 Reliance on individual initiative as the prime mover of
joint evaluation work
 Uncertain and uneven commitment of partners to the
outcomes of the joint evaluation
 Too little time allocated at the beginning of the process
to clarify and agree the objectives, concepts,
methodologies and ground rules
 Complex supervisory and management structures, with
too little delegation of authority, and too much micromanagement and control, resulting in bureaucratisation
and delayed processes and delivery
14
Horst Breier 2005
Findings
Selection, guidance and supervision of consultants
High direct and indirect costs for the lead donor(s)
Long gestation periods for evaluation results to
become available, entailing the risk of untimely
delivery of products and outcomes
Risk of conflict between the official and unofficial
agendas of the partners involved in a joint evaluation
15
Horst Breier 2005
Findings
Insufficient upstream participation of partner
countries - resulting in a lack of ownership and a risk
of sidelining the evaluation results within those
countries
Relatively weak reflection, in joint evaluation
activities so far, of the new aid modalities and of the
DAC agenda on Aid Effectiveness
16
Horst Breier 2005
Conclusions
The picture that has emerged from this study is one
of strong and invaluable leadership of the DAC
Evaluation Network in:
- Putting joint evaluations on the agenda
- Rooting them firmly in the donor community
- Providing practical inputs and guidance
Without the contributions of DAC Evaluation Network
members and observers who have been willing to take
up the challenges of joint evaluations, to invest
resources, creativity and leadership, and to stay with
this idea right from the first experiments in the 1990s,
we would be a long way from where we are today.
17
Horst Breier 2005
Conclusions
However, there are a number of new and emerging
challenges that need to be addressed - by donors
individually, by partner countries, and by the DAC
Evaluation Network collectively.
These challenges are listed in Chapter 3 which focuses
on options for the future. Many of the issues raised
need to be addressed urgently.
18
Horst Breier 2005
Conclusions
These options and challenges for the future are
focused around three basic considerations:
1. Improving the efficiency and effectiveness of joint
evaluations
2. Working towards enhanced developing country
participation and ownership
3. Streamlining the role of the DAC Evaluation Network
and focusing on strategic issues
19
Horst Breier 2005
Next Steps
 Network Members are invited to give feedback and
comments on the report, by a deadline of 21 June
 Feedback and comments should be sent directly to
[email protected] with copy to
[email protected] of the Secretariat.
20
Horst Breier 2005