Transcript Peer reviews: who is evaluating the evaluators?

Rob D. van den Berg, GEF Independent Evaluation Office
Mini-workshop IPDET, June 13, 2014
PEER REVIEWS: WHO IS EVALUATING THE
EVALUATORS?
OVERVIEW
Who is evaluating the evaluators?
 What should they be evaluated on?
 What are the criteria on which they should be
judged?
 How should they be evaluated?
 How to do a peer review?

2
WHO IS EVALUATING THE EVALUATORS?

Why has this question become important?




Increasing independence of evaluators (their performance is
no longer assessed by management)
Increasing professionalization (who checks whether they are
actually good professionals?)
Increasing coverage of evaluations (judging every program;
but who judges the evaluators?)
What should they be evaluated on?




Conduct: transparency, lack of bias, no conflict of interest,
ethics
Quality of work: design, implementation, reporting
Effectiveness and usefulness of evaluations
Etc…
3
WHAT ARE THE CRITERIA?





OECD/Development Assistance Committee (DAC)
Principles for Evaluation of Development Assistance
(1991)
Evaluation Cooperation Group (ECG) Good Practice
Standards (ongoing)
UNEG Norms and Standards (2005)
Various Ethical Guidelines (UNEG, AEA)
International Best Practice?


Some communities of practice may have specific norms and
standards: ALNAP?
See references at the end of the presentation
4
COMMON ELEMENTS (1)


DAC, ECG and UNEG agree on the importance of
independence, credibility and usefulness of evaluations
Independence:


Best international practice: ECG paper (“Template for Assessing
the Independence of the Evaluation Function”)
Independence has building blocks:




First and foremost: independent evaluations – “functional”
independence
Secondly: structural independence: reporting lines, budget, conflict of
interest issues, etc.
Thirdly: administrative/logistical independence
ECG evaluation offices tend to have all three; bilateral units and
UN units tend to have the first but have various degrees of
independence at the structural and administrative levels
5
COMMON ELEMENTS (2)

Credibility
Agreement on the need for high quality, but:
 Specific quality norms and standards are relatively new
 Transparency
 Accountability for what has been done
 Quality assurance


Usefulness
Focus on strategic issues
 Evaluation coverage
 Links between central and decentralized evaluations
 Knowledge products / knowledge sharing

6
HOW SHOULD THEY BE EVALUATED?
Self-assessment (done by many)
 Independent Review (UNESCO, WHO, WB, ADB)
 Quality Assurance (many)
 Evaluation (FAO/2007)
 Advisory / Oversight Panels (several)
 Professional Peer Reviews (several)
 A combination of the above

7
PEER REVIEWS

UNEG Peer Reviews:













ECG Peer Reviews


UNDP (2005)
UNICEF (2006)
WFP (2007)
OIOS (2008)
GEF (2009)
UNIDO (2010)
UN-Habitat (2011)
UNEP (2011)
FAO (2012)
UNDP (2012)
GEF (2014)
UN Women (2014)
IFAD (2011)
Bilateral Peer Reviews

Belgium, Germany
8
UNEG FRAMEWORK FOR PEER REVIEWS
UNDP and UNICEF were pilot peer reviews, leading
to:
 DAC/UNEG Framework for Peer Reviews 2006
 UNEG Framework for Peer Reviews 2011
 Shift in ownership; peer reviews are now perceived
as a UNEG instrument
 UNEG has established a peer review task force
and is willing to provide funds to smaller UN
agencies
 Collaboration with DAC continues

9
HOW TO DO A PEER REVIEW (1)


Governing body or senior management need to ask for peer
review
UNEG Task Force assembles a panel of peers in consultation
with the organization to be peer reviewed


Early panels were big and aimed for representation of
stakeholders and professional community
Panel writes TORs and assembles the budget


TORs follow from peer review framework but need to focus on
specifics of the organization (mandate, role, modalities etc.)
Budget divides out in budget for organization and budget for peer
panel




Internal budget of organization: time of staff, meeting rooms etc.
External budget: panel members and advisor/consultants that need to
be funded
Funding of external budget: is shifting from bilateral donors to
organization that is peer reviewed
Budgets are coming down from about $200k to less than $50k
10
HOW TO DO A PEER REVIEW (2)




The panel writes a “normative framework” on which the
evaluation function will be reviewed
The evaluation function performs a self-assessment on
the basis of the normative framework
The panel studies the self-assessment and visits the
organization for interviews
Some early peer reviews conducted field visits which
were abandoned due to costs and issues of
representativeness (1st GEF: peer review made
statements on interactions with stakeholders on the
basis of two country visits, whereas the GEF is active in
more than 150 countries)
11
HOW TO DO A PEER REVIEW (3)



Panel prepares further work on the basis of the selfassessment, first visit and evaluation reports
Second visit: further interviews to fine-tune findings and
fill gaps
Since 2011: “peer exchange” session of peers with staff
of evaluation unit




Usefulness of this confirmed in FAO and UNDP peer reviews
Draft peer review report sent to organization for factual
error checking and errors of analysis
Final peer review report presented to organization
Management response and follow-up action
12
FINDINGS: INDEPENDENCE

Functional independence is universally
recognized
 Many
organizations feel it is sufficient to externalize
evaluations: outsource them, establish
independent teams
 This poses problems for both the credibility and the
usefulness of evaluations
 Coherence
between evaluations
 Links between central and decentralized evaluations
13
FINDINGS: INDEPENDENCE (2)

Structural Independence is problematic
 UN
governance forbids full structural independence
(?!?)
 Next best option is for CEO to delegate authority to
evaluation function
 UN agencies have been moving in the direction of
structural independence thanks to peer reviews

Administrative/logistical independence
 The
details… continue to cause problems…
14
FINDINGS: CREDIBILITY

Level of professionalism should be further improved






50% of staff of UN evaluation units do not have evaluation
training
Many evaluations still follow the “expert” model for
evaluations
Final responsibility for the evaluations should shift to the
evaluation function
Theory-based evaluations are increasing
Comparability between evaluations: problematic
Evaluations at the central level should build on findings
from decentralized evaluations
15
USEFULNESS
Strategic focus of evaluations can be improved
 Expert evaluations always end up with
recommendations to increase the priority of the
area/sector evaluated
 Link to RBM system of the organization is often
weak
 Stakeholder involvement has increased over time
but could be further strengthened
 New ways to share knowledge are not yet fully
incorporated

16
LESSONS LEARNED



In 2012 UNEG and DAC commissioned a “lessons learned”
study on the peer reviews
Report was presented and discussed at 2013 Annual
General Meeting of UNEG
Main finding: peer reviews have been successful in
strengthening evaluation function



Independence further strengthened
Credibility increased
Usefulness: reviews have been agenda setting



Refocus programming to increase strategic value of evaluations
Better linkages with RBM and decentralized evaluations
Main recommendation: embed peer reviews in broader
system to promote professionalisation
17
ISSUES FOR THE FUTURE

Peer reviews are voluntary: they need to become
obligatory


Peer reviews should be part of a larger system to
promote professionalism


Funding from a common pool in the UN?
It is one instrument amongst many (self-assessment,
reviews, evaluation, quality assurance; for the future:
accreditation?)
Emerging new “best practice“ on how to do peer
reviews
18
TOWARDS A NEW MODEL FOR PEER REVIEWS

Smaller panels
 Early
panels had 6 to 8 members; now 4 is seen as
the ideal size
More emphasis on the self-assessment; higher
investment of evaluation office, but creates
better understanding at the start
 One instead of two peer review visits; extra
emphasis on the “peer exchange” session
 Shorter reports focusing on strategic issues

19
REFERENCES

OECD/DAC Principles for Evaluation of Development
Assistance (1991)


Evaluation Cooperation Group (ECG) Good Practice
Standards (ongoing)





www.ecgnet.org – go to “key documents”
UN Evaluation Group (UNEG) – Norms and Standards (2005)


http://www.oecd.org/development/evaluation/2755284.pdf
www.uneval.org go to “norms standards”
For peer reviews: go to “Papers and Publications” and “UNEG-DAC
Peer Reviews”
AEA: www.eval.org
ALNAP: www.alnap.org
IFAD peer review: www.ecgnet.org go to “ECG evaluations”
20