Transcript Session 3 - AsDB_Country Evaluation

ADB’s Evaluation Architecture
and
Country Evaluations
Guidelines
Mala Hettige
Senior Advisor
Independent Evaluation, ADB
Presentation at the meeting of Sub-Regional Development Institutions
Johannesburg, 27 April 2014
Asian Development Bank (AsDB)

Established in 1966 – in Manila, Philippines
67 shareholding member countries
Provides $ 21b in development assistance
About 760 projects
5 regional department and PS department
Managed by MDG, 6 VPs
3000 staff in HQ and 29 RMs + 3 rep. offices

Administrative Budget about $600 million






Independent Evaluation @ AsDB

Abides by OECD’s 4 dimensions of independence

DGE reports to the Board through the Development
Effectiveness Committee (DEC)

Reports public upon DG-IE approval, discussed under Chair, DEC

DGE appointed/removed by DEC – 5 year non-renewable term.

Work Program and separate budget approved by Board

DGE has authority over staff recruitment

Staff subject to backward and forward conflict of interest policy
Major Evaluation Products
Annual Reports
Country Assistance Program Evaluations
Corporate Evaluation Studies
Impact Evaluation Study
Thematic Evaluation Studies
Project/Program Performance Evaluations
Technical Assistance Program Evaluations
Real- time Evaluation Studies
Project Validation Reports
Other Evaluation Reports
4
Accountability and
Learning
5
Evaluation Information System
(EVIS)
A database of evaluation lessons
 Provides users easy access and search capability
 Lessons by sector, theme, topic, country etc.
 EVIS is the first system of its kind
 Accessible through http://evis.adb.org

6
Management Action Record
System (MARS)
Tracks ADB Management actions on
recommendations
 IED Validates actions taken by Management
 Reports this validation to the Board
 Computerized and available to staff and Board
 Departments/offices update progress on actions

Country Evaluations and ADB
Operations
Country
Partnership
Strategy
Country
Evaluation
Implementation
8
Project
Identification
/Preparation
Appraisal /
Approval
History of Country Evaluations in ADB
CAPEs (Country Assistance Program Evaluations)
1998
first CAPE – Peoples Republic of China
2002 Mongolia 2002 questioned choices made
2005 Bhutan had recommendations + criteria ratings
2006 Indonesia added ratings by sector
2007 India had contribution to themes+ value added
2007 China had Strategic and Institutional positioning
2008 Sri Lanka considered the implications of conflict
2008 Philippines considered political economy factors
2009 Cambodia top-down + bottom up assessment
2010 Lao existing guidelines applied
2014 Tajikistan assessment of objectives- ongoing
Evolving Guidelines for CAPEs in AsDB






From 1998 to 2006 no guidelines
First Country Evaluation Guidelines 2006
Revised Guidelines issued 2010
CAPE/ validation of a country self evaluation
mandatory prior to approval of a CPS
Now doing a few revisions to the guidelines
Also preparing validation guidelines
Background: GPS on Country Evaluations

GPS for CSPE established in 2008

Based on OECD–DAC evaluation principles

Draws on the 2007 ECG Review of CSPE

A core GPS criteria allows comparability

An optional GPS criteria helps accountability and
learning
Application of GPS: of Country Evaluation

Name of study differs and number differs

Coordinated with the upcoming country strategy

Substituted with validations or learning products

About 9-12 months, 25-75 pps, $ 250+,

Lending/non lending, about 10 years (2-3 cycles)

Application of principles, approach taken differs

Overall Rating and Evaluation Criteria followed by about half. But
try not to emphasized much
Key Aspects of CPSE

Objectives:


Accountability – Board on results over a period
Learning - engage with stakeholders on future
CPS
Timing : to feed into the next strategy
Coverage:




Long enough to assess results and
Recent to ensure findings are relevant
GPS for Country Strategy and Program Evaluation

GPS for CSPE established in 2008

Based on OECD–DAC evaluation principles

Draws on the 2007 ECG Review of CSPE

A core GPS helps allows comparability

An optional GPS helps accountability and learning
Evaluation Criteria
Core Criteria
Optional Criteria
Relevance
Positioning
Effectiveness
Coherence
Efficiency
Institutional Development
Sustainability
MDB Performance
Impact
Borrower Performance
Partner Coordination
Relevance

GPS : Relevance refers to the degree to which
MDB strategy and program were consistent to
critical needs and long-term plans of the country
Relevance –> Strategic Relevance

Can include optional criteria on positioning –
how well did MDB respond or anticipate
 Evolving development challenges
 Priorities of the government
 Build on MDB’s comparative advantage
 Consider support from other development partners
 Also can include coherence – extent to which there
were measures to
 Foster internal + external synergies within MDB
 Foster external partnerships that promote costeffective division of labor through
complimentarities
Efficiency

GPS: Efficiency refers to the extent to which the
design and delivery of assistance were most cost
effective.
Effectiveness

GPS: Effectiveness refers to the extent to which
the assistance instruments achieved the
intentions and objectives set.
Sustainability

GPS: Sustainability refers to the likelihood that
actual and anticipated results will be resilient to
risks beyond the program period.
Impact

GPS: Impact refers to an MDB’s contribution to
long-term changes in development conditions.

Optional Criteria





Institutional Development
MDB Performance
Client Performance
Partnership Performance
Other concerns





By sector?
What about cross sector impacts?
By objectives?
How to attribute and measure?
How to account for unintended impacts?
IED Website
www.adb.org/evaluation
Thank You!
[email protected]
www.adb.org/evaluation
Follow Us
https://www.facebook.com/adbevaluation
https://twitter.com/adbevaluation
http://www.youtube.com/evaluationatadb
http://www.scribd.com/adbevaluation/
http://www.soundcloud.com/adbevaluation/