Transcript V Konferencja Ewaluacyjna Warszawa, 23.10.2009 Manuel Fernando Castro The Role of M&E Systems in Performance Budgeting: Good-practice and lessons learned from experience V Annual Evaluation Conference The.

V Konferencja Ewaluacyjna
Warszawa, 23.10.2009
Manuel Fernando Castro
The Role of M&E Systems in
Performance Budgeting: Good-practice
and lessons learned from experience
V Annual Evaluation Conference
The Role of M&E Systems in Performance
Budgeting: Good-practice and lessons learned from
experience
Warsaw, October 2009
Overview
► What do we mean by M&E and PB
► Strategic context for M&E and Performance Budgeting
► Role of country-level M&E systems in Performance
Budgeting
► Examples of good practice countries (Mexico, Chile, UK)
► Lessons from international experience
Overview
► What do we mean by M&E and PB
► Strategic context for M&E and Performance Budgeting
► Role of country-level M&E systems in Performance
Budgeting
► Examples of good practice countries (Mexico, Chile, UK)
► Lessons from international experience
What is M&E?
Monitoring
“M”
Regular tracking of
indicators to establish
whether planned
targets have been
achieved
Evaluation
“E”
Analytical efforts to
answer specific
questions about
performance of a
program/activities
Impact evaluation
“IE”
Analytical efforts to
relate cause and effect.
Establishing what would
have happened in the
absence of the
intervention
Each tool has strengths, costs, limitations
5
What is Performance Budgeting?
• “Form of budgeting that relates funds allocated to
measurable results” (OECD, 2005a)
• 3 different modalities
Presentational
Linkages with
allocation
Main objective
Evidence of
utilization
Performance-informed
Direct/Formula funding
No links
Indirect/Loose link
Direct/Tight link
Accountability
Planning and/or
accountability
Resource allocation and
accountability
Most OECD (e.g., UK,
Australia , Mexico) +
Chile.
Mostly in sectors
(e.g., Denmark's school
funding, Chile's per capita
6
funding for clinics)
Sweden,
Denmark,
Colombia
Overview
► What do we mean by M&E and PB
► Strategic context for M&E and Performance Budgeting
► Role of country-level M&E systems in Performance
Budgeting
► Examples of good practice countries (Mexico, Chile, UK)
► Lessons from international experience
There is no simple/direct relation between spending
and results
•
Similar increases in public spending can be associated with
totally different outcomes…
Spending data for 1990s from World Development Indicators database. Child mortality data from Unicef 2002. Other data from World Bank .
School completion from Bruns, Mingat and Rakatomalala 2003.
More spending does not necessarily mean better outcomes
There is no simple/direct relation either from
growth rates to results
• Tanzania and Bangladesh have similar GDP per capita, but life
expectancies are 55 years and 65 years.
• Philippines 2006 per capita income was 30% higher than in
Moldova, but Moldova’s maternal mortality ratio was 16 as
opposed to 162 per 100,000 live births in the Philippines.
• Despite remarkable economic growth, India’s maternal
mortality rate has worsened and is six times worse than
China's and 14 times worse than Chile's.
Higher economic growth, can mask the differences in
important outcomes of development.
Budget officials and policy makers need to respond
challenging questions
STRATEGIC
Are we doing the right
things?
OPERATIONAL
Are we doing things
right?
• What are the underlying causes of results? Are this
direct , indirect, intended, non intended?
• What programs/policies should fbe unded, strengthened
or changed, and how?
• Is program implementation following anticipated paths?
• Are promised activities taking place as planned?
LEARNING
Are there better ways
of doing them?
• Are there best practices or lessons learned that can be
applied?
M&E can be associated with better project outcomes
(assist with improved management)
• 80 - 90% WB projects rated high or substantial in M&E were rated
satisfactory or better in their outcomes
• Those with negligible ratings had satisfactory only in 32% of cases
Overview
► What do we mean by M&E and PB
► Strategic context for M&E and Performance Budgeting
► Role of country-level M&E systems in Performance
Budgeting
► Examples of two good practice countries (UK, Mexico)
► Lessons from international experience
Key factors
• Evaluation measures results, Performance Budgeting tides
results to funding.
•To be relevant and sustainable PB demands continuous
performance information:
• But …multiple requirements to fulfill
• Clear program logics / results chains
•Timely, and quality data
•Good indicators and targets
•Robust methods and tools
•Quality standards and procedures
•Mechanisms to ensure utilization and influence
13
The right kind of information is frequently missing
• Conflicting demands of PI, wrong incentives, and low M&E
capacities often lead to
- Lack of clarity on the role of PI for budgeting
- Duplication and overspending
- Not standardized methods and tools (limitations and
strengths)
- Low quality and scarce use
M&E systems help tackle performance information
problems
• Ensuring supply of performance data (records, stats, balanced
M&E data, etc.)
• Stimulating demand, (use in policy, transparency,
accountability, etc)
• Guaranteeing quality and credibility of data (robust
methodologies, standards)
• But also increasing access to PI (integrated IT systems,
reports, etc).
Country-level M&E systems are powerful budgetary
tools
• Facilitate budget shift from traditional management (input/ activitybased) to performance management (output/outcome-based)
• Emphasize effective resource allocation for planning and
budgeting
• Help countries assess if using scarce resources cost-effectively
• Respond to elected officials and the public’s demands for
accountability in the budget
• Enhance the need for managers in the public sector
• Stress knowledge and learning for improving budget decisionmaking
Bottom line of the M&E and PB relationship
• NO Performance Budgeting possible without systematic
and good quality M&E information.
• NO effective M&E without links with the budget process
Development of PB and M&E systems need to go hand in
hand
Overview
► What do we mean by M&E and PB
► Strategic context for M&E and Performance Budgeting
► Role of country-level M&E systems in Performance
Budgeting
► Good practice country cases
► Lessons from international experience
United Kingdom: The Public Service Agreements System
Main architecture features:
- High level Strategy (PMSU) and Delivery Units (PMDU)
- Ministerial public agreements on services to be delivered
- Targets, indicators and managers for each delivery agreement
- Ministries’ PSA Delivery Board (chaired by managers)
- Regular performance monitoring (PMDU, Cabinet)
- Dif evaluation types (expending reviews, program, impact, audits)
- Public dissemination of strategy and agreements
•H.M. Treasury key player. Prime Minister’s (PMSU) and (PMDU)
provide advise and regular updates directly to the Prime Minister.
United Kingdom: The Public Service Agreements System
Strengths
• “Explains what departments plan to deliver in return
for…significant extra investment”
• “[Sets out] demanding national targets” (Nt. Standards)
•
Multiple agencies but H.M. Treasury guidelines allow coordination
• Highly credible evaluations (externally contracted)
• “[Reflects] the Government’s key priorities and [focus] on the
outcomes that matter most to the public”
• Establishes responsible “managers” of each PS
• Leading role of Treasury and Prime Minister’s office
Mexico: The Performance Evaluation System – SED
Main architecture features:
-
Independent Evaluation Unit: CONEVAL council composed of academics
Single Federal Evaluation Guidelines
“Logic Frameworks” and standards for all F. Programs
258 Performance indicators (output and outcome based)
6 evaluation methods: Impact, consistency, indicators, process, design, perf.
Annual Evaluation Plan (More than 100 annually).
• The MoF is the system’s “champion” with active support and
participation of the President’s Office.
Mexico: The Performance Evaluation System – SED
Strengths
• CONEVAL Model brings credibility
- Contracts out evaluation (market of evaluation)
- increases quality, regularity, and timeliness of evaluation work
- Promotes capacity development
•
•
•
•
•
Unique poverty measurement schema as part of the system
Robust and credible evaluations, strengthening monitoring
All evaluations reported to Congress (with significant budget initiative)
Powerful role of MoF and Presidency increases utilization
Active role of Ministries and agencies in defining evaluation agenda
Chile: The Management Control System
Main architecture features:
- Ex ante cost-benefit analysis for investment programs
- 1,550 performance indicators for all government programs (2006)
- “Program evaluations”: desk reviews of log frame (14 each year)
- Rigorous impact evaluations (4 each year)
- Comprehensive Spending Reviews: selective programs (2-5 each year)
- Agency management improvement programs linked to staff payments
• The MoF (Budget Directorate) designed and uses the
system. Congress has no spending initiative.
Chile: The Management Control System
Strengths
• Performance information used to identify need for evaluations
• Evaluations conducted externally, in fully transparent process, and are
highly credible
• All M&E findings reported publicly and sent to Congress
• M&E system closely linked to the budget information needs of MoF
• Performance information used to set targets for ministries – these are
largely met
• MoF closely monitors extent of utilization of evaluation findings
Overview
► What do we mean by M&E and PB
► Strategic context for M&E and Performance Budgeting
► Role of country-level M&E systems in Performance
Budgeting
► Good practice country cases
► Lessons from international experience
Main risks and difficulties
1. Deficient budget planning - No technical definition of program
objectives, goals and indicators
2. Weak system design, unclear institutional arrangements
- How much “M” & “E” is enough? , who plan, conduct ,and use of evaluations?
- Self-evaluations or independent evaluations?
- Who funds the system? (a) earmarked budget funds?; (b) costs “absorbed” by
ministries? How it links with the budget?
3. On supply side: caveat multiple, competing, uncoordinated systems
4. On demand side: Budget “rigidity”, incentives are key to achieve
utilization
5. Low staff capacities (needs PS managers)
No strong preconditions but “Success” factors
include
1. High Gov. commitment with quality of public spending
2. Powerful “champion”, ideally centrally driven by capable MoF Ministry
3.Careful attention to both demand and supply sides
4.Sustained, continuous effort
5. Piecemeal to ensure assimilation
6.“Quick gains” to avoid the costs of a too timid implementation
7. Avoiding complex systems (keep it simple)
Useful resource material
Keith Mackay, How to Build M&E Systems for Better Government, World Bank,
2007. http://www.worldbank.org/ieg/ecd/better_government.html
Ernesto May et al. (eds.), Towards the Institutionalization of Monitoring and
Evaluation Systems in Latin America and the Caribbean, World Bank, 2006.
http://web.worldbank.org/WBSITE/EXTERNAL/COUNTRIES/LACEXT/0,,contentMDK:20893139~pageP
K:146736~piPK:146830~theSitePK:258554,00.html
Manuel F. Castro, Insider Insights: Building a Results-Based Management and
Evaluation System in Colombia. World Bank, 2008.
http://www.worldbank.org/ieg/ecd/docs/ecd_wp_18.pdf
Gonzalo Hernandez Licona, The Role of Evaluation in Mexico: Achievements,
Challenges and Opportunities. Quality of Public Expenditure. Note No. 2. World
bank, 2008. http://siteresources.worldbank.org/MEXICOEXTN/Resources/MXNewsletterQualityofPubExpenditure-No2.pdf
Useful resource material
Teresa Curristine, Performance Information in the Budget Process: Results of the
OECD 2005 Questionnaire, in OECD Journal on Budgeting, 2005 5(2),pp.87-131.
http://www.oecd.org/document/38/0,3343,en_2649_34119_39921702_1_1_1_1,00.html
OECD, Performance Budgeting in OECD Countries, OECD, 2007.
http://www.oecd.org/document/38/0,3343,en_2649_37405_39921702_1_1_1_37405,00.html#B2
Independent Evaluation Group, M&E Tools, Methods and Approaches, World
Bank, 2004. http://www.worldbank.org/ieg/ecd/me_tools_and_approaches.html
Independent Evaluation Group, Influential Evaluations: Evaluations that Improved
Performance and Impacts of Development Programs, World Bank, 2004.
http://www.worldbank.org/ieg/ecd/influential_evaluations.html
World Bank website on Building Government M&E Systems:
www.worldbank.org/ieg/ecd/