Transcript Phare Interim Evaluation Scheme

Lessons from Programme
Evaluation in Romania
First Annual Conference on Evaluation
Bucharest
18 February 2008
Objectives of Session 2

increase the awareness of participants of the existing
programme evaluation in the Romanian public sector
through the Interim Evaluation of Phare programmes;
and
 identify lessons from interim evaluation for the
Romanian Administration in the further use of
evaluation in Public Administration.
Approach to Session 2
 3 Components.
 For each component:
 Short presentations (max 10 minutes)
 Question and answer/ discussion (max 15 minutes)
 Conclusion (5 minutes)
 Session Conclusions (max 15 minutes)
The three components
Component 1
 Short presentation on Phare interim evaluation (origins,
position in the Phare Management System, link between
IE and monitoring, the actors)
Component 2
 Short presentation on key lessons learned from IE reports
(Sector, Country Summary and Thematic Reports)
Component 3
 Short presentation on lessons learned for the future use of
IE in Romanian Public Administration, (structures,
management and quality issues in IE)
Supporting Material in the
Conference Binder
 Session Abstract
 Simplified Guide to Interim Evaluation
 Introduction to the IE Website
www.evaluarephare.ro
Component 1: Interim Evaluation is
Programme Evaluation in Romania
 Late 1990s: Results monitoring of Phare –
(Monitoring and Evaluation combined)
 2001: IE introduced when Phare became the major
pre-accession instrument (Separate Interim
Evaluation):
 IE is a management tool
 IE has very close links to the monitoring system
 There is a well established group of actors needed to
make IE successful
Current Interim Evaluation Project
 20 sector programme evaluation reports – 10 each for
2007 and 2008.
 3 Country Summary Reports for the Joint Monitoring
Committee meetings.
 3 Three Thematic evaluation reports on:
 Procurement practices and procedures
 Review of Results Indicators
 Phare lessons learned for structural funds
 1 Applied Methodologies Report
 1 Quality Assurance Review report
Effects of the Evaluation process
 Provides independent assessment of sectors and
programmes according to relevance, efficiency,
effectiveness, impact and sustainability
 Provides overall assessments for the Joint Monitoring
Committee meetings
 Alerts the authorities through the “Early Warning”
system
 Supports the monitoring system through
recommendations to improve
 Programme performance
 Systems for monitoring and control
The key elements of the IE system
Key evaluation capacity elements
 A sound methodology
 Independent evaluation capacity
 A Central Co-ordinating Unit
 A separate Quality Assurance Group
Wider elements
 A formal programme monitoring structure
 Adoption of the programme cycle management approach
 Support from High Civil Servants and the Political Branch
Discussion Questions for Component 1
 Is the Phare Interim Evaluation System a good model to
be used for Programme Evaluation in Romania?
 How should the system be (a) established (b)
resourced?
Component 2: Some Key Lessons
from IE Reports
 Programme objectives need to be aligned to National and Sectoral




strategies that are kept up to date – there is not enough attention
paid to updating the strategies.
Switching to a National Procurement System is a complex task
that needs careful planning, a lot of training and ad hoc support –
the current procurement system should be simplified.
Success comes from investing in people – the Romanian
administration needs to strengthen its training and development
structures and resources.
Monitoring systems should focus more on results than on
activities or inputs – Further results based reform is needed.
A modern strategic management system (e.g. for Structural
Funds) requires that responsibility for budgets and results should
be decentralised – there is too much control at the centre of the
administration.
Discussion Questions for Component 2
 Do you agree with the lessons from IE on the previous
slide?
 Who are the decision makers that are responsible for
each lesson?
 How can the programme evaluation process influence
the decision makers?
Component 3: Future use of Programme
Evaluation in the Romanian Administration
Lessons from the Interim Evaluation Scheme are:
 A properly defined and resourced structure for monitoring
and evaluation is needed for the Programme Evaluation
system to work.
 Communication of evaluation conclusions and
recommendations to decision makers is the most important
but difficult aspect of the evaluation process
Discussion Questions for Component 3
 What is needed for the Programme Evaluation process to be
more effective?
 Is the Ministry of Finance a good place for the Central coordination of Programme Evaluation activity?
 Should all Ministries and Agencies have a programme
evaluation capacity?
 Should all programmes be subjected to programme
evaluation?
 If Yes – at what frequency?
 If No – how should be programmes to be evaluated be selected?
A final question
 A Programme Evaluation system is expensive to create
– So, how should the Value of Programme Evaluation
be assessed?