Transcript Slide 1

ROM reviews
Saskia Van Crugten
NSA-LA Thematic programme - Management of the Project Cycle and
Monitoring of Project Implementation
Brussels, 17-18 September 2013
Origin of ROM reviews
May 1999: the Council of the EU invited the EC to establish a
performance-based monitoring system with the objective to
strengthen Monitoring, Evaluation and Transparency.
EuropeAid established the Results-Oriented Monitoring
system (ROM).
Initially, ROM bridged the gap between EC Task Managers
(Brussels) and the project (in the field)…
…today however,
most EC Project Manager work in Delegations.
22
Main features of ROM system
 OECD DAC evaluation criteria
 PCM / Logical Framework Approach
 For on-going and closed projects/programmes (on-going
ROM, ex-post ROM, regional ROM)
 Standard methodology for all projects
 Selection of projects for ROM
 Mission procedure
 Questions guiding the assessment
 Report formats
 Grading system
 External experts
33
Selection of actions for ROM
 Interactive process between ROM contractors, EU Del, ROM
Task Manager, Headquarters
 All projects with budgets exceeding 1 million € and a sample
of smaller projects
 All projects that have been implemented for at least six
months and at least six months remaining before closure
Implementation
Start
End
6 months
6 months
ROM period
 ROM should be coordinated with evaluation activities (e.g.
not within six months of a mid term review).
 Currently, roughly 1/3 of DEVCO projects/programmes are 44
“ROMed” every year.
Timing of ROM missions
The ROM exercise is to be timely and quick.
 5 days for field visit of an ongoing project.
 1 day preparation
 2 days reporting.
Ex-post ROM can require more days (up to 10) especially to
identify interlocutors.
ROM reports available 15 days after the field visit.
55
ROM documents
 Project Synopsis (PS): a concise overview on project
background, intervention logic and administrative data.
 Background Conclusion Sheet (BCS): guiding questions
to assess a project and give grades for each of the criteria
 ROM Report (MR): findings and grades for ROM criteria,
conclusions and recommendations.
 Response Sheet (RS): comment on report quality, reply
to recommendations and feedback from other stakeholders
(to be filled in by Task Manager).
66
ROM assessment criteria - BCS
Relevance
•1.1 For target group
•1.2 For partner government
•1.3 For E.U.
Quality of design
•1.1 Intervention logic
•1.2 Stakeholder support
•1.3 Crosscutting issues in design
Efficiency
•2.1 Availability of means / inputs
•2.2 Implementation of activities
•2.3 Output achievement
•2.4 Partner contribution
Effectiveness
•3.1 Output achievement
•3.2 Outcome achievement
Impact Prospects
•4.1 Direct impact
•4.2 Indirect impact
Potential Sustainability
•5.1 Financial/ economic viability
•5.2 Target group ownership
•5.3 Policy support
•5.4 Institutional capacity
Horizontal Issues
•6.1 Quality Systems, Monitoring
and Evaluation
•6.2 Technical Cooperation
•6.3 EC visibility
Cross-cutting issues
•7.1 Gender
•7.2 Environment
77
•7.3 Good Governance
•7.4 Human Rights
Using the BCS
2.
1.
88
ROM grading system
99
ROM performance categories
 Made to describe the performance of a project as a whole;
 not quantitative averages of the grades for the
performance criteria, but qualitative categorisations.
Category
I
II
Subcategory
Very good performance
Good performance
II a
II b
II c
III
IV
Performing with problems
Not performing, major difficulties
Definition
At least three "a", no "c", no "d"
No "c", no "d"
Maximum of one "c", no "d"
Maximum of two "c", no "d"
At least three "c", no "d"
At least one "d"
10
Why ROM reviews ?
 For project managers and EC task managers: ROM
informs stakeholders on project performance and helps
to focus on results.
 For higher level of EC management: ROM provides
statistics on overall portfolio performance and highlights
problems and success stories.
 For knowledge production: ROM can be a source of
lessons learnt via meta-analysis/evaluations).
 For project beneficiaries: opportunity to discuss
project progress and outputs.
11
11
What a ROM review is not ?
 An Audit
 A mechanism replacing internal monitoring by EC staff
 An in-depth analysis of a project/ programme
 An assessment of EC Task Managers performance
12
ROM and Quality assurance in the
project cycle
Final
Project
Evaluati
on
Programm
ing
Follow
-Up
Identificati
on
-oQS
ClosureG1
PPCM
Formulatio
n
oQSG
2
Internal
Monitoring
Implementatio
n
DEL
managed
HQ
managed
Project
Mgmt
Accountabili
ty
Lessons
Learnt
13
13
ROM reviews for supporting
project management
Using external experts ROM offers fresh eyes to scrutinize a
project:
 Is the project, with all due flexibility, pursuing its planned
course and will it attain its objectives?
 What can or should be done to improve project performance?
 Are there issues which could be subject to a more in-depth
analysis (evaluation)?
14
14