Transcript Slide 1

Interim Evaluation of EU Pre-Accession
Programmes in Turkey
(Service Contract No. 2007/142-454)
Kick-off for
2008 Monitoring and Evaluation
25 February 2008
A project financed by the European Commission.
The views expressed are those of the MWH Consortium
and do not necessarily reflect those of the European Commission
IE Contract - The MWH Consortium
2
IE Turkey - Team and Contact Details
Contract
start/finish
Team Leader
Experts
6 September 2007 – 5 December 2008
(15 months)
Ms. Liz Cunningham
Mr. Steven O’Connor
Ms. Sevil Geveci
Ms. Ayla Hekimoğlu
Contact details
Mr. Kamil Sorgun
Bogaz Sokak, 27/5,06700 G.O.P.
Ankara.
Tel: 0312 467 97 87
3
Interim
Evaluation
Framework
Secretariat
General for EU
Affairs
DG Enlargement - Unit E4
Consultation
System Feedback
Follow-up of IE
Recommendations
Delegation of the
European
Commission
Task Manager – Jan Behrens
Contract
Methodology
Quality Control
System Oversight
Planning
Coordination
Feedback
MWH Consortium
Interim Evaluation Team
Interim Evaluation
Input
IE Conclusions &
Recommendations
Project Stakeholders
4
Interim Evaluation – What it is NOT
X Interim Evaluation is not audit....
X Interim Evaluation is not control.....
X Importantly, Interim Evaluation is not
monitoring by the Commission
services
5
Interim Evaluation – What it IS
 Part of a general policy of evaluating all Community
expenditure
 Helps to improve relevance, effectiveness, impact and
accountability of EU funds
 Provides stakeholders with independent assessments
of the state of implementation
 Participative and capacity building exercise
6
Interim Evaluation – WHY we do it
 To assess the state of implementation of the projects
 To assess the state of implementation of initiatives in the
sector
 To provide information to improve implementation of
ongoing projects and sector performance
 To provide lessons learned for future projects
 To provide lessons learned in relation to the project cycle
and its implementation
7
How do monitoring and evaluation differ?
They are different, but interrelated functions,
as they both contribute knowledge as a basis
for accountability and enhanced performance;
– Monitoring: “Are we doing things right”?
while
– Evaluation, in addition, “Are we doing the
right things?” and “Are there better ways of
achieving the results?”
Definition of monitoring,
evaluation and audit
The Process – The 5 Criteria
Relevance: Design issues; Were real needs correctly
identified, are they being met? Are indicators sufficient?
Efficiency: How well the activities were transformed
into results; ‘have things been done right?’
Effectiveness: Were results used? Did the project
achieve its purpose?
Impact: Immediate, intermediate and wider/global
Sustainability: Will outcomes continue after the
project has ended?
10
Different Kinds of Impact
Criterion
Definition
Focus
Immediate
impact
The extent to which outputs are Direct beneficiaries
being used by the
beneficiaries
(Output: what is now in place that
was not there before)
Intermediate
impact
The extent to which the outputs
are used by others not directly
targeted by the project (‘spin
offs’; planned/unplanned or
positive/negative)
Direct and indirect
beneficiaries
Wider/global
impact
The contribution of the outputs,
immediate, intermediate impacts
to socio-economic
developments
Society in general
11
Example of Impacts
Example: Laboratory equipment for testing hazardous
chemicals
Output
• Laboratory
equipment
installed
Immediate
Impact
• Equipment in
use and data
generated in
use
Intermediate
Impact
• Other
laboratories
upgraded in
line with
standards
set
Wider/global
Impact
• EU
standards for
testing and
response to
dangerous
chemicals
met; health
of population
protected
12
Ratings – Six levels
Summary
HS
Definition
Explanation
Highly Satisfactory
All specified objectives expected to be
achieved within lifetime of the project
S
Satisfactory
The project is expected to largely achieve its
objectives
MS
Moderately
Satisfactory
Not fully satisfactory but not weak enough to
deserve a bad rating
MU
Moderately
Unsatisfactory
Not fully unsatisfactory, but not good enough
to justify a good rating
U
Unsatisfactory
Most of the objectives will not be achieved
HU
Highly
Unsatisfactory
The project is not expected to achieve any of
its objectives
13
Interim Evaluation – Rating Model
Criterion
Sustainability
Overall
rating
↓→
↓→
→•
↓→
↓→
↓→
→•
→
→
→
→•
Relevance
Efficiency
↓→
↓→
↓→
Total for sub-sector
↓→
↓→
Total for the Sector
→
→
Project
Effectiveness Impact
Sub-sector 1 title
Project title 1
Project title 2
Project title 3
Total for sub-sector
Sub-sector 2 title
Project title 4
Project title 5
14
Interim Evaluation - Information Base

Documentation (Accession Partnership, Regular
Reports, national strategies, policy documents,
Monitoring Reports, project and programme level etc.)

Financial data (up-to-date status)
 Structured Interviews with stakeholders (policy
makers, management, EUSG, ECD, CFCU, experts,
contractors, Twinning Advisers, beneficiaries etc.)

Surveys/questionnaires: depending on project

Other analysis as required (e.g. databases)
15
Information Base – Documents Required
• Monitoring Reports
Good quality Monitoring Reports (on time, completed
correctly, accurate and up to date information)
• Other documentation/Per Project
1. Project Fiche
2. Technical Assistance – ToR, Inception Report, Progress
Reports, important output documents (TA reports etc.),
Minutes of meetings etc.
3. Twinning – Contract, Quarterly Reports, output documents,
Minutes of meetings etc.
4. Supplies/works – tender dossier, reports of evaluation
committees etc. as required
5. Grant schemes – guidelines, list of applicants/grantees
16
IE Reporting Process – The Results
1. Findings are presented in a draft Sectoral Interim
Evaluation Report containing:
 Conclusions and Recommendations of the
independent Evaluators
 Sectoral recommendations
 Operational recommendations aimed at specific
projects
 ‘Early warnings’
2. The draft report is finalised taking into account the
comments as and where Evaluators judge necessary
3. The finalised report is debriefed where stakeholders
discuss the recommendations and agree a plan of
action
17
Interim Evaluation Reports 2007-2008
Interim Evaluation – Linked to SMSC
Sector
Current Status
1
Internal Market
Completed
2
Political Criteria
At final stages
3
Environment/Transport
Ongoing
4
Ongoing
5
Public Finance, Statistics, Support to
European Integration
Cross Border Cooperation
6
Regional Competitiveness
Starts early March 2008
7
Human Resource Development
Starts mid April 2008
8
Civil Society
Starts late April/early May 2008
9
Rural Development
Starts mid May 2008
Starts early March 2008
18