Transcript Slayt 1

Bölgesel Rekabet Edebilirlik Operasyonel Programı’nın
Uygulanması için Kurumsal Kapasitenin Oluşturulmasına
Yönelik Teknik Yardım
This project is co-financed by the European
Union and the Republic of Turkey
Technical Assistance on Institutional Building for the
Implementation of RCOP in Turkey
Evaluation: main concepts,
types and criteria
Ankara, 15 December 2011
Laura Trofin (PhD): TAT Non-Key Expert , Monitoring and Evaluation
[email protected]
1
Structure
This project is co-financed by the European
Union and the Republic of Turkey
Part 1
• What is evaluation?
• Different types of evaluation and evaluation
systems
• The evaluation cycle
Part 2
• The evaluation criteria – relevance, effectiveness,
efficiency, impact, sustainability
Group exercise 3
IPA Funds Monitoring and Evaluation 12-16 December 2011
2
What is evaluation?
This project is co-financed by the European
Union and the Republic of Turkey
1.
Definitions of evaluation
2.
Evaluation, monitoring and audit
3.
Purpose of evaluation
4.
Principles of evaluation
5.
What happens as a result of evaluation?
IPA Funds Monitoring and Evaluation 12-16 December 2011
3
Defining Evaluation
This project is co-financed by the European
Union and the Republic of Turkey
• Some definitions:
• “A judgement of interventions according to the results, impacts and
needs they aim to satisfy” (EU Commission)
• “…the process of assessing the extent to which project, programme or
policy objectives have been achieved and how economically and
efficiently” (Economic and Financial Evaluation: Measurement,
Meaning and Management, Michael Mulreany, IPA Ireland, 1999,)
• “A critical and detached look at the objectives and how they are being
met” (UK Treasury)
• Involves judgement on basis of criteria
• More comprehensive than monitoring
• Applies to policies, programmes and projects
IPA Funds Monitoring and Evaluation 12-16 December 2011
4
Evaluation, monitoring and audit
This project is co-financed by the European
Union and the Republic of Turkey
• Evaluation
• Assessment of the efficiency, effectiveness, impact,
relevance and sustainability
• Monitoring
• Ongoing analysis of project progress towards achieving
planned results with the purpose of improving
management decision making
• Audit
• Assessment of the (i) legality and regularity of project
expenditure and income…(ii) whether project funds
have been used…in accordance with sound financial
management...and (iii)…for purposes intended.
IPA Funds Monitoring and Evaluation 12-16 December 2011
5
Monitoring is….
This project is co-financed by the European
Union and the Republic of Turkey
• …an ongoing, continuous, systematic process
• Some monitoring questions
• How much has been spent?
• What did we get for it?
• Who benefited?
• Are we on track?
• Uses financial and performance indicator data
IPA Funds Monitoring and Evaluation 12-16 December 2011
6
Audit…
This project is co-financed by the European
Union and the Republic of Turkey
• …is always carried out by professionally qualified
auditors
• …aims to provide professional assurance
• …is sometimes divided between a financial audit
and a performance audit.
• Performance audits are similar to evaluation but
confide their studies to an analysis of efficiency,
economy and effectiveness.
IPA Funds Monitoring and Evaluation 12-16 December 2011
7
Monitoring and Evaluation
This project is co-financed by the European
Union and the Republic of Turkey
•
Monitoring and evaluation are linked processes
• But monitoring is ongoing
• Evaluation is discrete
•
Evaluators use monitoring information
• Indicators
•
Monitoring acts as an early warning system
• Highlights problems or areas that require
evaluation
IPA Funds Monitoring and Evaluation 12-16 December 2011
8
Purpose of Evaluation
This project is co-financed by the European
Union and the Republic of Turkey
• 3 main purposes
• To address accountability concerns
• To assist in the allocation of resources
• To help improve programme management?
• Overarching Purpose
“To learn through systematic enquiry how to better
design, implement and deliver public programmes
and policies” (EU Guide)
IPA Funds Monitoring and Evaluation 12-16 December 2011
9
4 Principles of evaluation
This project is co-financed by the European
Union and the Republic of Turkey
•
(EuropeAid Project Cycle Management
Guidelines)
1.
Impartiality and independence – from
programming and implementation
2.
Credibility – use of skilled and independent
experts
3.
Participation of stakeholders – to ensure
different perspectives are taken into account
Usefulness – of findings and recommendations
4.
IPA Funds Monitoring and Evaluation 12-16 December 2011
10
Evaluation Characteristics
This project is co-financed by the European
Union and the Republic of Turkey
• Evaluations should be:
• Analytical
• Systematic
• Reliable
• Issue-oriented
• User-driven
IPA Funds Monitoring and Evaluation 12-16 December 2011
11
This project is co-financed by the European
Union and the Republic of Turkey
What happens as a result of
evaluation?
• Continuation of the programme as planned?
• Minor reorientation of the programme (management
issues, etc.)?
• Major restructuring of the programme (change of
beneficiaries, etc.)?
• Stopping of the programme?
• Change of future projects or programmes, taking into
account the lessons learned?
• Change of policies and subsequent programming?
IPA Funds Monitoring and Evaluation 12-16 December 2011
12
This project is co-financed by the European
Union and the Republic of Turkey
Different types of evaluation
and evaluation systems
1.
Formative v summative evaluation
2.
Impact evaluation/experimental design v
theory-based evaluation
3.
External v internal evaluation
4.
Centralised v decentralised evaluation
IPA Funds Monitoring and Evaluation 12-16 December 2011
13
This project is co-financed by the European
Union and the Republic of Turkey
Formative v summative
evaluation
•
Summative evaluation
• Accountability focus
• What has been achieved?
• What's the value of a programme?
• Is this programme worthwhile?
•
Formative evaluation
• Development or learning focus
• How can we improve performance and delivery of
programme?
•
Both relevant and useful to public sector
IPA Funds Monitoring and Evaluation 12-16 December 2011
14
This project is co-financed by the European
Union and the Republic of Turkey
Impact evaluation v theory based
evaluation
Impact evaluation/experimental design
• Primarily concerned with asking “how do we know if
the programmes and projects we are evaluating are
successful?”
• Related more with summative evaluation
• Tries to “scientifically” combat 2 problems:
1. what is the project responsible for?
2. will implementers only make positive data/views
available?
ED chooses an experimental group, and a control group
IPA Funds Monitoring and Evaluation 12-16 December 2011
15
This project is co-financed by the European
Union and the Republic of Turkey
Impact evaluation/
Experimental design
The problems with ED are:
1. method-driven
2. asks what works? But not why, or how?
3. ignores side-effects
4. elitist – ignores the views of stakeholders?
IPA Funds Monitoring and Evaluation 12-16 December 2011
16
This project is co-financed by the European
Union and the Republic of Turkey
Impact evaluation v theory
based evaluation
• Theory-based evaluation
• TBE criticises the importance given to methodology
• Programmes are complex, implemented in diverse
environments.
• Therefore we need theories on what programmes will
work where and why.
• ED can be inconsistent. What if the same programme
in 5 cities brings different results. Contexts can be
ignored.
• Will a programme or project that is successful in one
setting be similarly successful in an alternate, broadly
similar setting?
• Related more with formative evaluation
IPA Funds Monitoring and Evaluation 12-16 December 2011
17
Theory-based evaluation
This project is co-financed by the European
Union and the Republic of Turkey
• Main problem with TBE…
• If programmes can work in a variety of
different ways, how do we choose a
hypothesis which can be tested in the first
place?
IPA Funds Monitoring and Evaluation 12-16 December 2011
18
External v internal evaluation
This project is co-financed by the European
Union and the Republic of Turkey
• Strengths of internal evaluation model
• Helps to improve evaluation demand
•
better TOR, quality control – if skilled
• Better than external for some work
• Indicators
• Formative evaluation, how can programme be
improved?
• Weaknesses
• Will not have sectoral expertise for some work
• May not be perceived as independent
• Difficult to recruit and retain skilled staff
IPA Funds Monitoring and Evaluation 12-16 December 2011
19
External v internal evaluation
This project is co-financed by the European
Union and the Republic of Turkey
•
•
Strengths
• Flexibility
• Wider range of expertise
• Better at summative evaluation, i.e. is the
programme worthwhile?
Weaknesses
• Can be expensive
• Doesn’t help develop internal capacity
(sometimes)
• Commercial pressures may limit independence
IPA Funds Monitoring and Evaluation 12-16 December 2011
20
This project is co-financed by the European
Union and the Republic of Turkey
Centralised v decentralised
evaluation
• Centralised model (Ireland, CSF 2000-2006)
• Organized by MOF or central evaluation unit
• Ensures focus on cross-OP issues
• Consistent approach
• Lower evaluation costs for MAs
• Unit looks after technical work (TOR, indicators)
• MAs can focus on core management tasks
• OP evaluations more credible because external to MA
• Decentralised (Ireland, CSF 1994-1999)
• Responsibility of each OP MA
• Allows MAs to tailor evaluations specifically to their needs
• Can call on MACSF ECU for technical expertise
• Romania 2007-2013 - mix between centralised/decentralised
IPA Funds Monitoring and Evaluation 12-16 December 2011
21
Other types
This project is co-financed by the European
Union and the Republic of Turkey
• Ad-hoc
• Thematic
• Horizontal
• Strategic
• Operational
IPA Funds Monitoring and Evaluation 12-16 December 2011
22
The Evaluation Cycle
This project is co-financed by the European
Union and the Republic of Turkey
• Before (ex-ante evaluation)
• Aim is to improve allocation of resources and
programme design
• During (interim evaluation)
• External developments and their implications?
• Is the programme meeting its objectives?
• Can we improve programme management?
• After (ex-post evaluation)
• What has been achieved?
• What difference did it make?
IPA Funds Monitoring and Evaluation 12-16 December 2011
23
Part 2 – Evaluation Criteria
This project is co-financed by the European
Union and the Republic of Turkey
• What's the basis for evaluation judgements?
• Common EU approach has 5 criteria
• Relevance (including Rationale)
• Effectiveness
• Efficiency
• Impact
• Sustainability
IPA Funds Monitoring and Evaluation 12-16 December 2011
24
Relevance
This project is co-financed by the European
Union and the Republic of Turkey
-Are socio-economic development programmes
relevant to the needs of stakeholders?
-Have circumstances changed since the start of the
programme?
-Do these changes render the programme irrelevant?
-Why is public money delivering the programme?
-Could the private sector not meet the needs of the
stakeholders?
-Where is the market failure?
IPA Funds Monitoring and Evaluation 12-16 December 2011
25
Effectiveness
This project is co-financed by the European
Union and the Republic of Turkey
• Are the programmes achieving their objectives?
• Are outputs being produced that correspond to
the needs as expressed in the programme
design?
IPA Funds Monitoring and Evaluation 12-16 December 2011
26
Efficiency
This project is co-financed by the European
Union and the Republic of Turkey
• Are the programmes providing value-for-money?
• Could the outputs be produced cheaper?
• Are unit costs too high?
• Even though targets may be reached, are they
being reached in a way that makes the programmes
too costly to continue?
IPA Funds Monitoring and Evaluation 12-16 December 2011
27
Impact
This project is co-financed by the European
Union and the Republic of Turkey
• What changes as a result of the programmes?
• Are there benefits to the programmes simply
beyond their outputs?
IPA Funds Monitoring and Evaluation 12-16 December 2011
28
Sustainability
This project is co-financed by the European
Union and the Republic of Turkey
• Will the effects of the programmes, or indeed the
programmes themselves, have a life beyond their
implementation date?
• Will alternative sources of funding be found?
IPA Funds Monitoring and Evaluation 12-16 December 2011
29
Summary/Concluding Points
This project is co-financed by the European
Union and the Republic of Turkey
• Relationship between monitoring and evaluation
• Purpose of evaluation ….
• Need to have an evaluation framework
• Evaluation questions or criteria
• Focus varies over policy/programme cycle
IPA Funds Monitoring and Evaluation 12-16 December 2011
30
Group exercise 3
This project is co-financed by the European
Union and the Republic of Turkey
Define the type of evaluation RCOP would need in 20122013, and the evaluation criteria it should cover
THINK ABOUT:
• moment in time
• purpose of programme and of evaluation
• results
• type (internal/external, interim/ex-ante,
summative/formative, theory/experimental,
centralised/decentralised)
• criteria
• extra - data which you will use
IPA Funds Monitoring and Evaluation 12-16 December 2011
31