Transcript Slayt 1

Bölgesel Rekabet Edebilirlik Operasyonel Programı’nın
Uygulanması için Kurumsal Kapasitenin Oluşturulmasına
Yönelik Teknik Yardım
This project is co-financed by the European
Union and the Republic of Turkey
Technical Assistance on Institutional Building for the
Implementation of RCOP in Turkey
MONITORING & EVALUATION
for RCOP
Ankara, 12 – 16 December 2011
Monitoring & Evaluation, 12-16 Dec 2011
1
This project is co-financed by the European
Union and the Republic of Turkey
SECTION I.
MONITORING VS VALUATION
•
Clarification of definitions
Monitoring & Evaluation, 12-16 Dec 2011
2
This project is co-financed by the European
Union and the Republic of Turkey
Distinction between evaluation,
monitoring and audit
• Evaluation • Assessment of the efficiency, effectiveness,
impact, relevance and sustainability of aid policies and actions
• Monitoring • Ongoing analysis of programme/project progress
towards achieving planned results with the purpose of
improving management decision making
• Audit • Assessment of
 the legality and regularity of project expenditure and income i.e.
compliance with laws and regulations and with applicable
contractual rules and criteria;
 whether project funds have been used efficiently and
economically i.e. in accordance with sound financial
management;
 whether project funds have been used effectively i.e. for
purposes intended.
 Primarily a financial and financial management focus, with the
focus of effectiveness being on project results.
Monitoring & Evaluation, 12-16 Dec 2011
3
This project is co-financed by the European
Union and the Republic of Turkey
Evaluation-monitoring-project
management
• Monitoring: to be aware of the present situation
• Evaluation: to rationally connect the present situation with
the desired future situation
• Project management: operations and transactions
(contracting, approval of reports, payments…) but activities
leading to these rely on monitoring and to (interim)
evaluation outcomes
Potential overlaps of monitoring and project management
activities
Segregation of duties, clear definition of responsibilities
Cooperation
Monitoring & Evaluation, 12-16 Dec 2011
4
This project is co-financed by the European
Union and the Republic of Turkey
Distinction between and
evaluation monitoring
Monitoring
• What? Monitoring is an integral part of a day-to-day management.
• How? Monitoring embodies the regular tracking of inputs, activities,
outputs, reach, outcomes, and impacts of development activities at the
project program, sector and national levels
• Why?Monitoring provides information by which management can
identify and solve implementation problems and assess progress
towards project's objectives
Evaluation
• What? Evaluation is an assessment that refers to design,
implementation and results of completed or on-going project / program /
policy.
• How? Evaluation should be systematic and objective. Key criteria to be
used are: relevance, developmental efficiency, effectiveness, impact
and sustainability.
• Why?Evaluation should provide credible and useful information to
enable the incorporation of lessons
Monitoring & Evaluation, 12-16 Dec 2011
5
This project is co-financed by the European
Union and the Republic of Turkey
Distinction between and
evaluation monitoring
Monitoring
• focused on daily management issues
• typical questions: “How many?” "When?” “How?” “For
how much?”
• assess whether activities are implemented effectively
and efficiently
Evaluation
• addresses strategic questions: “So what?”(impact and
sustainability) and “Why?” (relevancy)
• analysis is deeper and seeks for actual cause-results
relationships
• perception of “big picture"
Monitoring & Evaluation, 12-16 Dec 2011
6
This project is co-financed by the European
Union and the Republic of Turkey
Distinction between and
evaluation monitoring
Monitoring
• usually means a system
• data collected and analyzed more or less frequently
• according to a predefine timetable (Performance
Measurement Plan)
• regularity and continuity of data collection
• methodology used to analyze it
Evaluation
• flexibility in specifying, which aspects of the program
should be assessed, when and how.
Monitoring & Evaluation, 12-16 Dec 2011
7
This project is co-financed by the European
Union and the Republic of Turkey
Distinction between and
evaluation monitoring
Monitoring
• Part of modern project management
• Generates useful information for project manager
• where are bottlenecks?
• how are we doing towards our objectives?
• are expenses under control?
• Utility is the primary feature
Evaluation
• Target groups: donors, planners, assistance recipients and wider
public
• Have we achieved our goals?
• Are our results sustainable?
• Have we learned anything for the future?
• Focus on
• transparency of evaluator’s approach
• revealing cause-effect relationships between subsequent
layers of analysis.
Monitoring & Evaluation, 12-16 Dec 2011
8
This project is co-financed by the European
Union and the Republic of Turkey
Distinction between and
evaluation monitoring
Monitoring
• Project management needs  rapid assessment
methods
• fast feedback
• not very expensive
Evaluation
• Scrupulous research methodologies
• representative surveys
• comprehensive quantitative analyzes
Monitoring & Evaluation, 12-16 Dec 2011
9
M&E – Comparative characteristics
This project is co-financed by the European
Union and the Republic of Turkey
Characteristics
Evaluation
focused
Subject:
usually
aspects
Character:
incidental, flexible subject &
methods
continuous, regular, systematic
Primary client:
stakeholders and external audience
programme management
Approach:
objectivity, transparency
utility
Methodology:
rigorous research methodologies,
sophisticated tools
rapid appraisal methods
Primary focus:
focus on relevancy, outcomes,
impact and sustainability
focus on operational efficiency and
effectiveness
Objectives:
to check outcomes / impact, verify
developmental hypothesis
to identify and resolve implementation
problems
to document successes and
lessons learned
to assess progress towards objectives
Monitoring & Evaluation, 12-16 Dec 2011
on
Monitoring
strategic
addresses operational management issues
10
M&E system diagramme
This project is co-financed by the European
Union and the Republic of Turkey
OVERALL
OBJECTIVES
Evaluation
INTERVENTION
LOGIC
SOURCES OF DATA
M&E TOOLS
FOLLOW-UP
EXTERNAL REPORTS AND DATA
FINAL EVALUATION, SURVEY,
STATISTICAL ANALYSIS
FINAL EVALUATION REPORT
PROGRAMMING
BENEFICIARIES OPINION
MID-TERM EVALUATION
BENEFICIARIES SURVEY
MID-TERM EVAL. REPORT
REVISING CONTRACTS
EXTERNAL EXPERTS' OPINION
STRUCTURED INTERVIEW,
FOCUS GROUPS
QUARTERLY MONITORING REPORT
QUANTITAIVE DATA ON
PROJECT PROGRESS
PIPELINE ANALYSIS
MANAGEMENT MEETINGS
IMPLEMENTERS' PERIODIC
REPORTS
REVIEW OF PERIODIC REPORTS
REPORT APPROVAL PROCESS
PRODUCTS DELIVERED BY
CONTRACTORS
REVIEW OF PRODUCTS
AND SERVICES
PRODUCT REVIEW REPORT
DIRECT OBSERVATIONS
SITE VISITS
SITE VISIT REPORT
PROJECT
PURPOSE
ACTIVITIES
MEANS &
RESOURCES
Monitoring
RESULTS
Monitoring & Evaluation, 12-16 Dec 2011
11
This project is co-financed by the European
Union and the Republic of Turkey
SECTION II.
EVALUATION
• EU Regulations on evaluation
• Introductory guide on basic notions
Monitoring & Evaluation, 12-16 Dec 2011
12
COUNCIL REGULATION
1085/2006 establishing IPA
This project is co-financed by the European
Union and the Republic of Turkey
Article 22 Evaluation
• Commission shall regularly evaluate
• the results and efficiency of policies and
programmes
• effectiveness of programming whether the
objectives have been met
• Objectives of Evaluation
• to enable the COM to formulate recommendations
• results shall feed back into programme design
and resource allocation
Monitoring & Evaluation, 12-16 Dec 2011
13
This project is co-financed by the European
Union and the Republic of Turkey
COM REGULATION 718/2007
implementing 1085/2006
• Title II Common rules for implementation/Chapter V
Evaluation and monitoring
• Article 57 Evaluation
• Objectives: to improve the quality of
 effectiveness and consistency of the assistance
 strategy and implementation of the programmes.
• Subject: Multi-annual indicative planning documents (ex
ante evaluation)
• Forms:




Strategic
Ex ante
Interim
Ex post
Monitoring & Evaluation, 12-16 Dec 2011
14
This project is co-financed by the European
Union and the Republic of Turkey
COM REGULATION 718/2007
implementing 1085/2006
Article 57 Evaluation (cont’d)
• Interim:
 At least one during implementation
 Specifically when the monitoring reveals significant
departure from the goals initially set.
• Ex post
 the responsibility of the Commission
 Shall include identifiable IPA component-specific results
• The results of ex ante and interim evaluation shall be taken
into account in the programming
Monitoring & Evaluation, 12-16 Dec 2011
15
This project is co-financed by the European
Union and the Republic of Turkey
COM REGULATION 718/2007
implementing 1085/2006
CHAPTER III/Implementation/Section 1 Framework for
implementation and principles
Article 78 Implementation principles in the event of
participation in Community programmes and agencies (as
amended by COM REGULATION 80/2010)
• All evaluations shall be carried out by the COM prior to the
conferral of management powers on the beneficiary country,
• After the conferral




BC is responsible for interim evaluation;
COM’s rights to perform any ad-hoc evaluations remains;
ex-ante and ex-post remain with COM
BC’s right to carry out ex-ante and ex-post as it deems
necessary.
Monitoring & Evaluation, 12-16 Dec 2011
16
This project is co-financed by the European
Union and the Republic of Turkey
COM REGULATION 718/2007
implementing 1085/2006
Title III Regional development and human resources development/Chapter III
Implementation/Section 3 Evaluation and monitoring
Article 166 Evaluation
• ex ante
• Beneficiary countries - under the responsibility of the operating structure.
• For each operational programme separately.
• If duly justified and in agreement with the COM BC may carry out a single ex ante
evaluation covering more than one operational programme.
• Shall aim to
• optimise the allocation of budgetary resources under operational programmes
• improve programming quality
• identify and appraise
• the disparities, gaps and potential for development,
• the goals to be achieved,
• the results expected,
• the quantified targets,
• The coherence of the strategy proposed
• and the quality of the procedures for implementation, monitoring,
evaluation
• Annexed to OP
Monitoring & Evaluation, 12-16 Dec 2011
17
This project is co-financed by the European
Union and the Republic of Turkey
COM REGULATION 718/2007
implementing 1085/2006
Article 166 Evaluation (cont’d)
• Interim
• During the programming period
• Beneficiary countries
• Linked to the monitoring of OPs, in particular where
•
•
this reveals a significant departure from the goals initially set
or
proposals are made for the revision of OPs
• Results shall be sent to the sectoral monitoring committee for
the OP and to COM
• Evaluations shall be carried out by experts or bodies, internal
or external, functionally independent of the authorities
• The results shall be published
Monitoring & Evaluation, 12-16 Dec 2011
18
Purpose and principles
This project is co-financed by the European
Union and the Republic of Turkey
Logical elements of purpose of evaluation:
•
•
•
•
•
Assessment;
Systematic and objective;
Ongoing or completed project, programme or policy;
Design, implementation and results;
Determine the relevance and fulfillment of objectives,
developmental efficiency, effectiveness, impact and
sustainability;
• Provide information that is credible and useful;
• Enabling the incorporation of lessons learned into the decisionmaking process of both recipients and donors;
Monitoring & Evaluation, 12-16 Dec 2011
19
This project is co-financed by the European
Union and the Republic of Turkey
Purpose and principles
Principles of the approach to evaluation:
• Impartiality and independence of the evaluation process
from the programming and implementation functions;
• Credibility of the evaluation, through use of appropriately
skilled and independent experts and the transparency of the
evaluation process, including wide dissemination of results;
• Participation of stakeholders in the evaluation process, to
ensure different perspectives and views are taken into
account;
• Usefulness of the evaluation findings and recommendations,
through timely presentation of relevant, clear and concise
information to decision makers.
Monitoring & Evaluation, 12-16 Dec 2011
20
Evaluation types
This project is co-financed by the European
Union and the Republic of Turkey
Ex ante
Article 2 of the Financial Regulation
• Use of budget appropriations
• principles of sound financial management
• quantified objectives must be identified
• „mobilisation of Community resources must be
preceded by an evaluation to ensure that the
resultant benefits are in proportion to the
resources applied.”
Monitoring & Evaluation, 12-16 Dec 2011
21
Evaluation types
This project is co-financed by the European
Union and the Republic of Turkey
Ex ante
• Implementation Rules for FR – ex ante evaluation shall
identify:
• need to be met in the short or long term;
• objectives to be realised;
• results expected and the indicators needed to measure
them;
• added value of Community involvement;
• risks, including fraud;
• lessons learned from similar experiences in the past;
• volume of appropriations, human resources and
administrative expenditures;
• monitoring system to be set up.
Monitoring & Evaluation, 12-16 Dec 2011
22
Evaluation types
This project is co-financed by the European
Union and the Republic of Turkey
Ex ante
• Process that supports the preparation of interventions
- related to programming ~ quality assurance
• Purpose: to gather and analyse information
• define objectives,
• ensure that these objectives can be met,
• that the instruments used are cost-effective,
• reliable later evaluation will be possible
• Independent from planners
Monitoring & Evaluation, 12-16 Dec 2011
23
Evaluation types
This project is co-financed by the European
Union and the Republic of Turkey
Interim
• Assessment of the quality of programme implementation
based on the first outputs and results
• relevance of the adopted strategy;
• newly occured factors having an impact on the
implementation;
• objectives have been defined accurately;
• indicators are relevant;
• so-far effectiveness and extrapolation;
• management quality of the project implementation;
• reliable collected data, including the monitoring system
• Independent from implementors
Monitoring & Evaluation, 12-16 Dec 2011
24
Evaluation types
This project is co-financed by the European
Union and the Republic of Turkey
• Ex post
• After programme completion
• Examining long-lasting effects
• impact (more) visible
• verification to what extent objectives have been
achieved
• Sustainability – estimated character
• Anticipated and unexpected effects
• Source of information for future programming
excercises
Monitoring & Evaluation, 12-16 Dec 2011
25
Major tasks for an Evaluation Unit
This project is co-financed by the European
Union and the Republic of Turkey
• Identifying the need for an evaluation and selecting the
topics/themes to be evaluated;
• Designing the evaluation, including preparing the Terms of
Reference;
• Drafting tender documents for the evaluation study and
selecting the contractor according to the established rules;
• Briefing the contractor and the parties involved, and
supporting the evaluation mission;
• Ensuring the production of a high quality evaluation report
and of the dissemination of evaluation findings and
recommendations.
Monitoring & Evaluation, 12-16 Dec 2011
26
Tools and key documents
This project is co-financed by the European
Union and the Republic of Turkey
Primary tools available to support a project evaluation:
• Terms of reference for the evaluation mission;
• The project’s Logframe matrix - to help assess what has
been achieved against plan;
• Programme planning documents;
• Monitoring reports (internal and external);
• IPA Interim Evaluation Reports;
• IPA Monitoring reports
• Minutes of SMCs and other organisations involved in
monitoring
Monitoring & Evaluation, 12-16 Dec 2011
27
Key documents produced
This project is co-financed by the European
Union and the Republic of Turkey
• Terms of Reference for the evaluation mission,
• Final Evaluation Mission Report
• should mirror the structure of the main evaluation
criteria
• taking into account
• the nature of the project,
• the stage at which the evaluation is carried out,
• the users for whom the report is prepared.
Monitoring & Evaluation, 12-16 Dec 2011
28
Evaluation Criteria
This project is co-financed by the European
Union and the Republic of Turkey
Relevance
• @ significance; pertinence
• To what extent do we do the right things ? Does it make
sense?
• ~ to the identified problems or real needs to be addressed
• The appropriateness of project objectives to
• problems that it was supposed to address
• physical and policy environment within which it operated
• The extent to which the objectives of a programme/project are
consistent with beneficiaries’ needs, country needs, global
priorities.
Monitoring & Evaluation, 12-16 Dec 2011
29
Evaluation Criteria
This project is co-financed by the European
Union and the Republic of Turkey
• Relevance (cont’d)
• Relates primarily to programme design
• extent to which correctly address the identified
problems or real needs. (Needs to be kept under
review throughout the life of the project in case changes
occur)
• logic and completeness of the project planning process
• internal logic and coherence of the project design
• at two points in time: when the project was
designed, and at the time of the evaluation.
• Focus
• identification of real problems or needs
• correct beneficiaries
Monitoring & Evaluation, 12-16 Dec 2011
30
Evaluation Criteria
This project is co-financed by the European
Union and the Republic of Turkey
Efficiency
• How well the various activities transformed the
available resources into the intended results
(outputs)
• quantity, quality and timeliness.
• value-for-money: similar at lower cost in the same time
• Focus: the quality of day-to-day management, i.e:
•
•
•
•
•
budget
personnel, information, property, etc;
risk and change
relations/co-ordination with actors
time (deadlines)
Monitoring & Evaluation, 12-16 Dec 2011
31
Evaluation Criteria
This project is co-financed by the European
Union and the Republic of Turkey
Effectiveness
• Contribution made by results to achievement of the
Project Purpose
• Benefits accruing to target groups
• Did we achieve our objectives? To what extent did our
outputs produce the desired outcomes?
• By how far the intended beneficiaries really benefited from
the products or services it made available?
• Focus:
• whether the planned benefits have been delivered and
received;
• appropriateness of the indicators of benefit used;
• how unplanned results may have affected the benefits received;
Monitoring & Evaluation, 12-16 Dec 2011
32
Evaluation Criteria
This project is co-financed by the European
Union and the Republic of Turkey
Impact
• Effect of the project on its wider environment, and its
contribution to the wider policy or sector objectives (as
summarised in the project’s Overall Objective).
• Outcome: relationship between the project’s purpose and
overall objectives
• Benefits received by the target beneficiaries had a wider overall
effect on larger group
• Focus:
• to what extent the planned overall objectives have been achieved,
• how far that was directly due to the project;
• unplanned impacts’ effects on overall impact;
• Ex: higher standard of living, increased food security,
democratic rule of law
Monitoring & Evaluation, 12-16 Dec 2011
33
Evaluation Criteria
This project is co-financed by the European
Union and the Republic of Turkey
Sustainability
• Continuation of benefits, effects generated by a programme/project
after its termination.
• Likelihood of benefits produced to continue to flow after external
funding has ended,
• Focus:
•
•
•
•
•
•
Ownership
Policy support and the responsibility of the beneficiary institutions
Institutional capacity, commitment
Economic and financial factors, socio-cultural aspects, gender equality
Technology, environmental aspects
Financial sustainability
• Ex: a micro-credit scheme that is generating enough money for the
scheme to operate, cover risks and develop its staff
Monitoring & Evaluation, 12-16 Dec 2011
34
A Quick reflection
This project is co-financed by the European
Union and the Republic of Turkey
• Appropriateness is not about money. It is about what
was achieved against what was needed.
• Effectiveness is not about money. It is about how
much it was achieved.
• Efficiency is about how much was achieved and how
much money was spent.
• Sustainability is not about money. It is about the
embedment of what was achieved in the given context
(natural, societal, economic…)
Monitoring & Evaluation, 12-16 Dec 2011
35
Decision options
This project is co-financed by the European
Union and the Republic of Turkey
Look out:
• one may be very efficient but not effective!
• or very effective but not relevant!
• to do things right is fine, but to do the right
things is for sure far more important!
Monitoring & Evaluation, 12-16 Dec 2011
36
Decision options
This project is co-financed by the European
Union and the Republic of Turkey
•
•
•
•
Continue project implementation as planned
Re-orient/restructure the project
To stop the project (mid-term evaluation);
Modify the design of future projects or
programmes in light of lessons learned (ex-post
evaluation)
• Modify policies, co-operation strategies, and
subsequent programming or identification
exercises (sector, thematic or cross-sector
evaluations)
Monitoring & Evaluation, 12-16 Dec 2011
37
Performance Ratings
This project is co-financed by the European
Union and the Republic of Turkey
•
Performance rating in two steps:
1.
•
2.
Numerical rating on each criteria for each component of the sector
from –2 to +2:
Conversion of numerical overall rating into a qualitative rating.
Qualitative ratings
• « Highly satisfactory » The programmes reviewed are expected
•
•
•
to achieve or exceed all the objectives set during their lifetime;
most numerical ratings in the range of 1 to +2
« Satisfactory » The programmes reviewed are expected to
largely achieve the objectives; most numerical ratings in the
range of 0 to 1
« Unsatisfactory » The programmes reviewed are not
expected to achieve most of the objectives; most numerical
ratings in the range of –1 to 0
« Highly unsatisfactory » The programmes reviewed are not
expected to achieve any of the objectives; most numerical
ratings in the range of -1 to –2
Monitoring & Evaluation, 12-16 Dec 2011
38
Quality Assessment of IE Reports
This project is co-financed by the European
Union and the Republic of Turkey
• Criteria – Rate – Remarks
• A. General: Does the report design appropriately fit the
evaluation?
• B. Sound sectoral overview: to what extent are the sector
composition and priorities appropriately described?
• C1. Sound analysis: to what extent are the facts and data
adequately analysed?
• C2. Sound analysis: to what extent have the indicators of
achievement been adequately considered and have they
been used properly where was possible?
• D. Robust Findings in the implementation evaluation: do
the Conclusions follow logically from, and are they justified by,
the data described in the Sectoral Overview?
Monitoring & Evaluation, 12-16 Dec 2011
39
Quality Assessment of IE Reports
This project is co-financed by the European
Union and the Republic of Turkey
• E. Impartial conclusions: does the report provide value judgements
based upon the five evaluation criteria of relevance, efficiency,
effectiveness, sustainability and impact?
• F. Useful recommendations: to what extent do the Recommendations
follow logically from the Conclusions? Are they operational? Do they
clearly address the monitoring sector and are they targeted to the
different stakeholders?
• G. The executive summary: to what extent is the executive summary
a synthesis and does it meet the requirements set out in the template
guidelines?
• H. Annexes: to what extent do the Annexes support the analysis in the
main text?
• I. Overall style, structure and text design: within the template’s
framework, to what extent is the text easily readable and accessible to
the various categories of readers so that the main messages are easily
detectable?
Monitoring & Evaluation, 12-16 Dec 2011
40
Quality Assessment of IE Reports
This project is co-financed by the European
Union and the Republic of Turkey
Verbal
Unacceptable Poor Sufficient Good Excellent
/
adequate
Single
-2
category
-1
Entire
report
-14
-5 to 5
to -6
<=-15
0
1
2
6 to
14
>=15
Taking into account the contextual constraints on the evaluation,
the overall quality rating of the report is considered to be:
Monitoring & Evaluation, 12-16 Dec 2011
41
Why Results Are Ignored?
This project is co-financed by the European
Union and the Republic of Turkey
• Implications may not be presented in a way that
nonresearchers can understand.
• Results sometimes contradict deeply held
beliefs.
• Vested interest in a program.
Monitoring & Evaluation, 12-16 Dec 2011
42
This project is co-financed by the European
Union and the Republic of Turkey
SECTION III.
MONITORING
• EU Regulations on monitoring
• Monitoring system and its evaluation
Monitoring & Evaluation, 12-16 Dec 2011
43
MONITORING
This project is co-financed by the European
Union and the Republic of Turkey
•
•
•
•
@ follow up , controlling
A continuing observation
uses systematic collection of relevant and selected data
to provide management and the main stakeholders of a
programme/project with indications
• of the extent of progress and achievement of
objectives
• of process and impact.
Monitoring & Evaluation, 12-16 Dec 2011
44
This project is co-financed by the European
Union and the Republic of Turkey
COM REGULATION 718/2007
implementing 1085/2006
Title II Common rules for implementation/Chapter V
Evaluation and monitoring
Article 58 Monitoring in the case of decentralised
management (IPA Monitoring Committee)
• BC in agreement with NIPAC and COM
• to ensure coherence and coordination in the
implementation of the IPA components
• meeting the objectives set out in the financing agreements
and MIIFP
• based on the elements given by the SMCs
Monitoring & Evaluation, 12-16 Dec 2011
45
This project is co-financed by the European
Union and the Republic of Turkey
COM REGULATION 718/2007
implementing 1085/2006
IPA Monitoring Committee – proposals
• to COM, NIPAC and NAO for:
• any actions to ensure the coherence and
• crosscomponent corrective measures needed to
ensure the achievement
• to SMCs for:
• decisions on any corrective measures to ensure
the achievements of programme
Monitoring & Evaluation, 12-16 Dec 2011
46
This project is co-financed by the European
Union and the Republic of Turkey
COM REGULATION 718/2007
implementing 1085/2006
IPA Monitoring Committee – organisation
• Internal rules of procedure
• in compliance with a monitoring committee mandate
established by the COM
• within the institutional, legal and financial framework of the
BC
• Members: Commission, NIPAC, NAO, OS, strategic
coordinator
• COM and NIPAC co-chair
• Meetings at least once a year
• Intermediate meetings on a thematic basis.
Monitoring & Evaluation, 12-16 Dec 2011
47
This project is co-financed by the European
Union and the Republic of Turkey
COM REGULATION 718/2007
implementing 1085/2006
Article 59 Sectoral monitoring committees in the case of
decentralised management
•
•
•
•
Assist IPA Monitoring Committee
Attached to programmes or components.
May include representatives of civil society,
Objective:
• effectiveness and quality of the implementation
• in accordance with the related sectoral and/or financing
agreements.
• Proposals
• to the COM and NIPAC, cc. NAO
• for decisions on any corrective measures to ensure the
achievements of programme objectives
Monitoring & Evaluation, 12-16 Dec 2011
48
This project is co-financed by the European
Union and the Republic of Turkey
COM REGULATION 718/2007
implementing 1085/2006
Sectoral monitoring committee - reports
• to IPA monitoring committee on
• progress
• by priority axis/ measures or operations;
• results achieved, financial implementation
indicators
• any aspects of the functioning of the management
and control systems raised by the audit authority,
NAO or the competent accrediting officer
Monitoring & Evaluation, 12-16 Dec 2011
49
This project is co-financed by the European
Union and the Republic of Turkey
COM REGULATION 718/2007
implementing 1085/2006
Article 61 Annual and final reports on
implementation
• Sectoral annual report and sectoral final report by
OS
• Sectoral annual report: financial year
• Sectoral final report: whole period of
implementation
• SMC examination  to be sent to NIPAC, NAO and
COM
Monitoring & Evaluation, 12-16 Dec 2011
50
This project is co-financed by the European
Union and the Republic of Turkey
COM REGULATION 718/2007
implementing 1085/2006
Annual and final reports on implementation
• NIPAC sends annual and final reports based on
synthesis of sectoral reports to COM and NAO
• by 31 August each year
• progress made in relation to the priorities of MIIFP
and OPs
• financial implementation of Community assistance.
Monitoring & Evaluation, 12-16 Dec 2011
51
This project is co-financed by the European
Union and the Republic of Turkey
COM REGULATION 718/2007
implementing 1085/2006
Title III Regional development and human resources development/Chapter
III Implementation/Section 3 Evaluation and monitoring
Article 167 Sectoral monitoring committee Organisation
• To be established by OS for each programme
• single SMC may be set up for several programmes
within the same component
• meeting at least twice a year, at the initiative of BC or
COM
Monitoring & Evaluation, 12-16 Dec 2011
52
This project is co-financed by the European
Union and the Republic of Turkey
COM REGULATION 718/2007
implementing 1085/2006
Sectoral monitoring committee – Organisation (cont’d)
• Internal rules of procedure
• in compliance with a monitoring committee mandate established
by the COM
• within the institutional, legal and financial framework of the BC
• In agreement with the OS and the IPA monitoring committee
• COM and Head of OS co-chair
• composition to be decided by OS, in agreement with
COM
• Members: COM, NIPAC, strategic coordinator for RD and HR
components, OS
• Civil society and socio-economic partners where appropriate
• European Investment Bank in advisory role where EIB makes a
contribution
Monitoring & Evaluation, 12-16 Dec 2011
53
This project is co-financed by the European
Union and the Republic of Turkey
COM REGULATION 718/2007
implementing 1085/2006
Sectoral monitoring committee – Tasks
• Consider/approve/revise criteria for selecting within six months
of the entry into force of the financing agreement
• review progress in OP targets based on OS reporting
• examine (by reference to the indicators)
• results of implementation, achievement of the targets priority axis
and measures
• interim evaluations
• sectoral annual and final reports
• relevant parts and recommendations of annual audit activity report
• any proposal to amend the financing agreement
• propose to OS any revision or examination of the programme
(incl. its financial management)
Monitoring & Evaluation, 12-16 Dec 2011
54
This project is co-financed by the European
Union and the Republic of Turkey
COM REGULATION 718/2007
implementing 1085/2006
Article 169 Sectoral annual and final reports on
implementation
• OS to COM and NIPAC after examination by SMC
• sectoral annual report by 30 June each year
• sectoral final report six months after the final date of eligibility of
expenditure
• COM informs NIPAC and OS on admissibility of SAR within 10
working days from the date of its receipt – formal
• COM informs NIPAC and OS on its opinion on admissible
• SAR report within two months from the date of receipt
• SFR within a maximum of five months from the date of receipt
Monitoring & Evaluation, 12-16 Dec 2011
55
This project is co-financed by the European
Union and the Republic of Turkey
COM REGULATION 718/2007
implementing 1085/2006
Sectoral annual and final reports - content
• quantitative and qualitative progress in
• OP, priority axes, measures, operations
• against quantified targets
• financial implementation of OP, priority axis, measure:
• total expenditure paid out by the final beneficiaries included
in NF’s payment applications
• total expenditure committed and paid out by NF
accompanied by computerised forms listing the operations,
• total payments received from the COM
• breakdown of allocation by categories as listed in FA
Monitoring & Evaluation, 12-16 Dec 2011
56
This project is co-financed by the European
Union and the Republic of Turkey
COM REGULATION 718/2007
implementing 1085/2006
Sectoral annual and final reports – content (cont’d)
• steps taken by OS or SMC to ensure quality and
effectiveness
• monitoring and evaluation measures, incl. data
collection arrangements;
• summary of problems and subsequent measures;
• use of technical assistance.
• information and publicity activities;
• progress and financing of major projects.
Monitoring & Evaluation, 12-16 Dec 2011
57
Monitoring concepts
This project is co-financed by the European
Union and the Republic of Turkey
Activity and/or result oriented monitoring?
• Activity based monitoring
• identify activities/deliverables and deadlines for each
• activities to be completed in a timely and appropriate
manner
• Result based monitoring
• Results to be achieved
• Analogy to project management and monitoring
• Activities must be completed – legal and financial
consequences
• Results must be achieved – effectiveness aspect
Monitoring & Evaluation, 12-16 Dec 2011
58
ROM and Internal monitoring in EU
This project is co-financed by the European
Union and the Republic of Turkey
ROM
Internal Monitoring
Objectives
To provide independent assessment of
project performance, with focus on
‘results’
To provide advice and
recommendations to project
stakeholders
To generate aggregate data for
reporting
To support effective and
timely decision making
by project managers
To promote
accountability for
resource use and
achievement of results
Responsibl
e
COM
Project implementing
partners/contractors
Monitoring & Evaluation, 12-16 Dec 2011
59
ROM and Internal monitoring in EU
This project is co-financed by the European
Union and the Republic of Turkey
Method
ROM
Internal Monitoring
Short-visits to project sites by
independent experts, on a periodic
basis
Analysis of project records and
interviews with stakeholders
Standardised assessment
Project plans, ongoing
data collection, analysis
of data, progress reports
Consultation with
stakeholders
Participation in review
meetings
Monitoring & Evaluation, 12-16 Dec 2011
60
Key principles for information flow
This project is co-financed by the European
Union and the Republic of Turkey
• Results information needs at the project, program, and policy
levels  results information must move both horizontally and
vertically
• Horizontal: what information is being collected by their own
organization and by other
• Vertical: bureaucratical and hierarchical burdens
 need to coordinate and collaborate in sharing performance
information
• Responsibility at each level needs to be clear for
•
•
•
•
•
•
What data are collected (source)
When data are collected (frequency)
How data are collected (methodology)
Who collects data
Who reports data
For whom data are collected
Monitoring & Evaluation, 12-16 Dec 2011
61
Needs of Monitoring System
This project is co-financed by the European
Union and the Republic of Turkey
Ownership
• Demand part : who needs the information?
• Needs to be identified
• Critical issue: when levels of need and data collecting
/maintenance do not fit
• Ownership vs enforcement
• Legal
• Financial
• Technical
Monitoring & Evaluation, 12-16 Dec 2011
62
Needs of Monitoring System
This project is co-financed by the European
Union and the Republic of Turkey
Management
•
•
•
•
Who, how, and where the system will be managed
Data collection overlaps, duplication of data
Time lags in receiving data
Data maintenance
• Responsible levels
• Continuous development and trainings
Monitoring & Evaluation, 12-16 Dec 2011
63
Needs of Monitoring System
This project is co-financed by the European
Union and the Republic of Turkey
Credibility
• Report all data—both good and bad
• Deliberate non-reporting of information demonstrating failure
• Political pressure
• Reliability: the extent to which the data collection approach is
stable and consistent across time and space - measurement
of the indicators is conducted the same way every time
• Validity: indicators should measure actual and intended
performance levels
• Timeliness
• Frequency (how often are data collected?)
• Currency (how recently have data been collected?)
• Relevance (are data available frequently enough to support
management decisions?)
Monitoring & Evaluation, 12-16 Dec 2011
64
Design of a monitoring system
This project is co-financed by the European
Union and the Republic of Turkey
1. Analyse project objectives to clarify project
design
2. Review implementation procedures to
determine information needs at the different levels
of the project management structure
3. Review indicators
4. Design reporting schemes
5. Prepare an implementation plan for the
monitoring system
Monitoring & Evaluation, 12-16 Dec 2011
65
Analyse project objectives
This project is co-financed by the European
Union and the Republic of Turkey
• Time factor since original design: chaning project
environment and actors
• Logframe approach
• analysis of existing situation in light of future desired
situation
• planning operational details
• Logframe matrix
• A possible tool: project start-up workshop
• all stakeholders
• review project documents and key assumptions
• revise project objectives
Monitoring & Evaluation, 12-16 Dec 2011
66
Review implementation procedures
This project is co-financed by the European
Union and the Republic of Turkey
• Level of detail of information need and frequency of reporting vary
according to the level of management
• Define information needs to the different levels of the management
structure
• From project administrators to IPAMC
• Workflow to be defined – procedures manual
• Problem: users don’t know in advance what they need  request
more information than needed  users are not aware of what
information is available
• Continual review of users requirements through:
• planning and review meetings: what is lacking or redundant
• systematic dialogue with users on content and format
Monitoring & Evaluation, 12-16 Dec 2011
67
Review indicators
This project is co-financed by the European
Union and the Republic of Turkey
To avoid:
• selection of too many indicators
• selection of overly-complex indicators
• over-concentration on progress indicators
Monitoring & Evaluation, 12-16 Dec 2011
68
Design reporting schemes
This project is co-financed by the European
Union and the Republic of Turkey
• Reports
• Based on project administrators’ needs on daily/weekly/monthly
basis
• Based on the needs of the hierarchy
• Incidental, Interim, Final as per contract
• Meetings
• ‘monthly’, ‘Project Review/Implementation Meetings’
•
•
•
EUD/NAC office vs CA, OBs
Informal, but methodology shall be determined in advance (all
the projects, only problematic ones)
Who represents the organisation?
• Steering Committees
Monitoring & Evaluation, 12-16 Dec 2011
69
This project is co-financed by the European
Union and the Republic of Turkey
SECTION IV.
MANAGEMENT INFORMATION
SYSTEM
• Main concepts and hints
• An example: the Hungarian MIS
Monitoring & Evaluation, 12-16 Dec 2011
70
This project is co-financed by the European
Union and the Republic of Turkey
MIS
• All projects (service, supply, works, grant
schemes): Similar characteristics
• Follow the same rules: PRAG rules and local
• Efficient use of limited resources: No need to
rewrite again and again
• Harmonization and Standardization of Monitoring:
Common to all
Monitoring & Evaluation, 12-16 Dec 2011
71
MIS
This project is co-financed by the European
Union and the Republic of Turkey
Objectives
•
•
•
•
Projects achieved their objectives
Project implementations follow the rules
Project progressing as planned and performing well
Problems detected earlier
Functions
• Follow-up of implementation
• Management control tool
• Support to beneficiaries
• Immediate data availability for the hierarchy and
public
Monitoring & Evaluation, 12-16 Dec 2011
72
MIS
This project is co-financed by the European
Union and the Republic of Turkey
•
•
•
•
•
•
Common communication platform and channel
Enforces obeying the rules
Provides monitoring system integrity
Creates standards for reporting and data entry
Verifies the declarations; Reports and Visits
Improve the quality of monitoring activities
Monitoring & Evaluation, 12-16 Dec 2011
73
MIS
This project is co-financed by the European
Union and the Republic of Turkey
Progress
• Ensuring project to produce expected outputs:
• with planned activities
• within the contract budget and duration
• Monitoring the Progress on:
• Financial Progress
• Technical Progress
• Risk Assessment
• Performance indicators
Monitoring & Evaluation, 12-16 Dec 2011
74
MIS
This project is co-financed by the European
Union and the Republic of Turkey
Characteristics of the system
• Built with Participatory Approach
• Comprehensive and decentralized
• Open to all actors
• Easily Accessible (Web-based)
• Transparent System (Any data cannot be deleted or
changed without permission)
• Up-to-date data collection from the source
• Different user levels according to responsibilities
including project owners
Monitoring & Evaluation, 12-16 Dec 2011
75
MIS
This project is co-financed by the European
Union and the Republic of Turkey
Some points of consideration
• Capacity issues
• Participatory approach during development
and testing
• Database maintenance
• Need to assign staff specific to MIS (not IT)
• Source code issue
• For further upgrades
• Public procurement consequences
Monitoring & Evaluation, 12-16 Dec 2011
76
This project is co-financed by the European
Union and the Republic of Turkey
EMIR – the Hungarian model
www.emir.gov.hu
Monitoring & Evaluation, 12-16 Dec 2011
77