Transcript Evaluation of EU Structural Interventions

Evaluation of EU
Structural Interventions
Ex-ante Evaluation
Mid-term / Updated Mid-term Evaluation
Ex-post Evaluation
Project Assessments
P L A N U N G & F O R S C H U N G
POLI CY RESEARCH & CO NSU LT AN CY
Königsteiner Strasse 116
D-65812 Bad Soden
Germany
Phone: +49-(0)6196-654168
Fax: +49-(0)6196-654178
Internet: www.prac.de
Purpose
Explanation on the Purpose
The quality and adequacy of projects and fanding activities go hand
in hand with the quality and adequacy of the programme in which
the project is part of. A regional or sectoral development programme
(as e.g. the EU Structural Funds) aims at fostering inter-regional
convergence and/or stabilising important regional growth poles for
the national economy. The purpose is not to simply subsidise
institutions, business and individuals, Funding is the means and not
the end.
Programme evaluation has to consider the respective quality,
adequacy and implementation of a programme. Therefore the criteria
applied in programme evaluation need also be translated into the
quality assessment of the projects materialising the programme.
Hence, as a truism, a project can only be as good as the programme
and the programme can only be as good as the projects.
1. Ex-ante Evaluation
Structural Funds interventions are implemented in accordance to
Operational Programmes (or Single Programming Documents)
which must be (1) based on a reliable qualitative and quantitative
analysis of the socio-economic situation of the region including its
SWOTs and previous experiences, (2) the determination of a targetoriented development strategy, (3) the choice of appropriate
measures to materialise the strategy, (4) suitable projects and (5)
sound provisions governing financing, responsibilities, evaluation
etc..
The assessment of the consistency of the above listed contents of
an OP (or SPD respectively) is the core rationale of the Ex-ante
evaluation. Further tasks to be carried out in the context of the Exante Evaluation are (7) the quantification of the objectives and (8)
the estimation of the overall socio-economic impact.
Objectives of the Ex-ante
Evaluation
• Assessment of the appropriateness of the
plan in addressing the regional development
problems;
• Assessment of the consistency of analysis
and concept;
• Definition of suitable indicators and
quantification of objectives;
Objectives ...
• Analysis of the adequacy of the
implementation and monitoring
arrangements; and
• Impact assessment
Assessment of Appropriateness
• Lessons learned from
past experiences;
• Difficulties and
shortcomings
experienced and to be
tackled;
• Assessment of the
validity of data used in
previous programmes;
• Consideration of
thematical evaluations;
• Consideration of
experiences in other
programmes; and
• Verification of the
socio-economic
analysis and the
consistency of SWOTs
Consistency of SWOTs
• Strengths are static features to be supported
• Weaknesses are static features to be reduced
• Opportunities are dynamic features to be reaped
• Threats are dynamic features to be forestalled
Consistency of Analysis and
Strategic Concept
• Are the strategic objectives well specified ?
• Do the strategic objectives sufficiently respond to the
needs identified in the socio-economic analysis?
• Are the proposed measures justified by the strategy?
• Is the strategy sufficiently coordinated with other
related supra-national or national
policies/programmes (nat. regional policy, NAP, CIs,
Objective 3 etc.) ?
Suitable Indicators (examples)
• Basic indicators:
• Programme indicators:
• Measure related
indicators:
• Performance
indicators:
• Gross regional value
added
• Supported investment
• No. of supported
SMEs
• Disbursement/
allocated funds
Quantification of Objectives
• Related to INPUTS-OUTPUTS-RESULTSIMPACTS
• Determination of a realistic quantification of
indicators
• Comparison with results achieved in former
programme periods
• Proxy-indicators to monitor intangible
effects
Appropriateness of the
Implementation System
• Management and implementation responsibilities;
• Transparency;
• Efficiency and effectiveness of inter-institutional coordination;
• Competitiveness in project choice;
• Accountability in line with national and community
regulations (competition policy, additionality, horizontal
priorities, innovation, NAP etc.); and
• Partnership in programming and monitoring.
Impact assessment
• Micro-economic net effects (notably
employment, productivity, regional GDP);
• Macro-economic net effects for large
programmes (macro-econometric
estimations and simulations on short-term
demand, long-term supply, labour market
and government sector)
2. Mid-term Evaluation and its
Update
Mid-term Evaluation
•
•
•
•
Update of the analysis of relevance;
Update of the analysis of consistency of priorities;
Analysis of the validity of the objectives;
Comparison of plan and achievements (vertical
and horizontal);
• Assessment of the implementation performance in
terms of financial and administrative management;
• Contribution to the decision on the allocation of
the Performance Reserve;
Mid-term Evaluation ...
• The responsibility of the mid-term evaluation is
with the member state;
• The evaluation is to be carried out by an
independent assessor;
• An update of the mid-term evaluation is to be
elaborated one year before termination of the
CSF/SPD; and
• The evaluations are accompanied and approved by
the monitoring committees.
Analysis at Programme Level
• Were recommendations from former evaluations
sufficiently considered?
• Have the socio-economic problems (justifying the
intervention) changed?
• Is the SWOT analysis adequate?
• Is the Programme still consistent and coherent?
• Has the programme been implemented
professionally?
Analysis at Priority/Measure Level
• Financial implementation analysis
• Effectiveness analysis (plan vs.
achievement)
• Efficiency analysis (unit cost comparisons)
• Impact analysis (GDP, net employment)
Cf. EU Commission/DG Regio: Working Paper 8
Update Mid-term Evaluation
The role of the updated mid-term evaluation
is to confirm the implementation of
recommendations put forward in the midterm evaluation and to cover fields of
analysis which could not be evaluated
earlier.
Cf. EU Commission/DG Regio: Working Paper 9
3. Ex-post Evaluation
The Ex-post evaluation is carried out under the responsibility of
the EU Commission. It is elaborated as a global evaluation for
each of the regional(Objective 1 and 2) and sectoral (Objective 3)
programmes covering all targeted member states.
The main purpose is to assess the quality, effectiveness and
impacts of the programmes. It should clearly reveal the strengths
and weaknesses of the planning process, the strategic approach
and implementation. Lessons learned from this exercise should
contribute to better policy formulation in current and future
programming periods.
Scope of the Ex-post Evaluation
• Analysis of the relevance of strategy and the
expenditure at EU, national and regional levels;
• Effectiveness analysis;
• Efficiency and impact assessment;
• Assessment of the delivery system;
• Analysis of the Community value added; and
• Lessons learned for the future and policy
recommendations.
Relevance of Strategy and
Expenditure
• Are the strategies adopted by the
regions sufficiently relevant?
• Are the strategies appropriately
implemented in practice?
• Has there been any deviation of
the interpretation of the strategy
(new direction)?
• Is the philosophy of the strategy
still in line with that of the prior
evaluations?
• Were the funds accordingly
allocated (dispersion vs.
Concentration?
• Were the changes of the
strategy relevant or arbitrary?
• Was the aid intensity relevant?
• The degree of coherence in the
approaches across all
CSFs/SPDs of Objective 1 or
Objective 2 respectively?
Effectiveness Analysis
• Comparison of plan and realisation in terms
of policy priorities (e.g. private sector
support, innovation, business related
infrastructure, HRD etc.);
• Assessment of the formerly expected and
realised performance of the policy
instruments (e.g. financial engineering)
Efficiency and impacts
• Value for money assessment;
• Assessment of the net-effects (employment,
productivity, regional GDP etc.); and
• Macro-econometric model estimations for
major programmes (notably Objective 1)
Delivery System
• Assessment of the institutional management and
implementation capacity;
• Functioning of partnership;
• Sufficiency of transparency;
• Functioning of the monitoring system;
• Functioning of the project selection procedures;
• Appropriateness of the practical provisions to
ensure the legal underpinnings; and
• Functioning of the quality control mechanisms
Community Value Added
• Economic and social convergence;
• Internal and external Cohesion; and
• Institutional improvements (e.g. better
coordination)
`Lessons Learned` and Policy
Recommendations
• Conceptual level and supra-national regulations
(EU regional policy formulation);
• Justification of the financial contributions
and/or the level of additionality respectively;
and
• Implementation level (management, M&E
partnership etc.)
THE EVALUATION CYCLE
Ex-ante
Evaluation
Programming
Mid-term
Evaluation
Implementation
Mid-term
Update
Adjustments
Ex-post
Evaluation
Programme
results
Project assessments
Project Assessment within Structural
Funds Programmes
• Rules and funding regulations laid down in the
Programme Complement
• Supra-national, national and regional funding
regulations
• Assessment of the scope and aim of the
application
• Control/audit
• Funding approval … Proof of fund utilisation
Criteria listed the Programme
Complement
The PC includes the specific funding regulations and criteria for all
measures of the programme. Any institution/individual applying for
funding needs to comply to the provisions made in the PC. The PC
defines
•The valid funding provisions
•The thematical/sectoral scope of funding
•The pre-conditions of funding (e.g. specific target groups)
•Eligible expenditure
•Ceiling of the subsidy
•Period of validity
•Geographical coverage
Funding regulations
• Regulations of the EC (e.g. Council
Regulation 1260/1999)
• European competition law (e.g. Article 92)
• National competition law
• Regional funding provisions
Note: all regulations need to be co-ordinated
Project assessment grid (I)
• Is the project thematically suitable for
funding under the respective measure?
• Does the project contribute to the quantified
objectives of the measure?
• Is the project idea sufficiently original?
• Does the project suggest synergy effects for
the programme?
• Is the required co-financing adequate?
Project assessment grid (II)
• Is the applicant (or the group of applicants)
part of the target group?
• Is the applicant (or the group of applicants)
trustworthy
• …
Controlling/auditing
• Internal controlling of the managing
authority notifying fraud and irregularities
• Monitoring of the project progress by the
Managing Authority
• Controlling and issuing of expenditure notes
by the Paying Authority
• 5% random audits carried out by the
Independent Auditing Unit
Author: Rolf Bergs,
4 November 2004