Transcript www.fitzpatrick-associates.com

International Conference
Evaluation: Evidence-based Tools for Decision-making
Future Cohesion Policy: Implications for Monitoring and Evaluation
Budapest May 24-25, 2011
Dr Jim Fitzpatrick
Managing Director
Fitzpatrick Associates
122 Ranelagh Village
Dublin 6
Tel: +353 1 6280084
Fax: + 353 1 6219771
1
TOPICS
 issues in future Cohesion Policy affecting Monitoring and
Evaluation (M+E)
 emerging parameters of 2014-20 period
 the “new” Logical Framework
 challenges in implementing new approaches
 some practical suggestions
2
EU STRUCTURAL AND COHESION FUNDING
 20 years + of multi-annual programmes
 M+E central to the programming package

Objectives, priorities, targets/indicators

Agreed strategies, programmes

Regular monitoring

Formal evaluation
 Gradual development of M+E practice
 2014-20 will involve significant changes
CEE
entry
1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020
3
2014-20: INCREASING THE EFFECTIVENESS OF COHESION
POLICY
 the main focus of evaluations “tended to be on processes and financial
implementation rather than on the actual results of programmes”, (Ex Post Evaluation
of Cohesion Policy Programme 2000-06, Synthesis Report April 2010)
 a Member State-Commission National Strategic contract where Member States
(or Regions) “commit themselves to quantified and verifiable objectives”. (An
Agenda for a Reformed Cohesion Policy , Barca Report, April 2009)
 “specific binding conditionality in the areas directly linked to cohesion policy
would be agreed with each Member State and/or Region – depending on the
institutional context…” (Investing In Europe’s Future: Fifth Report on Economic, Social and
Territorial Cohesion, Nov. 2010)
Work of High Level Group on Future Cohesion Policy, Conditionality Task Force
(Feb.-April 2011), Evaluation Unit/Network, Hungarian Presidency High Level
Conference (March 31/April 1).
4
A CONFLUENCE OF RELATED STRANDS
Thematic
Concentration
Performance/
Results
measuring
outcomes
Binding
Contracts
Monitoring
And
Evaluation
Better
Evaluation/
CFA
Conditionality
Better
Information
5
EMERGING NEW ARCHITECTURE 2014-20
provides goals
Europe 2020 Goals
Common Strategic Framework (CSF) ?*
provides themes
regulatory requirements
Regulations
Partnership and Investment Contracts*
evaluation will inform content/progress
Operational Programmes
ex ante, ongoing, ex post
Themes/Axes (from menu)
* Possibly for Cohesion Fund, ERDF, ESF, EAFRD, EEF
6
STRENGTHENED CONDITIONALITY?
 already is various conditionality: regulatory (inc. M+E); strategic. Infrastructure
planning; institutional
 types of conditionality:
 “macroeconomic” – part of Stability/Growth Pact
 “ex ante” – preconditions (see existing categories)
 “structural reform” – structural/admin reform milestones
 “performance” – Programme/EC objectives
 incorporated in Partnership Contracts?
 possible financial incentives/sanctions?
 if implemented, could have implications for Monitoring/Evaluation
7
NEW LOGICAL FRAMEWORK
Programming
Monitoring
and Evaluation
Strategy
Needs
Thematic
Objective
Policy
Intended
Result
Actual
Result
Allocated
INPUTS
Actual
INPUTS
Targeted
OUTPUTS
Achieved
OUTPUTS
Other
Factors
Contribution - Impact

outcomes monitored/impacts evaluated

inputs/outputs “contribute” to outcomes

outcome indicators should be “responsive” to intervention
Source: V. Gaffey, Acting Director, Policy Development, DG Regio, Intervention Evaluation Conference, Budapest, May 24-25, 2011.
8
RE-DEFINING THE LOGIC
Monitoring
Old:
Inputs
Evaluation
Outputs
Results
(Short/medium)
IMPACTS
(Long term)
Monitoring
New:
Inputs
Results/
outcomes
Outputs
Evaluation
Impact
Source: based on DG Regio, Concepts and Ideas: Monitoring and Evaluation in Practice of European Cohesion
Policy 2014+, Draft, 30 March 2011
9
CRITERIA FOR A GOOD OUTCOME INDICATOR
Reasonable: capturing the essence of an outcome according to a reasonable argument about
which features of the outcome they can and cannot represent
Robust: reliable, statistically and analytically validated, and, as far as practicable, complying with
internationally recognised standards and methodologies;
Responsive to policy: linked in as direct way as possible to the policy interventions for whose
assessment they are used, while not being subject to manipulation;
Normative: having a clear and accepted normative interpretation (i.e. there must be agreement
that a movement in a particular direction or within a certain range is a favourable or an
unfavourable result);
Feasible: built, as far as practicable, on available underlying data, their measurement not
imposing too large a burden on Member States, on enterprises, nor on the citizens;
Debatable: timely available to a wide public, with room being built for public debate and for their
own revision when needed and motivated.
Source: F. Barca, P. McCann, Outcome Indicators and Targets – Towards a Performance Oriented EC Cohesion Policy,
High-level Group on Future Cohesion Policy, Meeting No. 8, 15 Feb. 2011
RRR-NFD instead of SMART! The challenge is in the “Third R”
10
THE RESPONSIVENESS ISSUE: SUB-CRITERIA FOR OUTCOME
INDICATORS?
 needs to be specifically related to the output?
 easier if OP is the only source of funding (national + EU)?
 needs to be about the beneficiaries?
 easier if there is a clear physical link, e.g. infrastructure
 needs a typology of intervention types*
*i.e. infrastructure, subsidies, service provision.
Use Pilots to develop this
11
CHALLENGES REGARDING OUTCOME/RESULT INDICATORS
Conceptual:
 the causation problem remains?
 a lot of judgement?
Communication:
 can we be confident this will address the issue?
 trade-off between communication and robustness
Capacity:
 who is going to define the indicators? A specialist task. Will need forensic
precision
Time-lags:
 when will attributable outcomes arise?
Conditionality:
 M+E insufficiently robust to support binding conditionality
12
SOME SUGGESTIONS FOR COMMISSION/MEMBER STATES
 skills, capacity, technical assistance, technical support
 M+E as communication tools? Simplify language? Consistent terminology, e.g.
“outcomes” v “results”
 revisit institutional context, inc. Monitoring Committees, Managing Authority
 Nature of Commission Guidelines and technical support
 use of regular Peer Review Groups, (“boots on the ground”) ongoing Evaluators
 merge Monitoring and Evaluation function in Member States
 greater role for Eurostat/National Statistical Offices (but avoid pure context
indicators)
 nature of Multi-annual Evaluation plans
13
And finally…
 keep the Regulations general and Guidelines detailed, specific
(not vice versa)
 don’t rush the Guidelines
THANK YOU.
14