Transcript Document

NATIONAL
EVALUATION
STRATEGY
MINISTRY OF PUBLIC FINANCE
MANAGING AUTHORITY FOR
COMMUNITY SUPPORT FRAMEWORK
Parliament Building, Bucharest, 2 November 2006
Why we need to evaluate?
It provides answers to worthwhile questions
 Evaluation can reduce uncertainty and improve planning
and implementation
BUT
if evaluation is to be useful and usable, it needs to
be seen as an integral part of decision making
and management and indeed the entire process
of democratic accountability.

The role of evaluation






To contribute to the design of interventions,
including providing input for setting political
priorities
To assist in an efficient allocation of resources
To improve the quality of intervention
To report on the achievements of the
intervention
To provide lessons for future intervention
To provide answerability
What is (not) the evaluation?

Evaluation is a judgement of interventions according to
the results, impacts and the needs they aim to satisfy.

Evaluation is not monitoring, audit or control. The
evaluation’s results do not lead to repercussions on
intervention’s managers or designers, but lead to
changes in its design, financial allocations or in the way
of implementation. It is a policy and management tool,
not a tool of control.
Why we need
a National Evaluation Strategy?




The purpose of evaluation is not well
understood
Confusion exists about the concept of
evaluation
Real demand for evaluation has come from
outside the Romanian administration in the
form of foreign donors
The institution of government is poor when it
comes to using evaluation as a management tool
Why we need
a National Evaluation Strategy?



There are already a significant number of
initiatives to improve policy making and the overall
quality of managing, implementing and evaluation
of publicly-funded programmes.
If brought closer together, synergy could be
obtained and thus the impact of the positive trends
could be much more significant.
The National Evaluation Strategy (NES) is an
important vehicle that could bring together all of
the disparate strands currently trying to drive
evaluation and will make the case stronger if
stakeholders are unified and in agreement.
Steps involved in drafting
the National Evaluation Strategy?

EC recommendation to extend to EU funded
programmes evaluation practice to national funded
interventions

The Single Action Plan

Review of the Romanian legal framework

Assessment of the evaluation culture in Romania
The Single Action Plan

Establishing the National Evaluation System
among 8 specific objectives






Elaboration of a National Evaluation System and
subsequent action plans
Implementation of the Action Plan for taking over the
PHARE interim evaluation
Implementation of the Action Plan for establishing the
National Evaluation System
Implementation of the Action Plan for raising
awareness on evaluation
Implementation of the Action Plan for development of
local evaluation capacity (within private market)
Approved by the Government in 2005
Review
the Romanian legal framework


Completed in February 2006
Findings


The Law on Public Finances no. 500/2002 does not include explicit
requirements to conduct evaluation exercises as part of the
financial management process. It is stipulated, however, that the
institutions managing public funds have the obligation to prepare
“annual performance reports” that set out, for each programme, the
objectives, the foreseen and the actual results obtained, indicators
and associated costs. These reports are to be attached to the
annual financial reports.
The Government Emergency Ordinance 45/2003 for the
management of local public funds does not include an explicit
requirement to conduct evaluations. When defining the term
"programme", however, it states that "indicators should be
established to allow the evaluation of the results obtained..", but
there is no explicit requirement to carry out an evaluation of a
programme or of other activities.
Review
the Romanian legal framework

Findings


The Law for Internal Audit 672/2002, regulating the
audit activities for institutions that use public funds
defines internal auditing as "an independent and
objective activity…that helps the public entity to fulfil
their objectives through a systematic and methodical
approach, that evaluates and improves efficiency and
effectiveness of the management system based on risk
management, control and administrative processes."
The Regulatory Manual of Parliament operations do not
explicitly include the right to perform evaluations of any
type. It does, however, foresee the right to initiate
parliamentary inquiries under certain conditions.
Review
the Romanian legal framework

Findings

The report concludes that by comparison with EU
evaluation regulations, Romanian legislation has
specific provisions that explicitly require evaluation only
for the co-financing budgets for EU and other donor
funding and for research programmes. For the other
types of activities, even though mention is made of it,
requirements to conduct evaluations are either not clear
enough or not enforced properly.
Review
the Romanian legal framework

Recommendations


Romanian legislation related to evaluation needs to be
upgraded in line with EU legislation and should address
all of the areas and institutions that are managing
public funds, as a management tool for public
administration. New legislation should be supported by
methodological guidelines that explain how to apply the
law.
Moving from the current annual budgeting cycle to a
multi-annual budget programming cycle would nurture
the development of the practice of evaluation.
Furthermore, evaluation would be easier to conduct if
budgets were allocated on the basis of identified
programmes and projects, as opposed to an allocation
by type of expenditure only.
Review
the Romanian legal framework

Recommendations

The institutional capacity of public administration to
manage programmes and projects should be improved

There should be a clear distinction between monitoring,
audit, control and evaluation.
Assessment of the evaluation culture

Conclusions


Evaluation has been a constantly developing practice for
donor-funded programmes, but is still under-developed
for the activities funded by the Romanian national
budget. However, there is clear evidence of growing
interest and evidence of actions taken at various levels
to improve the management of public policies and
programmes, including evaluation.
The legal base for evaluation is gradually improving. It
should be noted that equally important to enacting new
legislation is the enforcement process, because past
experience in Romania suggests that even when certain
legal provisions existed, they were either not well
understood or simply not enforced.
Assessment of the evaluation culture

Conclusions

Three main areas of further action can be identified:




Further development of the legal framework;
Preparing an adequate know-how base: methodologies, guidelines,
publications.
Increasing the capacity of the public managers to manage the
evaluation process
On the first two areas there are already a significant number of
initiatives to improve policy making and the overall quality of
managing, implementing and evaluation of publicly-funded
programmes. If brought closer together, synergy could be obtained
and thus the impact of the positive trends could be much more
significant. It is encouraging to notice that already this process of
co-operation between the various initiatives has started and it is
recommended that this is maintained and increased.
Assessment of the evaluation culture

Conclusions


The National Evaluation Strategy (NES) is an important vehicle
that could bring together all of the disparate strands currently
trying to drive evaluation and will make the case stronger if
stakeholders are unified and in agreement. It is to Romania’s credit
that it is taking this step and, given the current efforts, there is a
great opportunity now to be taken in developing an evaluation
culture which will underpin more effective governance within the
country.
Co-operation between the structures in charge of managing
evaluation for the NSRF process and the structures that will
manage the evaluation of nationally-funded activities is important
for at least two reasons: a) the experience of evaluating the NSRF
OPs will be a very valuable source of lessons; b) both structures
should aim to establish a unified approach to evaluation, applied for
European funds as well as for the national funds. To this end, it is
important to clarify how evaluation units in MA’s and line Ministry
PPU’s will work together.
Assessment of the evaluation culture

Conclusions


In terms of managing the evaluation process, it is the view of the
authors that evaluation is an integral part of management and as
such it should be embedded in the everyday practice of public
managers. It is the public managers themselves who must either
carry out internal self-evaluations or commission them to external
contractors. This is the sole way to promote evaluation as a
managerial tool.
The other use of evaluation is that of control, for accountability
reasons. This requires the commissioning of evaluation exercises
by someone other than the entities under evaluation and justifies
the existence of independent entities in charge of managing
evaluation processes, such as the Court of Accounts or independent
units created within the public administration apparatus.
Assessment of the evaluation culture

Conclusions

Moving from a legalistic framework to a performance
management framework is a necessary step if
evaluations are to move away from mere reporting and
become a tool for improvement. Experience suggests
that such a process may take up a considerable amount
of time and it is recommended that the managers of the
National Evaluation Strategy implementation bear this
in mind and are prepared for a long-term sustained
effort.
Assessment of the evaluation culture

Recommendations
 The National Evaluation Strategy should avoid overambitious objectives, particularly in the short to
medium term. Experience in other EU Member States
and as documented in the Evalsed Guide has shown that
a slow and steady burn has proved most effective in
building an evaluation culture in other Member States.
 Create stronger links and maintain regular
communication between the main actors who are
promoting the new practices of evaluation
 If possible, agree that one of these actors takes a lead
role in the implementation of the NES and the overall
promotion of the practice of evaluation in the country,
while each actor can retain leadership in their respective
areas.
Assessment of the evaluation culture

Recommendations



Establish first, between these actors, a common
understanding of the concepts related to evaluation, of
its scope for application and of its vocabulary. The
constant interchanging of terms and titles is too
confusing and if people at the top are confused, there is
no doubt this confusion will translate through the
system.
Attempt to harmonise the methodologies and manuals
that are currently being prepared by the various actors.
Once the NES has been agreed and approved, divide
the tasks such that each of the actors has clear
responsibilities for a certain set of actions.
Assessment of the evaluation culture

Recommendations
 Each actor should conduct a few pilot evaluations in
their respective areas and then should cross – reference
the results between the actors. This could involve
institutions which are most open to the idea so that a
model of good practice is established at an early stage,
that people can see how it is done and how it is used to
dispel any fears they might have. This will assist in
trying to move from a culture of control and reporting to
a culture of learning and analysing. The pilot
evaluations should be followed by a number of
subsequent exercise aiming to improve the
methodologies applied until a satisfactory process is
obtained, that can be transposed into a model and a
methodology manual.
Assessment of the evaluation culture

Recommendations

It is important to ensure that all Ministries have some
degree of ownership of the National Evaluation
Strategy, and this means intensive consultation and
involvement of heads of departments while preparing
and rolling it out.

At the same time as demand is being stimulated,
measures to increase supply and capacity should be put
in place, concentrating on inside and outside the public
administration system.
Principles of
the National Evaluation Strategy

Co-operation

close working between the structures in charge of
managing evaluation of the ongoing, pre-accession
instruments, the NSRF process and the structures that
will manage the evaluation of nationally-funded
activities is important for two reasons: a) the experience
of evaluating the NSRF OPs will be a very valuable
source of lessons; b) both structures should aim to
establish a unified approach to evaluation, applied for
European funds as well as for the national funds. To this
end, the evaluation units in MA’s and line Ministry
PPU’s should participate in the evaluation networking
activities.
Principles of
the National Evaluation Strategy

Co-operation

Cooperation between the Ministry of Public Finance
MACSF ECU and the Ministry of Administration and
Interior’s MA for the Administrative Capacity
Development OP is vital, as implementation of the
activity planned under the evaluation Key Area of
Intervention in the Administrative Capacity
Development OP will form a central part of
implementation of the Strategy.
Principles of
the National Evaluation Strategy

Ownership

this Strategy can only be implemented if there is clear
ownership of and commitment to the Strategy. Again
this will necessitate ongoing cooperation between the
MACSF in the Ministry of Public Finance and the MA
for the Administrative Capacity Development OP. The
Strategy has a greater chance of being implemented if
other line Ministries and the wider public administration
also feel that they stand to benefit from its
implementation. It is important to ensure that all
Ministries and local authorities have some degree of
ownership of the National Evaluation Strategy, and this
means intensive consultation and involvement of heads
of departments while preparing and rolling it out.
Principles of
the National Evaluation Strategy

Independence


Evaluations should be managed by someone other than the entities
under evaluation and this explains the existence of independent
entities in charge of managing evaluation processes, such as the
independent units created within the NSRF OP Managing
Authorities and the Public Policy Units.
Long-term vision

Moving to a performance management culture involves
understanding evaluations as a move away from mere reporting
towards becoming a tool for improvement. Experience suggests
that this may take up a considerable amount of time and it is
recommended that the managers of the National Evaluation
Strategy implementation bear this in mind and are prepared for a
long-term sustained effort.
Principles of
the National Evaluation Strategy

Independence


Evaluations should be managed by someone other than the entities
under evaluation and this explains the existence of independent
entities in charge of managing evaluation processes, such as the
independent units created within the NSRF OP Managing
Authorities and the Public Policy Units.
Long-term vision

Moving to a performance management culture involves
understanding evaluations as a move away from mere reporting
towards becoming a tool for improvement. Experience suggests
that this may take up a considerable amount of time and it is
recommended that the managers of the National Evaluation
Strategy implementation bear this in mind and are prepared for a
long-term sustained effort.
Principles of
the National Evaluation Strategy

Developmental

It is important to move in a planned and staged way from a culture of
control to a culture of learning and analysis. Institutions that are most
open to this idea could conduct some pilot evaluations (under the proposed
“evidence-based management” field of intervention), so that a model of
good practice is established at an early stage. People could thus see how
evaluation as a means of learning is done and which may dispel any
reservations they might have. Pilot evaluations could be followed by a
number of subsequent exercises aimed at improving the methodologies
applied until a satisfactory process is obtained, that can be transposed into
a model and a methodology manual.

Evaluation is an integral part of management and as such it should be
embedded in the everyday practice of public managers. It is the public
managers themselves who must either commission evaluations from
external contractors, or, when sufficient internal expertise exists, may even
initiate self evaluations. Taking a direct interest in the design and use of
evaluation is essential in order to promote evaluation as a managerial tool
Principles of
the National Evaluation Strategy

Transparency

this Strategy has been drafted following a lengthy consultation
process with key stakeholders in the Romanian public
administration, including officials in the Ministries of Public
Finance, European Integration and Administration and Interior, the
General Secretariat of the Government, the Romanian Parliament,
the Delegation of the European Commission and members of
Romanian academia, civil society and the policy research
community. As such, this Strategy expresses the divergent views on
the necessary future direction of evaluation of many public and
non-public stakeholders and is not the “voice” of either the MACSF
in the MPF, or the MA for the Administrative Capacity
Development OP.
Principles of
the National Evaluation Strategy

Accountability

The Strategy assigns implementation tasks to various public
bodies. It is essential, in order for the Strategy to be implemented,
that these public bodies are answerable for the implementation of
the tasks assigned to them. It is vital that both the MACSF in the
MPF and the Administrative Capacity Development OP Managing
Authority engage other line Ministries and local authorities to take
ownership of the Strategy’s recommendations and action plan.
General Objective of
the National Evaluation Strategy

The overall aim of the National Evaluation
Strategy is that “there will be a functioning
national evaluation system, the parts of which
reinforce each other; encompassing the public and
private sector and civil society; and contributing to
the effective management of public interventions
and the accountability of policy makers and public
managers. There shall be recognition of the
importance of evidence based policy making.”
Operational Objectives of
the National Evaluation Strategy


Evaluation training and HRD resources are
available from various sources (civil service, private
training providers, universities) and open to
participants from within and without the public
sector – starting with 2007
Professional networks are established to provide a
medium for the exchange of ideas/good practice,
professional development, the development of
standards, etc. They will enjoy the participation of
civil society as well as public service participants –
starting with 2007
Operational Objectives of
the National Evaluation Strategy


A central government ‘resource’ will be created as
a repository of evaluation expertise. It will
promote good practice by networking across all
departments – starting with 2007
There is capacity at all levels of government to
commission or carry-out evaluations, and to know
the right evaluation questions to ask – 2010 for
Central Public Administration; 2011 for Local
Public Administration
Operational Objectives of
the National Evaluation Strategy


Quality and appropriate monitoring systems are in
place at all levels of government (local and central)
and in any agency where public money is spent –
2011 for Central Public Administration; 2012 for
Local Public Administration
Evaluation is integrated into the procedures for the
formulation, delivery and follow-up of all public
interventions, regardless of funding lines.
Documented procedures will ensure diligent
follow-up of evaluation results/recommendations 2013
Funding sources of
the National Evaluation Strategy





PHARE 2005 – 4 million Euro
NSRF and Rural Development and Fisheries OPs
Technical Assistance priority axes– approximately
13 million Euro
Key area of intervention of the Technical
Assistance Operational Programme of NSRF – 8
million Euro
Key area of intervention of Administrative
Capacity Development OP – 30 million Euro
TOTAL approximately 55 million Euro
Fields of intervention of
the National Evaluation Strategy






Effective evaluation of the EU funded
programmes
Evaluation training for civil servants, private,
academic and civil society
Resource and network support
Evaluation Facility aiming at encouraging Public
Administration to commission evaluations
Obtaining evaluation relevant data
Embedding of evaluation in public interventions
design and management through a normative act