Transcript Slide 1

Measuring the Impact of the RDP
Issues being addressed at an EU level with
regards to measuring the impact of the
Rural Development programmes
B. Schuh
www.nrn.ie
1
Content
• Assessment of impacts in the RDPs – the
basics
• Process of assessing RDP impacts
• Evaluation architecture/ challenges to
overcome
• The seven fields of impacts (econ. growth,
employment creation, productivity,
biodiversity, HNV, water quality, climate
change)
www.nrn.ie
2
Assessment of impacts in
the RDPs – the basics:
Context baseline
Identifying strengths and
weaknesses
Hierarchy of Objectives
Objective related baseline
Definition of measure
• CMEF – the „bible“
• Intervention logic
• Additional
programme
specific
indicators
• Evaluation
questions
Inputs
Outputs
SWOT
Result
reference for impact
Strategy Objectives
Impact
supporting and counterproductive trends
www.nrn.ie
3
Process of assessing RDP
impacts:
Information gathering
and analysis
• Gauging the evidence
of change
• Identifying the drivers
of change
• Understanding
change and
concluding on future
interventions
Interpretation
Judgement
Conclusions and
Recommendations
Output & Result
Indicators
Common
Impact Indicators
Common Evaluation
Questions
(per Axis and horizontal)
Additional
Programme specific
indicators
Trends of
baseline indicators
Programme-specific
Evaluation
Questions
Evaluation
Results
Other information
(statistics,
surveys/inquiries…)
Body of evidence
www.nrn.ie
Reflection space
4
Evaluation architecture/ challenges to
overcome:
Factors determinig evaluation
architecture:
Methodological challenges to
overcome:
• Dealing with uncertainties
• Reducing complexity through a
consistent approach
• Constraints in utilization of the
evaluation results – the
evaluation and policy cycle
•
•
•
•
•
www.nrn.ie
The counterfactual assessment of impacts
 quasi-experimental design, nonexperimental design; DiD method
Taking into account and cross-relating
impacts at micro and macro level  e.g.
econometric modeling, CGE models,
system dynamics modeling
Netting out the programme effects by
reducing deadweight, leverage,
displacement, substitution and multiplier
effects
data collection and processing 
qualitative & quantitative data, FADN & Co
Bridging the gap between measuring
impact indicators and providing answers
on programme impacts  qualitative
methods as add-ons – interviews, CS
5
The socio-economic impact
indicators:
• Economic Growth:
– Calculation via DiD methods, quantification possible, micro-macro link through
modeling approaches
• Employment creation
– Calculation via Propensity Score Matching, Standard regression model,
Assessing employment effects at macro level - modelling
– Time lag, Missing critical mass – CS, Welfare effects
• Labour productivity
 Calculation via DiD methods, quantification possible, micro-macro link through
modeling approaches
 Limitations concerning measuring labour productivity  Competitive
Performance, Revealed Comparative Advantage, Growth Competitiveness
Indicator, Domestic Resource Cost
www.nrn.ie
6
The environmental impact
indicators – special issues:
• Data availability
• Systemic borders
– Environmental impacts do evolve also from those measures, which do
not deem that such impacts will occur
– Conception of the environment within the evaluation (ecosystem
functions vs. ecosystem services)
• Difficult to depict the full range of rather vast, complex fields
of environmental phenomena like “climate change” or
“Biodiversity loss”
• “evaluation” vs. assessment – aggregation methods not easily
applicable
• Cumulative impacts – crossing effects between environmental
www.nrn.ie
7
impacts
The environmental impact
indicators:
• Biodiversity
– Measured by Farmland Bird Index (FBI) – bottom-up aggregation of micro level
observations
– Crucial issue of regional/ national specifics, bottom-up assessment, additional
information (‘control’ other influences), more than birds
• High Nature Value farming/ forestry
Measured through:
• Land cover characteristics, especially farmland with a high proportion of semi-natural
vegetation and in some cases a diversity of land cover types.
• Farming practices, especially a low use of inputs (including livestock density) and specific
practices such as shepherding, late hay-cutting, orchard grazing and arable fallowing.
– Strong dependence on baseline data  comparison of baseline conditions,
main challenge indicator as „work in progress“
www.nrn.ie
8
The environmental impact
indicators 2:
• Water quality
– Measured as changes in gross nutrient balance (GNB), should be interpreted
as a potential risk indicator for water quality only; assessment by bottom up
approach – aggregation as methodological challenge (modelling – e.g.
RAUMIS)
– Many uncertainties remain  different land cover, land use and farming types
& atmospheric N fixation and deposition - measuring water quality in
agricultural catchment, net nitrogen balance as additional indicators
• Climate change
– Measured as net greenhouse gas emissions reduction and production of
renewable energy
– Limitations Additionality/net effects, Displacement of energy & production,
Boundary issues, issue of the temporal attribution of longer-term impacts to
the policy period of the spending
www.nrn.ie
9