Meta-Evaluation Some Reflections

Evaluation Conference, Warsaw 12/13 November 2012

Veronica Gaffey Head of Evaluation DG for Regional and Urban Policy European Commission

2 Regional Policy

1.

2.

3.

4.

5.

Presentation

Meta-Evaluation – just how ambitious do we want to be?

A more modest approach focused on the accumulation of knowledge Appropriate level – EU/MS or both?

The role of meta-analysis in ex ante evaluation Elements in the new regulation to support accumulation of knowledge, moving towards meta-analysis 3 Regional Policy

1. Meta Evaluation

•

What is it?

 Aim to improve decision making through systematic reviews on the effects of intervention •

Examples:

 Cochrane Collaboration – Health – since 1992  Campbell Collaboration – education, crime and justice, social welfare – since 1999 •

Is our evidence base sufficiently strong?

•

Issues of internal & external validity

Regional Policy 4

Our evidence base

• Of 266 completed evaluations examined by Expert Network in 2011, only 34% concerned effects; 66% implementation issues • Pilot review process for DG REGIO finds few evaluations of Cohesion Policy of really good quality • Difficult to judge quality of evaluations without terms of reference, budget and knowledge of context 5 Regional Policy

2.

Accumulation of Knowledge

•

Some experience here:

 Meta-analysis of 200 evaluations across Europe prepared for Polish Presidency Conference on Evidence based Cohesion Policy  Forthcoming Regional Focus on What we have learned from counterfactual evaluations of innovation and enterprise support (Daniel Mouqué)  Internal Analysis on Transport and Environmental Infrastructure – likely to be transformed into guidance •

Insights rather than proofs; questions rather than prescriptions

6 Regional Policy

3. Appropriate Level for Accumulating Knowledge

•

Experience shows it can be done at EU level, if

 Terms of Reference and budget are public  Context and method are explicit  "Why it works?" is addressed as well as "Does it work?" •

Language issue can be overcome

•

DG REGIO starting to build a library of MS evaluations

•

Also at Member State level, though

 Difficult to generate critical mass Regional Policy 7

4. Meta-Analysis in Ex Ante Evaluation

• Ex ante evaluation should be theory and evidence based • Testing the logic of intervention is at its heart – evidence from literature and other evaluations should be used • Unpick the triangle in the next slide….

Regional Policy 8

Regional Policy 9

5. Elements in the New Regulation

• Clearer articulation of objectives with associated result indicators and baselines – success can be judged • • Requirement for an evaluation plan Requirement for an impact evaluation of each priority during the programming period • Summary report in 2021 – a meta evaluation?

10 Regional Policy

Possible Next Steps…

• We continue to encourage more impact evaluations – using different and combined methods • Competition/Prize for good Terms of Reference?

• Structured exchanges of experience – based on themes • Building communities of practice – involving public administrations & academic experts 11 Regional Policy

Thank you for your attention

12 Regional Policy