BCURE Evidence-Informed Decision-Making Capacity Building Workshop 1st and 2nd June 2015 Pretoria, South Africa

Using Evidence for Policy and Practice

Philip Davies International Initiative for Impact Evaluation [3ie]

Philip Davies www.3ieimpact.org

How Evidence is Used in Policy Making

Instrumental Use

Involves acting on research results in specific, direct ways.

Conceptual Use

Involves using research results for general enlightenment; results influence actions, but in less specific, more indirect ways than in instrumental use

Symbolic Use

Involves using research results to legitimate and sustain pre-determined positions.

Philip Davies www.3ieimpact.org

How Evidence is Used in Policy Making Instrumental, conceptual and symbolic uses of evidence are

not mutually

exclusive, but can operate in different ways at different stages in the policy cycle and under different political contexts.

Philip Davies www.3ieimpact.org

The Educational Maintenance Allowance Instrumental Use of Evidence • • • • • Tested four variants of a means-tested conditional cash transfer paid to 16–18-year olds for staying in full- time education.

Two levels of payment (£30 and £40) to either the young person or a primary carer (usually the mother); Combined with different levels of a retention bonus (£50 and £80) and an achievement bonus (£50 and £140).

Male and female young people, and in both urban and rural areas.

Comparison groups using propensity score matching

Philip Davies www.3ieimpact.org

Case Study - The Employment Retention and Advancement (ERA) Demonstration • • • • •

The Policy Issue:

How to retain, and advance, low-pay/no-pay workers in the labour market Evidence Gathering: Systematic review of the global evidence on labour market and social welfare participation Secondary analysis of existing primary data (surveys and admin) Stakeholder analysis and involvement Empirical testing in six different labour market areas using experimental design (randomised controlled trial)

Philip Davies www.3ieimpact.org

• Case Study - The Employment Retention and Advancement (ERA) Demonstration Tested the likely impact and cost-effectiveness of:  A post-employment adviser service,  Cash rewards for staying in work and for completing training  In-work training support • Random assignment of over 16,000 people from six regions of Britain: ERA programme or a business-as usual control group.

• Seven years to complete – a strategic development evaluation • But milestone data in real time from Year 1

Philip Davies www.3ieimpact.org

The ERA Demonstration Project [Extreme] Summary of Outcomes • • • • • ERA evaluation had a differential effect on different types of low-pay/no-pay workers [Anticipate heterogeneity of impacts] Initial impacts in Year1 were not sustained Initial small impacts in Year 1 had larger impacts in Years 3 and beyond The timing of impact measurement/monitoring is important Requires regular built-in M&E

Philip Davies www.3ieimpact.org

Barriers to the Use of Evidence • Policymakers’ lack of familiarity with the research process • Researchers’ lack of familiarity with the policy process • Trust (lack of trust) of policymakers in researchers (vice versa) • Timeliness and availability of evidence • Physical access to evidence • Cognitive access to evidence ( i.e. lack of understanding) • Lack of clarity in the presentation of evidence

Philip Davies www.3ieimpact.org

Overcoming Barriers to the Use of Evidence • Interactions between researchers and policymakers increases the prospects for research use by policymakers (Lavis et al, 2005).

• • Early and ongoing involvement of relevant decision makers increase research utilisation (Lomas, 2000) Identify and use interpersonal networks and interactions • • • Identify willing and able knowledge brokers Separate strategic from operational demands for evidence Get policy makers to own the evidence – not just the policy

Philip Davies www.3ieimpact.org

Improving Communication of Evidence • Establish what research says and does not say • Establish the policy messages and policy

implications

• Use a 1:3:25 format • Be clear - plain English summary • Be persistent and opportunistic

Philip Davies www.3ieimpact.org

The ‘One’ in the 1:3:25 Format • A one page of main messages • The lessons decision makers can take from the research • Not a summary of findings • Implications of findings • No details of methodology

Philip Davies www.3ieimpact.org

The ‘Three’ in the 1:3:25 Format • These are the key findings of the study • The classic Executive Summary • Condensed to serve the needs of the busy decision maker • Focus on how the study will be useful for policy • Some brief mention of methodology • Some implications of policy and practice

Philip Davies www.3ieimpact.org

The ‘25’ in the 1:3:25 Format This should include: • Context/Background • Approach (Methodology in Appendices, not text) • Results • Implications • Knowledge gaps • References • Additional resources • Appendices

Philip Davies www.3ieimpact.org

Group Exercise • Identify networks of policy makers and evidence providers in your work environment who might help to overcome barriers to using evidence • Consider what actions might need to be taken in your work environment to get evidence into policy/practice • Consider how the capacity to find, appraise and use evidence might be developed in your work environment

Philip Davies www.3ieimpact.org

Philip Davies

Thank you Philip Davies Email: [email protected]

+44 (0)207 958 8350 Visit www.3ieimpact.org

www.3ieimpact.org