Transcript Nesta - The Alliance for Useful Evidence

Testing social policy innovation
Wednesday 12th February 2014
wifi: Nesta Guest
password: flourish01
follow us on twitter @nesta_uk
official event hashtag #sparkEC
Testing social policy innovation
Simon Flemington, Chief Executive Officer, LSE
Enterprise
wifi: Nesta Guest
password: flourish01
follow us on twitter @nesta_uk
official event hashtag #sparkEC
Testing social policy innovation
Hélène Giacobino, Executive Director, J-PAL Europe
wifi: Nesta Guest
password: flourish01
follow us on twitter @nesta_uk
official event hashtag #sparkEC
Support services for social policy experimentation in the EU
Improving Policies and Building Knowledge:
the Role of Creative Experimentation
Hélène Giacobino
Executive Director
J-PAL Europe
SPARK London Feb 12, 2014
The Need for Rigorous Evidence
• In a context of economic downturn, it is important to optimize
public expenditure
• Promoting the implementation of rigorous evaluation methods
to test the effects of new policies is a good way to achieve this
goal
• Evidence-based policy is possible and highly effective
• As long as we learn from our success and mistakes, this is fine,
because even a failed program helps us understand what went
wrong
• Without rigorous evaluation, everybody can favor their own pet
project – and even lessons from successes are lost
SPARK London Feb 12, 2014
Social Policy Experimentation
• Social policy experimentation tests the validity of policies by
collecting evidence about the real impact of interventions on
people
• The goal is:
– to bring innovative answers to social needs,
– to test impact on small scale interventions,
– to scale-up the ones whose results were convincing
SPARK London Feb 12, 2014
How to do SPE?
• It is possible to undertake SPE through different methods:
• Some of the most common non randomized methods are:
- Pretest-posttest (Before and after)
- Differences-in-Differences
- Regression discontinuity
- Statistical Matching
• They always rely on strong assumptions
• Randomized Controlled Trials (RCTs) give the most rigorous
results (internal validity)
SPARK London Feb 12, 2014
RCTs: a long history
• Experimental psychology: late 19th century,
• Education: early 20th century
• Experimental sociology, early 20th century:
– rural health education,
– social effects of public housing,
– recreation programs for delinquent boys
• Large-scale randomized clinical trials: a norm since 1962 Drug
Amendments
• Went through substantial debates but today widely accepted
• A boom in the 60’ in the USA, (250 RCTs done)
• 90’: J-PAL introduce RCTs in development economics (today: 450
RCTs)
SPARK London Feb 12, 2014
BUT … not every intervention can be
evaluated
• A well designed SPE should include:
- an explicit and relevant policy question
- a valid identification strategy
- a well-powered sample
- high quality data.
• Evaluation is not appropriate when:
– the sample size is too small
– the impact to measure is a macro impact
– the scaling-up of the pilot will modify the impact a lot
– the beneficiaries are in a context of urgency
SPARK London Feb 12, 2014
9
AND … not every intervention warrants
an impact evaluation
• Investing in some key rigorous impact evaluations is essential
to guide policy decisions and to responsibly allocate public
resources
• Rigorous impact evaluations should be used to test key
influential, strategically relevant or novel interventions
• No added-value to evaluate policies or programs that will
benefit a very limited number of people, or to test a policy
question that has been already rigorously evaluated in a
similar context (except within explicit validation program)
SPARK London Feb 12, 2014
10
Ethical Issues
• SPE should be designed to follow the ethical principles
applicable to evaluations and research projects involving
human subjects (even if there are no rules in the country of
experimentation) and be approved by an IRB.
• SPE should include rigorous protection of individual data
which is paramount
• SPE can have different designs to secures fairness, impartiality
and transparency (and cost effectiveness!)
SPARK London Feb 12, 2014
Financial Issues
• Rigorous SPE need good quality data: this is why, if the sample
is large and you don’t have existing data, it may be costly.
– but financing large-scale interventions without knowing
their effects can potentially result in a waste of resources!
• The costs depend on the program evaluated: it is more
expensive if you are dealing with long-term impact
SPARK London Feb 12, 2014
The « Black Box » Issue
• One common critique: “RCTs can provide whether an
intervention was effective, and even measure the size of the
impact. But they cannot answer the question of how or why
the impact came about.”
•
This is true only if final outcomes are measured. RCTs where
intermediate indicators that map to the “theory of change”
are collected, and where qualitative methods are also used,
can help us understand the how and why.
SPARK London Feb 12, 2014
Randomization is not a substitute for theory
• Although randomization guarantees the internal validity of
the estimate, in order to interpret the result, you need a
theoretical framework:
The extent to which findings do or don’t generalize
beyond a specific context depend on theory: only theory
tells you what is likely to matter in the context, and guide
replications
• Value of randomization is that you can more easily be
surprised:
– You cannot doubt the results when you found them: if
they are surprising, instead of shelving the results, you
(and others) have to think about what happened.
SPARK London Feb 12, 2014
External Validity:
an issue for all types of evaluation
• External validity: the extent to which we can be confident that the results
found in one context will generalize to other contexts
• External validity is a function of the program being evaluated (where is it being
implemented, how replicable is the program model, how much does program implementation depend on
context or interaction with the community), not of the evaluation methodology!
• Nonrandomized impact evaluations are undertaken on a specific program in
a specific location, with the downside that they do not control for selection
bias  weaker internal validity
• A randomized evaluation tests a particular question in a specific location, at
a specific time and at a specific scale. If properly conducted, and it has
strong internal validity
• If we cannot be confident that the evaluations measure the true impact of
the program in a specific context, then we can be less confident in
generalizing conclusions to another context
SPARK London Feb 12, 2014
External validity:
an issue for all types of evaluation
• The scope of an evaluation’s true external validity depends on
how the evaluation sample is determined:
• if the evaluation sample is representative of the target
population (randomly sampled from a larger population), then
the results are generalizable to that population.
SPARK London Feb 12, 2014
External validity:
solutions
• Replication is a strategy that enables us to understand how an
intervention functions in various settings
• SPE, as they force researchers to pay attention to context,
details, and realities on the ground, allow to test broad
theories in a credible way, and produce evidence that can
then feed back into our theories
• Theory and process evaluation data can help us understand
the mechanisms through which the impact was produced and
therefore to scale-up programs more confidently
SPARK London Feb 12, 2014
Conclusions
• Social Policy Experimentation is an important tool to help improve
social programs
• It is important to strategically allocate evaluation resources to key
influential, strategically relevant or novel interventions:
– to facilitate scale-up
– to encourage the replication of the policy in different contexts
– to provide valuable information for future policymaking
• Yet not every intervention can be evaluated
• And not every intervention warrants an impact evaluation
• Experimentation needs to be creative…
– If we are just trying, and accept the possibility of failure, we do
not need to think inside the box
– This mindset could revolutionize social policy
SPARK London Feb 12, 2014
Testing social policy innovation
Phil Sooben, The Economic & Social Research Council
Dr Simona Milio, LSE Enterprise
Hélène Giacobino, J-PAL Europe
Jonathan Breckon, Nesta
Arnaud Vaganay, LSE Enterprise
wifi: Nesta Guest
password: flourish01
follow us on twitter @nesta_uk
official event hashtag #sparkEC