Transcript Kittens are Evil

Why Payment by
Results Fails
My heresy:
Outcomes are not effective as
performance management
tools
#LittleHeresies
Payment by ‘Results’
=
Payment for data production
#LittleHeresies
Language
Outcomes-Based Performance Management

“Outcomes Based Evaluation”

“Outcomes/Results Based Accountability”

“Results Based Management”

“Payment by Results”
TM
= measure performance by the impact a
person/team/organisation/project has in the
world
#LittleHeresies
Research Findings
•
•
•
•
Measurement Problem: “Outcomes”
aren’t outcomes
Attribution Problem: Outcomes are
beyond the control of organisations
OBPM distorts organisations’ priorities
OBPM undermines good frontline
practice
#LittleHeresies
Measurement Problem:
OBPM doesn’t measure impact
#KittensAreEvil
“One clear and compelling
answer to the question of "Why
measure outcomes?" is: To see
if programs really make a
difference in the lives of
people.”
United Way of America
#KittensAreEvil
How do you measure an
outcome?
“Outcomes Based Evaluation”,
by Robert Schalock, 1995
#LittleHeresies
How to measure
an outcome
• Delivery group/control group
• In-depth qualitative research
• Large scale quantitative
research – designed by
participants
#KittensAreEvil
How to measure
an outcome
Minimum post-programme
research time?
18 months
#KittensAreEvil
Minimum income guarantee
study, USA, 1973:
600 families, 300 with intervention,
300 control group
$3m programme cost
$5m programme evaluation cost
Donald T. Campbell, “Assessing the Impact of Planned Social
#LittleHeresies
Change”, December 1976
10 Commandments
of Payment by Results:
“Commandment 2: Thy outcomes
shall be few”
“Commandment 3: Thy metrics
shall be simple”
Russell Webster, The 10 Commandments of Payment
by Results
#LittleHeresies
What does get measured?
• Start with existing data
• Develop new data – “the smallest
possible data set which can be
used to run the program”
• Use ‘Proxy’ measures
Mark Friedman, Results Based Accountability Implementation
Guide
#LittleHeresies
Body Mass Index (BMI)
“BMI is an attractive measure because it is an
easy, cheap and non-invasive means of assessing
excess body fat.”
“true measures of body fat are impractical or
expensive to use”
National Obesity Observatory, 2009
#LittleHeresies
#LittleHeresies
“factors such as fitness, ethnic origin
and puberty can alter the relationship
between BMI and body fatness.”
“therefore, BMI may not be an accurate
tool for assessing weight status at an
individual level.”
National Obesity Observatory, 2009
#LittleHeresies
“substantial” portions of women
were misclassified [as obese] by
BMI measures
Katz et al, “Obesity and Its Measurement in a Community-Based Sample of
Women with Systemic Lupus Erythematosus.” Arthritis Care & Research 63,
no. 2 (February 2011): 261–268 2011.
#LittleHeresies
“A picture of
obesity: NHS tells
bodybuilder she
must lose weight
and exercise
more…”
Metro, 26th March,
2014
#LittleHeresies
“What became evident was the need to create an
impact measurement tool that is easy for all types
and size of organisation to use but which can
also generate management information of
sufficient quality to enable investment decisions
to be made with confidence.”
Sportworks, Full Report, 2012
#LittleHeresies
“These measures of
impact are based on
the degree to which
sport for development
practice reduced the
risk of young people
facing negative social
outcomes.”
Sportworks, Final
Report, 2012
#LittleHeresies
#LittleHeresies
Payment by Results
=
Management by Targets
#LittleHeresies
The attribution
problem:
Outcomes are beyond the control of
organisations
#LittleHeresies
“In complex systems, patterns
emerge as a result of multiple
interactions between agents… they
are coherent in retrospect but not in
advance.”
Snowden, D. ‘Managing for Serendipity or why we
should lay off ‘best practice’ in KM, ARK
Knowledge Management Vol. 6, Issue 8, 2003
#LittleHeresies
Programme Logic Model
Robert Schalock & Gordon Bonham “Measuring
outcomes and managing for results”, Evaluation and
Program Planning, 2003
“Complexity is almost always the
enemy of effective PbR because it
inevitably results in unintended
consequences.”
Russell Webster, The 10 Commandments of PbR
#LittleHeresies
“Results decision-making uses
results…as the starting point for
making decisions. It is a business-like
process that starts with ends and
works backwards to means.”
Mark Friedman, Results Based Accountability
Implementation Guide
#LittleHeresies
“the more important the performance
measure… the less control the program
has over it. This is a paradox at the heart of
doing performance measurement well”
Mark Friedman, Results Based Accountability
Implementation Guide
#LittleHeresies
“Don't accept lack of control as an
excuse… If control were the overriding
criteria for performance measures
then there would be no performance
measures at all. ”
Mark Friedman, Results Based Accountability
Implementation Guide
http://www.youtube.com/watch?v=xH_6_8NOfwI
Theoretical problems:
•
•
Measurement problem: Outcomes don’t
measure impact in people’s lives
Attribution problem: Outcomes are
beyond the control of organisations
#LittleHeresies
What happens when
people implement OBPM?
What’s the evidence?
#LittleHeresies
A very quick summary of the
evidence….
“The overall conclusion from
international experience of
implementing an outcomes
approach is that the journey is long
and the results are disappointing.”
Wimbush, Erica (2011): ‘Implementing an outcomes approach
to public management and accountability in the UK—are we
learning the lessons?’, Public Money & Management
OBPM creates:
•
“Goal displacement”
•
“Creaming”
“Making the numbers”
•
Burt Perrin “Effective Use and Misuse of
Performance Measurement”, American
Journal of Evaluation, 1998
#LittleHeresies
Targets for results “frequently
distort the direction of
programs, diverting attention
away from, rather than towards,
what the program should be
doing.”
Burt Perrin, “Effective Use and Misuse of Performance
Measurement”, American Journal of Evaluation, 1998
#LittleHeresies
“Unintended consequences”:
•
•
focusing on those who are
easiest to help
“difficult” clients are skipped in
favor of the “easy” ones
S van Thiel and F. L. Leeuw “The Performance Paradox
in the Public Sector”, Public Performance and
Management Review, 2002
#LittleHeresies
“Ossification, a lack of
innovation, tunnel
vision and
suboptimization”
S van Thiel and F. L. Leeuw “The Performance Paradox in
the Public Sector”, Public Performance and Management
Review, 2002
#LittleHeresies
“Target based performance
management always creates
‘gaming’” (my emphasis)
Bevan, G. and Hood, C. “What’s measured is what
matters: targets and gaming in the English public
health care system”, Public Administration, 2006
#LittleHeresies
Campbell’s Law
“The more any quantitative social
indicator is used for social decisionmaking, the more subject it will be to
corruption pressures and the more apt
it will be to distort and corrupt the social
processes it is intended to monitor.”
Donald Campbell, Assessing the Impact of
Planned Social Change,1976
#KittensAreEvil
The Work Programme:
“the available evidence to date
suggests that providers are
engaging in creaming and
parking, despite the differential
payment regime.”
Work Programme evaluation: Findings from the
first phase of qualitative research on programme
delivery, DWP, 2012
#LittleHeresies
Triage “parks” disabled people on Work
Programme
Independent, Monday 28th January, 2013
“Work advisers 'pushing jobless into
self-employment”
BBC, 3rd February, 2013
“Private health contractor's staff told to
cut 999 calls to meet targets”
Guardian, Wednesday 23 January 2013
“NHS targets 'may have led to 1,200
deaths' in Mid-Staffordshire”
Daily Telegraph, March 2009
#LittleHeresies
“A4e employee forged signatures to
boost job placement numbers”
The Guardian, 6th March, 2012
“Serco gave NHS false data
about its GP service 252 times”
Guardian, Thursday 20 September 2012
“G4S and Serco face £50 million
fraud inquiry”
The Independent, 12th July, 2013
#LittleHeresies
Where does gaming occur?
2. Define an
outcome
based on
proxy data
1. Accept
you can’t
measure
outcomes
3. Set targets
for
performance
4. Providers plan
activity to meet
outcome targets
8. Improve
programme or
game the
system?
5. Providers
deliver activity
7. Realise
limits of
influence
6. Gather
outcomes
data
Options for Managers
High Cost
Improve programme to
meet service user needs
Teach
to the
Test
Uncertain Effect on
Results Data
Certain Effect on
Results Data
Cherry
Pick &
Park
Reclassify data
Make up data
Low Cost
Options for Managers
Game
#LittleHeresies
Theory helps us understand
the evidence…
Effects of proxy measures
+
Lack of control
=
Gaming
#LittleHeresies
Impact on
frontline practice
#LittleHeresies
Results-Based Accountability
•
•
•
•
Makes “it more difficult to engage with and
build relationships with homeless and at
risk young people”
Has significant impacts on the daily practice
of workers
Reduces the time available to create a
sense of belonging
Reduces the time to “develop young
people’s life skills”
Lynn Keevers (et al) “Made to Measure: Taming Practices with Results-based
Accountability”, Organization Studies, 2012
What Social Workers do…
• 86 per cent of time is system driven filling in forms for accountability and
discussing them with colleagues.
• The 14 per cent of time spent face to
face with a family member is not
developmental.
Hilary Cottom, Relational Welfare, 2011
#LittleHeresies
What Social Workers do…
“The dialogue between Ryan and Tom
is dictated by the forms and their need
for data and information. This squeezes
out any possibility of the sort of
conversation that might be needed to
develop a supportive relationship as a
first step in fostering change.”
Hilary Cottom, Relational Welfare, 2011
#LittleHeresies
Frontline practice
= Reversal of relationship between
worker/client
From: how can I help you achieve
your goals?
To: how can you help me achieve
my outcome targets?
#LittleHeresies
What gets measured
is what gets done
What do you want your staff’s
purpose to be?
Hitting targets?
Or helping people as they present
to your service?
#LittleHeresies
PbR: Doomed to fail
•
•
Outcomes don’t measure impact in
people’s lives
Outcomes are beyond the control of
organisations
•
PbR distorts organisations’ priorities
•
PbR undermines good frontline practice
#LittleHeresies
If not OBPM, then what?
Some learning to start from…
•
•
Results are what matter, but
targets prevent people from doing
the work that’s needed
You can’t make
people/organisations accountable
for outcomes
#LittleHeresies
Service Managers
•
•
•
•
Understand your clients’ actual
needs
Design your services around your
clients, not around targets
Measure to improve, not to be
accountable
Create transparency and critique
#LittleHeresies
to improve
Commissioners:
Commission the organisations that
do these things.
Get feedback direct from users, not
through paperwork
Change the system to enable and
empower the frontline to do a good
job for clients
#LittleHeresies
If you do this, you create a system
which:
•
Promotes learning, not compliance
•
Creates better outcomes for clients
•
•
Supports the organisations which
create impact
Is cheaper: because you’re
eliminating waste
#LittleHeresies
Thanks for listening
Toby Lowe
E: [email protected]
Twitter: @tobyjlowe
#LittleHeresies