Transcript Document

Measuring success through
outcomes:
A way forward
Les Hems July 2014
Why Outcomes
Measurement is
important
1
‘For purpose’ is a better concept than ‘not-for-profit’
Limitations of existing terminology:
| Not-for-profit – contrast with for-profit business - but all
organisations need to make a surplus to be sustainable
| Charity – dependence on grants and donations - but all
organisations need a reliable source of revenue to be sustainable
Increasing preference for alternative terminology:
| For-purpose – mission driven - creating social change and impact
| Social enterprise – co-creation of commercial and social value
Increasing interest in alternative structures:
| Co-operatives and mutuals – owned and controlled by members
| Civil Society – organisations embedded in communities
| Co-operative Collective Impact – real community collaboration
2
Measuring outcomes rather than Value For Money
For-purpose: The problems with VFM measurement
1. Financial measurement:
limited measure of value
2. We allocate resources
to the things we can measure
3: Stakeholders are often
left out of decision making
What are outcomes and impact?
| Outcomes are the significant positive or negative changes
for clients (customers / participants), their families, the
broader community and other stakeholders.
| Outcomes:
 Can be short, medium and long term
 Can be direct and indirect (spill over)
 Can be for target beneficiaries and other stakeholders
 Can be economic, social, human and environmental
| Impact is the net effect of these outcomes
5
Inputs to impact via outputs and outcomes
Input: The resources you use to undertake tasks and
activities – not just financial – human capital etc.
Activities: The programs and tasks you undertake to
generate outputs
Output: Tells you an activity has taken place and is usually
quantitative
Intermediate outcomes: The short term change that occurs
as a result of an activity
Outcome: The long term change that occurs as a result of
an activity – the net benefit to recipients – the intended
results
Impact: The long term net benefit to society
6
Why measure outcomes? External perspective
Clients / consumers
| Help consumers make good choices and to achieve their goals
Funders – Government, foundations, donors and corporates
| Recognised limitations of funding activities and outputs
| “Investment” in prevention, early intervention and breaking the cycle
| Client directed care
| Payment by Results (PbR) / for Outcomes, / for Success – linked to
Social Impact Bonds
| Collective impact initiatives focused on outcomes
Evidence
| Focus on what works – the desired outcomes being achieved
Measurement and reporting
| Increasing use of approaches such as Social Return on Investment,
Results Based Accountability
Radical / disruptive changes in complex systems
Measuring outcomes is at the
heart of social change – interest
in social impact has been
growing and will become the
norm:
| Higher expectations
| Ineffective responses to
wicked social problems
| Increasing demand
| Reducing tax revenues
Why measure outcomes? Internal perspective
Mission and strategy
| Organisations’ mission statements should focus on outcomes
| Outcomes are the fundamental purpose of an organisation and
therefore the focus for organisational strategy
Performance measurement
| Measuring outcomes is the most important measurement of an
organisation / program performance
Benchmarking
| Emulating best practice – outcomes frameworks, outcomes
reporting and social impact statements
High performing organisations are learning organisations
| Organisational learning – the development of a framework and
measurement of outcomes generate learning
12
Key concepts for
measuring outcomes
13
Key concepts: Stakeholders
| Stakeholder: Any actor or agency that has an effect on,
or is affected by the intervention:
 They contribute to or experience the change
 Can be individuals, families, organisations (e.g. different
tiers of government), collectives, the environment
| Key elements for outcomes measurement:
 Stakeholder mapping
 Network analysis – interdependence between
stakeholders
 Stakeholder engagement
14
Network analysis – partnerships and collective impact
15
Key concepts: Client segmentation
Clients / customers / participants are not all the same:
| Demographics
| Vulnerabilities
| Capabilities
| Expectations and aspirations
Map client segments and their related outcomes
| Many to many relationships between client segments and
outcomes
How will the program / intervention respond to these different
client segments and outcomes?
| High, medium and low – clients & complexity of service
Key concepts: Materiality
| Focus on the stakeholders experiencing material outcomes
 If I left this outcome out of the measurement, would
somebody make a different decision about the program under
analysis:
| Consider relevance and significance to identify materiality
| Iterative process:
 Make an initial determination to decide which stakeholders to
include in the stakeholder engagement.
 Then, consult with stakeholders to identify their perspective.
| You can’t measure everything.
| Stakeholders with material outcomes will be a subset of the
stakeholders and outcomes you identify
17
Key concepts: Theory of change
| A depiction of how a program or intervention:
 Is meant to work
 What it is supposed to achieve
 Identifies any cause and effect relationships between activities
and outcomes
| Explores the how and the why
| Combines evidence and direct experience
| A framework for evaluating programs, and designing
and predicting the outcomes of a program
| Can also be used to develop an ‘outcomes hierarchy’
for an organisation as a whole
18
Key concepts: Impact
| Impact refers to the net benefit to society - the ADDED
VALUE of your programs
Key concepts to consider:
| The counterfactual – what would have happened
otherwise – “Deadweight”
| Transferring the problem to or outcome from elsewhere
– “Displacement”
| The proportion of the change which relates to your
organisation’s activities – “Attribution” or
“Contribution”
| The proportion of the change which relates to working
collaboratively – “Interdependence”
19
Principles for outcomes measurement
Principle 1: Involve your stakeholders to measure social value
| Measure what matters from the perspective of your stakeholders – the
people or organisations that have experienced change as a result of your
activities.
Principle 2: Measure and understand the theory of change
| Apply appropriate methods and resources to understand and them
measure the change (positive and negative, intended and unintended)
Principle 3: Be accurate and credible
| Added value and impact needs to take into account:
 What would have happened anyway? Deadweight
 How much did other organisations contribute to the change?
Attribution / Interdependence
 Have the issues moved somewhere else? Displacement
20
Theory of change –
the foundation of
outcomes
measurement
21
Theory of change
| A theory of change depicts how a program is intended to
achieve meaningful, positive changes for clients — not
simply a description of what the intended change is
22
Demonstrating outcomes leads to more inputs
| Feedback loop - demonstrating the outcomes – both in words and
numbers – can be instrumental in generating more inputs - the
evidence of what works wins tenders, attracts investors, and
provides collateral for marketing and fundraising.
23
Example: Community housing for disadvantaged youth
Stakeholder Inputs
Activities
Outputs
Outcomes
Residents
•Access to
housing
•Formation
of groups
•Number of
groups formed
•Number of new
friends
•Increased community
inclusion
•Increased sense of
security
•Improved personal health
•Reduced
number of
hours family
member is
engaged
•Increased income for
family
•Less time spent on care
for family member
•Health and
welfare service
usage
•Less usage of intensive
child support services
•Reduced costs to
Department of Health
and Human Services
•Reduced cost to State
medical system
Family
members
Government
24
•Accommodati
on
•Not
•Maintenance
applicable
costs
•Management
costs
•Staff
•Not
expenses
applicable
Example: Community housing for disadvantaged youth
Activities
Provision of
secure &
affordable
housing
Referral to and
provision of
health,
employment
and other social
services
Building of
inclusive
communities
25
Outputs
Access to
housing
More regular
sleeping
patterns
Appropriate
treatment
accessed
Use of
employment
services
Participation in
group activities
Intermediate
outcomes
Increased sense
of personal
safety
Decreased
financial stress
Decreased
mental stress
Reduced
substance
misuse
Greater ability
to apply to jobs
Greater social
engagement
Long-term
outcomes
Increased
sense of
security
Improved
personal
health
Increased
employmen
t readiness
Increased
community
inclusion
Combines evidence and direct experience
| Preparatory desk review of evidence from:
 Academic research and evaluations – national & international
 Population level indicators and validated measurement tools
 Existing organisational programs
| Direct experience of those:
 Delivering the service
 Receiving the service
 Other stakeholders
| Iterative process
 Develop prototype from evidence and those delivering the
service
 Refine ToC through beneficiaries and other stakeholders
26
The total value created by community services
Inputs from
stakeholders
Outputs & outcomes
for stakeholders
Program Logic
Social Return on Investment
Commercial Return on Social Investment
Recognising all inputs and value created
| Financial– not just revenue but balance sheet
| Manufactured– the physical assets, internal systems you
use
| Intellectual– the research and development, innovation and
piloting you undertake
| Human– the knowledge, capabilities & experience,
wellbeing & health of all those who contribute– paid/unpaid
| Social and Relationship– networks, trust and societal norms
– relationship value – within communities (place and
interest) and families – between organisations
(Interdependence)
| Natural– the environment
28
Measuring Change
29
Quantitative research – evidencing outcomes
| Once we have identified and defined outcomes through the
qualitative stakeholder engagement in the Theory of
Change, we can quantify them.
Key questions:
| How do we know how many people have experienced this
outcome?
| How do we know the extent to which this change has taken
place?
| By definition - need to measure change over time:
 Retrospective construction – “How are you today?” and “Has
this got better or worse over the last two years?”
 Baseline and follow-up – embedding measurement in to an
organisation
30
Some basic principles for quant work
| Maximise the utility of all relevant previous work – external
and internal
| Build a counterfactual:
 Ideally, establish a control group to assess deadweight
 If not, use self reporting, ask “What would have happened if
you had not received help?”
| Appropriate sample size and diversity:
 Be pragmatic as can be resource intensive
 Use of client records
| Pilot your measurement tool / instrument!
| Do you need incentives?
| Do you need Ethics Approval?
Indicators
| Indicators are key component of measuring outcomes
i.e. the changes that have taken place
| Indicators can be Objective e.g. no. of health clinic
visits or Subjective e.g. self-reporting on physical health
| Intermediate outcomes achieved can be used as
indicators to measure the final outcome
| Lead indicators can also be used to assess likelihood
of final outcomes being achieved:
| Some indicators will work for multiple programs
| Be pragmatic but don’t cut corners: You may not have
access to data required to measure ideal indicators.
32
Evidencing outcomes – distance travelled
Evidencing outcomes – distance travelled
Outcome
Achievem
ent Level
Indicators
- I am always short of money for basic expenses
Very Low - I am unable to pay any of my regularly scheduled bills
- I have no idea of what my outstanding debts are
Low
Improved
money
Average
managem
ent
- I often find myself short of money for basic expenses
- I often default on my regularly scheduled bills
- I know that I owe money but don’t know how much or how I can pay it back
- I have tried using a budget but found it difficult to follow
- I occasionally fall behind on my regularly scheduled bills
- I am aware of the extent of my debts and want some help to clear them up
- I am using a budget regularly but sometimes tend to lose track of my expenses
- I am making some changes in my habits to be able to afford my regularly
High
scheduled bills
- I am seeking help to begin to payback my previous debts
- I am able to purchase my basic requirements and stay within a budget
- I never default on my regularly scheduled bills
Very High
- I have been able to eliminate all my outstanding debts OR
- I am following a payment plan to wipe out my outstanding debts
34
Evidencing outcomes - Example
Clients End
Position
Improved Money Management
Sample
Size
Total
35
Very Low
5
Very Low
Low
Medium
High
Very High
Client's Initial Position
Low
Medium
High
6
7
30
60
4
6
2
Very High
Approaches for
measuring outcomes
36
What’s the best approach for measuring outcomes?
| There is not a ‘best’ approach
| Choose the most suitable and appropriate approach based
on your needs
| One program, one organisation or a collective (shared
measurement)
| There may be a role for using multiple tools depending on
your needs
| Remember the principles of good outcomes measurement
 Involve stakeholders
 Understand what changes
 Understand impact or added value
37
Social Return On Investment
| A way of understanding how effectively money is spent
| A way of comparing the value of change created to the
investment required to create the change
SROI = [Value of change]
[Investment]
e.g. $1: $3.60
| It is an enhanced form of cost-benefit analysis which
considers triple-bottom-line benefits and investments
(economic, social, environmental)
| It measures outcomes, not outputs
| It measures change that matters to stakeholders
| It can be evaluative or predictive
SROI – a principle-based approach
1. Involve Stakeholders
2. Understand what changes
3. Value the things that matter
4. Only include what is material
5. Do not over claim
6. Be transparent
7. Verify the result
39
Suitable applications for an SROI
| Organisational learning – finding out what works and why
| In circumstances where you require a financial valuation:
 Where value for money - in its broadest sense - needs to be
demonstrated
 Social impact bonds
 Requirement of a funder
| Where a common unit of measurement ($) would be useful:
 Compare across projects / programs within an organisation
 Different types of outcomes i.e. social, economic and
environmental
| Where there are multiple interventions and contributors –
collective impact:
 SROI provides a framework for understanding your unique
contribution
40
SROI – strengths and weaknesses
41
Strengths
Weaknesses
• Stakeholder focused process provides
great insight into stakeholders and
programs
• Powerful as a design and evaluation
tool
• Provides a framework for
understanding ‘added value’
• Is a basket of best practice tools – can
be applied in its full form or using
component parts
• Speaks the language of some
funders/decision makers
• Has wide applicability across sectors
and subject matter
• Overemphasis on the ratio
• Heavily reliant on subjectivity and
assumptions -will evolve and
strengthen over time
• The valuation process is often a source
of discomfort/contention
• It is complex and resource intensive
• Often used in a snapshot or
retrospective context which constrains
data quality
• Comparison of ratio is inevitable but
not always appropriate
Measuring outcomes – a proposed approach
| Outcomes Hierarchy for an Organisation – derived from a
high level organisational Theory of Change
| Select program(s) for outcomes measurement and
organisational learning
| SROI including program level Theory of Change
 Outcomes Measurement Framework – based on Total Value
Framework
 Embed measurement processes in to organisational systems
| Formative process evaluation – feedback learning in to
programs in real time
| Stories as well as numbers
| Reporting – dashboards, infographics, spreadsheet models
42
Reporting outcomes
43
Case study: MLC Foundation & Lifeline Research
Foundation
The MLC Community Foundation and the Lifeline Research Foundation
undertook an impact study to measure the social value created by Lifeline
Crisis Support Chat Service in suicide prevention in Australia.
Social Outcomes
Three main social outcomes were identified for the service users:
| Reduced suicidality/self harming potential
| Improved resourcefulness
| Enhanced belonging
In addition, two areas of cost re-allocation were identified for public services:
| Reduced use of hospital services
| Reduced call outs for police and emergency services
| See flyer, report and presentation
The Social Value Created
Total Social Value
Crisis Intervention
(High Risk)
Crisis Aversion
(Medium Low Risk)
Emergency Service
Medical Service
Total
Present Value of
Social Value
$2,367,059
$2,233,075
$3,166,579
$469,884
$1,639,367
$7,642,889
$2,987,339
$443,286
$1,546,573
$7,210,273
Total Value of Inputs $860,517
Social return on investment ratio = ($1:$x) $8.4
“FFA does good things for people and
allows them to do positive things”
“Feels like a weight has been lifted off my shoulders”
“Without loan, there would be no transport, would
not be able to get around would still be trying to figure
out how to pay Cash Converters
“The payments are flexible, they said that if I wasn’t able to afford
the repayment in 12 months, it could be changed to 18”
IMPROVED
QUALITY OF LIFE
REDUCED ANXIETY
Ability to improve
credit profile
Explore new
opportunities
Easier access
to future credit
Enthusiastic about
new opportunities
Discovered
stability in their life
Take advantage of
new opportunities
Ability to spend
within means
Prepared for any unexpected
future payments
“...no one else
provides a service
like this”
“I like the idea of it
being a small loan”
Liberated from past
constraints
Access to
favourable
payment terms
Ability to meet
future bill
payments
Ability to pay
off debts
Appropriate
Ability to budget for
future payments
Awareness of outstanding
bills and terms
Made changes
to household
expenditure
“I got into alcohol, got
drunk everyday, I don’t
want to go back there”
fi
ng
n a n ci
Leave behind
negative influces
Decreased bouts
of depression
3
Undertake
previously
impossible tasks
Access to
immediate
credit
1
Opportunity to
make a new start
Confident of
improving life
Regain
independence
2
Knowledge of how
to prepare a
personal budget
Referral to
financial
counsellor
“There is the human element here
“...they’re like your friends”
“...these people really care about you and are not like Cash
Converters who are just out to get you”
Develop trust In
loan officers
Develop a positive
future outlook
INCREASE IN
SELF ESTEEM
Ability to get things
done for self and
family
“I just wanted to move things
along and not have to wait”
46
“It has allowed her to have
a whole new start”
“Has allowed me to start
life afresh
“The loan saved my life”
“God heard my prayers”
Set new life goals
Trust and support of
family and friends
“I was ecstatic when I was approved,
I had found some freedom in my life”
Read more from
case studies
Reporting outcomes
47
Be prepared
| Client directed care will become the norm:
 Increasing competition between providers
 Greater emphasis on marketing and communication
 Risk of reliance on brand instead of market intelligence
| Evidence of impact - and value for money - will assist
decision-making and allocation of resources
| Increasing interest in social finance which requires:
 A Payment by results / outcomes mechanism to trigger reward
payments
 A rigorous method for measuring changes in outcomes
 Monetising the value of achieving outcomes – cost savings to
Government and/or value of improved outcomes
Challenges: External
Complexity
| Complex problems with multiple outcomes – limitations of linear cause
and effect
 Soft systems & Collective Impact
 Maximise use of existing evidence and knowledge
Funding systems
| Most existing government administrative & funding data is based on
activities and outputs
 Develop own data systems in preparation for outcomes based funding
Standards
| No reporting requirements for outcomes nor repository for outcomes
and frameworks
 Encourage self reporting and celebrate excellence
Challenges: Internal
Human resource
| Limited expertise / competency and capacity
 Capacity Building and co-production
Organisation systems
| Lack of outcomes related data and and reporting
 Implement case management systems
Organisational change
| Inertia – organisation culture - resistance to change
 Case studies of success
Costs
| Dollars and time
 This is an investment that will yield long term value
Measuring outcomes to achieve success:
The Way Forward
Collective Impact
Experience
Biggest challenge for ForPurpose organisations
Source: Leth & Hems (2013
For further information
Les Hems
Director of Research and Development
Net Balance Research Institute
+61 448 588 450
[email protected]
Twitter: @LesHems
Erica Olesson
Service Line Lead Net Balance Research Institute & New
Zealand co-ordinator
[email protected]
52
Stakeholder engagement: Resources
| Stakeholder engagement:
 Prove and Improve http://www.proveandimprove.org/
 AccountAbility
http://www.accountability.org/images/content/5/4/542
/AA1000SES%202010%20PRINT.pdf
53
Theory of Change: Resources
| ActKnowledge’s Theory of Change community:
http://www.theoryofchange.org/
| New Philanthropy Capital’s guide to Theory of Change:
http://www.thinknpc.org/publications/theory-of-change/
54
Indicators and data collection: Resources
| Guide to Measuring Well-being:
http://www.neweconomics.org/publications/measuringwell-being
| Outcomes Star:
http://www.outcomesstar.org.uk/
| National Accounts of Well-being:
http://www.nationalaccountsofwellbeing.org/
| The Global Value Exchange
http://www.globalvaluexchange.org
| Sample size calculator
http://www.nss.gov.au/nss/home.nsf/pages/Sample+siz
e+calculator
55
References
SROI Network http://www.thesroinetwork.org/
SROI Guide – on your USB
Results-Based Accountability http://raguide.org/
Logical Framework Approach, Uni of Wolverhampton
Introduction – on your USB
| Code of Good Impact Practice – on your USB
|
|
|
|
56