Transcript Value for Money in international development using Social

Workshop: Value for Money (VfM) in
international development using Social
Return on Investment (SROI)
Natalie Nicholles
nef (new economics foundation)
19th October 2010
© nef consulting
Housekeeping
•
•
•
•
Ground rules
Fire exits
Toilets
Workshop materials
© nef consulting
Agenda
09:30
Big picture introduction to Value for Money and SROI
10:00
Crash course on SROI: understanding change
10:45
Tea/coffee break
11:00
Crash course on SROI: measuring change
11:30
Crash course on SROI: valuing change
12:30
Lunch
13:15
Crash course on SROI: attribution & partnership working
13:45
Crash course on SROI: forecasting long term benefits
14:45
Tea/coffee break
15:00
Worked-through example & organisational decision making
15:45
Answering key questions
17:00
Close
© nef consulting
Aims for the day
1. To present an overview of what Value for
Money (VfM) means
2. To introduce ways of capturing Value for
Money in international development through the
Social Return on Investment approach
3. To share nef’s work on this subject to date
4. To facilitate discussion on how to further
develop demonstrating value and impact of
international development work
© nef consulting
new economics
foundation (nef)
• Founded in 1986
• Economics Think Tank working to promote:
– Innovative solutions that challenge mainstream
thinking on economic, environmental and social
issues
• nef consulting is the consultancy arm. It exists to
promote and disseminate nef solutions
– History of working with public sector on implementing
value for money solutions
© nef consulting
Problems with measurement
1. Financial measurement:
limited measure of value
2. We allocate resources only
to the things we can measure
3: Stakeholders are left out of
decision making
© nef consulting
The challenge
• Measurement across the ‘triple bottom
line’
The economy
© nef consulting
The environment
People
Value for Money (VfM)
• VfM is about making sure that spending
achieves as much as possible
– A way of making decisions about how to use
limited resources
• Sometimes mistaken for lowest unit cost
– Without a measure of quality or effectiveness,
risk false economies
• E.g. Children in care
© nef consulting
Treasury VfM Definition
“VfM is defined as the optimum combination of
whole-of-life costs and quality (or fitness for
purpose) of the good or service to meet the
user’s requirement. VfM is not the choice of
goods and services based on the lowest cost
bid.”
HM Treasury (2006) Value for money guidance
© nef consulting
Treasury VfM Definition
“Wider social and environmental costs and benefits for
which there is no market price also need to be brought
into any assessment. They will often be more difficult
to assess but are often important and should not be
ignored simply because they cannot easily be costed.”
HM Treasury Green Book
1.Effectiveness, not lowest cost
2.Wider costs and benefits
3.Whole-of-life costs
© nef consulting
Value for Money
Resources /
Investment
Often VfM is
understood by
comparing unit costs
Money
People
Environment
© nef consulting
Service & Wider
Outcomes
Economic
Inputs
Outputs
Real VfM is achieved
by comparing outcomes
with investment
Social
Environmental
How does SROI fit in?
• Framework for telling us how effectively money
is spent – make VfM decisions
• Adjusted form of cost-benefit analysis.
SROI = value of positive + negative outcomes
investment (or cost)
• Essentially a measure of the efficiency of
achieving outcomes
• It considers triple-bottom-line benefits and
investments – economic, social & environmental
© nef consulting
History of SROI
Mid 1990s:
REDF &
Jed Emerson
© nef consulting
Early 2000s:
new economics
foundation
Mainstreaming:
nef consulting
& others
SROI today
•
•
•
•
Third Sector: SROI Guide
Local Government: commissioning
Think tanks/public policy
Central Government
– Department of Health
– National Audit Office: VfM
– DFID: VfM
• Pilot: HIV Aids Alliance
• International interest
– Europe, Australia, Canada, Asia
© nef consulting
Crash Course on SROI:
Understanding change
© nef consulting
SROI Principles
1.
Involve Stakeholders
2.
Understand what changes
3.
Value the things that matter
4.
Only include what is material
5.
Do not over claim
6.
Be transparent
7.
Verify the result
© nef consulting
SROI Process
1. Engage stakeholders to identify outcomes
2. Data collection
• Outcomes
• Deadweight, attribution, displacement
• Benefit period and drop off
3. Model and calculate
• Valuation of non-traded outcomes
4. Report
© nef consulting
Features that enhance
VfM measurement
• Focus on outcomes
– Places quality and effectiveness at heart of analysis
• Measures what matters, not what is easiest
– Stakeholder-informed
– Grapples with outcomes measurement
• Values traded and non-traded outcomes
– Failing to monetise non-traded outcomes effectively they
gives these a value of zero
• Rigour and transparency
– Concerned with impact: deadweight, displacement,
attribution, drop off
© nef consulting
Develop a theory of
change
Inputs
Activities
Outputs
Outcomes
Output: Tells you an activity has taken place and is
usually quantitative (e.g. number of people trained)
Outcome: The change that occurs as a result of an
activity (e.g. improved well-being of training participants)
© nef consulting
Case study: African Community
Development Foundation (ACDF)
Background
• ACDF started of as a U.K community based organisation, supporting
socially isolated Africans in various ways.
• In 2009, ACDF sent 28 volunteers as a pilot in Kisumu, Kenya known as Diaspora Development Associates (DDA).
• ACDF specialise in team volunteering
• Kenya has a preference for formal employment rather than selfemployment but ormal employment hard to get
• High level migration from Kisumu to Nairobi to secure jobs
• Due to preference for formal employment, very limited public interest
for business training in Kisumu so organisations do not provide it.
• The Government of Kenya in 2008 launched an ‘enterprise fund’ to
encourage people to start their own business but there was hardly
any uptake of these funds in Kisumu.
© nef consulting
Case study: African Community
Development Foundation (ACDF)
Activity
• ACDF delivered Train the Trainer courses to
infrastructure organisations and to direct beneficiaries.
• This comprised of DDAs with skills covering- business
set up, business management, book-keeping,
organisational development, fundraising etc.
© nef consulting
Exercise 2: Impact
Mapping
• What changes do the
stakeholders experience as
a result of the project? I.e.
what is the story of change?
© nef consulting
Impact map
Stakeholders
Kisumu’s
unemployed
resident
X
Kisumu’s
infrastructure
organisations
ACDF’s
volunteers
(DDAs)
State
© nef consulting
Inputs /
Activities
Train the trainer
courses
on:
business set up,
business
management,
book-keeping
Outputs
Outcomes
Outputs vs. outcomes
• Outputs do not always lead to the desired
outcome
• Key challenge in determining VfM is measuring
outcomes
– Outcomes measurement rarely takes place
– Biggest obstacle in our work across
sectors/organisations is the lack of outcomes data
© nef consulting
Crash Course on SROI:
Measuring & valuing
change
© nef consulting
Measuring outcomes
• Some outcomes are ‘hard to measure’
– subjective, qualitative, diverse, far removed
from the activity, long term
• Systems and tools for capturing complex
outcomes are improving
– distance travelled
– translating into quantitative format that
enables aggregation
– collecting complex information simply
© nef consulting
Evidencing outcomes
• Distance travelled
© nef consulting
Example: National Accounts
Model of well-being
www.nationalaccountsofwellbeing.org
© nef consulting
Evidencing outcomes
• Select ‘ways of knowing’ that an outcome (a
change) has taken place = indicators
Stakeholder
Outcome
Indicators
Stakeholder (A): long
term unemployed
Kisumu resident
Beneficiary (A) has
sustainable,
meaningful
employment
• (A) sustains
employment for at
least 6 months
• (A) reports levels of
job satisfaction
• (A) improved
financial situation
© nef consulting
Ways of measuring
Stakeholder
Outcome(s)
Indicators (Data
Collection Method)
Stakeholder (B):
Volunteers (DDAs)
Increased employability
• (B) increase in salary
Increased well-being
e.g. confidence,
empowerment,
aspiration
• (B) reported
Stakeholder (C):
Kisumu State
Improved economic
situation
• Increase in tax take
Stakeholder (D): UK
State
Improved economic
situation (from
volunteering)
• Increased tax take
from salary increase
© nef consulting
improvements in wellbeing
• Reduction in use of
state services (if
relevant)
Valuing
• SROI analyses value to all material
stakeholders, not just the one funding the
activity.
• Key question: “what does this (change) mean to
you?”
Exercise
• Imagine your employer gave you a day off. What will you
do on your day off? What does this free time mean to
you?
• If you had to put a financial value on this, what would it
be? I.e. what is this day off worth to you?
© nef consulting
Valuing and Pricing
Value means different things to different people.
• How much is your house/flat worth?
• Who sets the price of fish?
BUT
• What does it mean to a Kisumu resident to have a job?
• SROI uses financial proxies to estimate the social value
of non-traded goods to different stakeholders.
• Common currency
© nef consulting
Subjectivity of value
Value is adjusted...not by any
accurate measure, but by the
higgling and bargaining of the
market, according to that sort
of rough equality which,
though not exact, is sufficient
for carrying on the business of
common life.
© nef consulting
Subjectivity of value
Can I put a value
on anything?
Can I find a value
which, though not exact,
is sufficient for helping
me evaluate change?
© nef consulting
Financial proxies
• Sometimes this is straightforward
– mainly with outcomes to the State (e.g. value of
health)
• More difficult with non-traded outcomes
– These do not have a ‘price’ and so require a proxy, or
stand-in (e.g. emotional well-being)
– Standard economic valuation techniques
• Contingent Valuation
– Willingness to pay (e.g. new park) or willingness to accept
compensation (e.g. noise pollution)
• Revealed Preference
– Hedonic pricing (e.g. high risk job); Travel cost method (e.g. local
service); Observed spending on related goods (e.g. leisure)
© nef consulting
Monetisation
• Each outcome is then valued
• Where no direct financial value is available, we use
financial proxies to represent the social value created
Stakeholder
Outcome
Possible proxy value
Beneficiary (A): long term
unemployed Kisumu
resident
Beneficiary (A) has
sustainable, meaningful
employment
£ Value of increased earning
potential
OR
£ Cost of fees for a skills
development training course
Stakeholder (B):
Volunteers (DDAs)
Increased well-being e.g.
confidence, empowerment,
aspiration
Contingent valuation: what
would (B) pay for to get
similar level of well-being?
Revealed preference – what
do we observe (B) does to
get a similar level of wellbeing?
© nef consulting
Valuation exercise
Exercise:
1. In pairs, select 2 outcomes from the ACDF
impact map
2. Think about options for financial proxies for the
outcome.
Remember, we are putting a value on the
outcome for that particular stakeholder!
© nef consulting
Crash Course on SROI:
Attribution & Partnership
working
© nef consulting
Understanding impact
• Deadweight: what would have happened anyway?
• Attribution: how much is down to this project, and how
much down to other factors?
• Displacement: have we just moved an outcome to / from
somewhere else?
• Benefit period: how long does the outcome last, and
does the effect ‘drop off’ over time?
© nef consulting
Impact: Attribution
• Attribution – how much credit can your
organisation take for the outcomes?
– Expressed as a percentage
– Not an exact science
• Methods:
– Based on your understanding of the journey
of change
– Involve stakeholders – interviews or surveys
– Consult with other organisations
© nef consulting
Attribution stages
1. No account of attribution levels
Effectively attribution = 100%
2. Informed estimate of attribution levels
Qualitative research allows SROI practitioner to divide
attribution between contributors
3. Quantitative research
Quantitative stakeholder research gives average
stakeholder’s view on contributors
4. Academic research
Academic research can help refine (although not replace)
stakeholder research
© nef consulting
Attribution
Quantitative stakeholder
research
Time (years)
© nef consulting
Partnership working
© nef consulting
Projecting into the
future
• Benefit period – the length of time over which outcomes
are expected to endure
– e.g. benefits of an employment training programme
may endure for some years after the course
• Drop off – the rate at which benefits decrease over time
– e.g. it is likely that benefit for training participants
wears off as time goes on
• Time value of money : discount rate
• NB. Comparing value in different countries
© nef consulting
Inputs
• Size of investment
– If just looking at one project within organisation, need
to work out its ‘share’ of inputs (Full Cost Recovery)
– Where multiple funding streams, need to have
identified the share of benefits when scoping
if looking at return on just one of these investments
– Include financial and non-financial inputs e.g.
volunteer time
© nef consulting
Example &
Organisational decisionmaking
© nef consulting
Hypothetical Example
Outcome: capture the value of the influence on national
policy-making i.e. changes for stakeholders that result from
the national policy.
INGO in Nigeria
• Part of National Task Group on Sanitation
• Successfully completed community-led total sanitation
pilot (CLTS)
• Advocated within the Task Group for drafting of National
Strategy for Scaling-up Rural Sanitation and Hygiene
• Key lesson: INGO’s advocacy on the National Strategy
is not the outcome – it is a means to an end.
© nef consulting
INGO Impact Map
Outcomes
Value of
improved
health
to the
individual
Value of
increased
economic
activity to
the State
© nef consulting
Value to the individual
• No monitoring in place to track impact of the National
Strategy
• Forecasted using secondary research and stakeholder
engagement
Outcome Incidence
Assumption 1: An additional 2% of the rural population is
provided with improved access to sanitation.
Disease
Diarrhoea
Schistosomiasis
Ascariasis
Hookworm
© nef consulting
Number of incidences
avoided
29596
19319
85601
9111
Value to the individual
Impact
• Deadweight (population reached) -0.25%
– baseline trend improvements in sanitation for rural communities
(UNICEF/WHO)
• Attribution to National Strategy 75%
– significant change but concurrent factors included: International
Year of Sanitation 2008 which saw increases in funding and
other new initiatives
• Attribution to INGO 5%
– number of other actors involved in Task Group e.g. UNICEF,
DFID. Credit for role in policy formation is 35% but
implementation involved other actors including local government
and this reduced attribution to 5%.
© nef consulting
Value to the individual
Proxies – value of health to individual is made up of:
• Increased income: fewer days lost to illness (working
age population)
– Based on secondary research on average rural wage in Nigeria
• Direct costs associated with illness (all ages)
– Based on WHO research into transport & medical expenses
• Value of increased well-being (all ages)
– Non-traded outcome.
– (Limited) stakeholder engagement to find out what investments
rural Nigeria might make to improve well-being.
– Cost of generator divided by 5 for value to individual.
• All proxies converted in US$ using World Bank PPP
© nef consulting
Value to the individual
Benefit Period
• Health improvement dependent on continued access to
sanitation
– WaterAid research suggests 5 years benefit period
• Policy assumed to be in effect for 3 years
Drop-off
• WaterAid research suggests 16% drop-off over 3 years.
–
–
–
–
Year 1: 2%
Year 2: 4%
Year 3: 10%
Years 4 & 5: 20% per year (nef estimate)
Discount rate: 10% (DFID)
© nef consulting
Value to the individual
Social Value
• Total value of improved health to individuals as a result
of policy $22,561,910.66
• INGO share of value $1,128,095.53
• Cost incurred by INGO during policy influencing
$14,863.00
Return on investment
$1:$75.90
• Caution: benchmarks needed to understand whether
ratio represents VfM
© nef consulting
Value to the individual
Sensitivity Analysis ($1:$75.90)
Assumptions in model systematically varied.
• Decreasing outcome by 95% = $1:$11.81
• Well-being proxy is a sensitivity
– Doubling proxy = doubling ratio
– Removing proxy = $1:$1.21.
– Proxy informed by limited stakeholder engagement therefore
worth investigating further
• Reducing attribution to 2% reduced ratio to $1:$30.36
© nef consulting
Decision-making:
organisational
• Model to
commission/plan/
design services or
projects
• Recommendation
to DFID
• Adoption by Local
Authorities
© nef consulting
Organisational
Decision-Making
© nef consulting
Sustainable Commissioning
Commissioner &
service user priorities
1.
Activity
2.
Output
Community strategy
& Corporate priorities
3.
4.
Service level
outcomes
Camden
Community
outcomes
–social
–economic
–environment
National outcome
frameworks
© nef consulting
5.
Value
–Quantitative
–Qualitative
–Monetiseable
Where value
accrues:
- To Service
- Camden wide
- central
government
Answering key questions
• As a sector, how do we take this forward as a
sector in terms of demonstrating the value and
impact of our work?
1. Is this is an approach that can be useful?
2. In what circumstances can it be useful?
3. What are the limitations of this approach?
4. Where are the opportunities to for this to be
adopted?
5. Any other reflections/thoughts on the approach?
© nef consulting
Summary: SROI as VfM
measurement
• Focus on outcomes
– Places quality and effectiveness at heart of analysis as only
outcomes tell you that change has occurred
• Values traded and non-traded outcomes
– Enables consideration of wider costs and benefits
– Failing to monetise non-traded outcomes effectively they gives
these a value of zero
• Stakeholder informed
– Ensures that you measure what matters to those experiencing
the change – i.e. ‘quality of fit’ for service user
• Rigour and transparency
– Concerned with impact: deadweight, displacement, attribution,
drop off
© nef consulting
Summary
• SROI is a tool for assessing VfM and for
achieving continuous improvement
– Proving and improving
• Not an ‘all or nothing’ approach,
complement existing M&E systems
• Next steps – from Bond or nef
– Pilot?
© nef consulting