Transcript Lorem ipsum dolores - Marie Stopes International

Impact 2
An innovative tool for measuring the impact of
reproductive health programmes
Overview and Training
SLIDE 1
Outline
1.
Background
2.
What’s new and different
3.
How it works
4.
Types of results
5.
Using the model-- quick demo and activity
6.
Using results in your work
SLIDE 2
Why use Impact 2?
SLIDE 3
Why we need models
Output
Commodities
and services
delivered
Outcome & Goals
Women using
FP
Health,
demographic,
economic
impacts
Challenge: difficult (or impossible!) to measure outcomes &
goals – so use Impact 2 instead!
SLIDE 4
What it does

Estimates wider impacts of family planning services

Helps service providers, donors & governments to communicate:
– Demonstrate value for money
–
Compare FP with other health interventions

Set realistic programme goals

Monitor progress over time

Motivate staff
SLIDE 5
Impact 2 – what’s new?
2009
Impact Estimator 1.2
2010
REACH Calculator 1.2
CYPs  Births averted
Maternal deaths averted
$s saved
etc.
FP services 
FP users
Market share
CPR contribution
2012
Impact 2
SLIDE 6
What’s new?

Impacts are no longer based on CYPs- instead run off
service provision data by method/year

We can now estimate annual impacts (ie maternal deaths
averted in 2010)

Different ways to see impacts– not just 1 result!

You can now set goals for reducing maternal deaths and
unsafe abortions nationally.
SLIDE 7
How it works
Past and/or planned future
service provision to impact
Impact
service/product
provision
Estimated
users
CPR contribution,
market share,
maternal deaths averted , etc.
Impact goal to services provision
SLIDE 8
Background Information

Excel-based
–


Transparent & simple to use for non-experts
Open-source
–
All tools & training available on MSI website late March 2012
–
Pre-loaded with best data from UN, WHO, DHS etc for all developing countries
(Very little data entry required)
Expert collaboration & review
–
Presented at Global Family Planning Conference, Senegal 2011
–
At least 8 organisations (inc DFID) using older Impact 1.2
–
Peer review of methodology: DFID, UNFPA, Guttmacher, LSHTM, Population Council,
Futures Institute, Futures Group, PSI, IPAS, IPPF, EngenderHalth
SLIDE 9
Family planning services/products provided
Count backwards
Products provided
Units needed per year
Apply continuation and mortality
rates to past LAPM clients
CPR contribution
Estimated Family Planning Users
Market share
Pregnancies averted
Miscarriages and
stillbirths averted
Births averted
Child deaths averted
(Due to improved birth
spacing)
Maternal deaths
averted
DALYs averted
(maternal morbidity &
mortality,
child mortality)
Abortions averted
Unsafe abortions
averted
Costs saved to
healthcare system
(Direct treatment costs:
SLIDE 10
ANC, delivery, PAC, pregnancy
and birth complications)
Post-abortion care (PAC) services provided
Market share
(PAC provided/abortions
nationally)
Maternal deaths
averted
(unsafe abortions x
unsafe abortion
mortality)
DALYs averted
(years lost due to
maternal mortality; and
unsafe abortion related
maternal morbidity)
SLIDE 11
Safe abortion services provided
Market share
Unsafe abortions averted
(safe abortions x probability unsafe
abortion per safe abortion)
Maternal deaths
averted
(unsafe abortions x
unsafe abortion
mortality)
DALYs averted
(years lost due to
maternal mortality; and
unsafe abortion related
maternal morbidity)
(safe abortions
provided/abortions
nationally)
Direct costs saved
to healthcare
system
(cost of PAC for women
needing treatment)
SLIDE 12
Limitations


Data availability
–
Infrequent
–
Often regional rather than national
Reliant on assumptions: LAPM discontinuation rates, mortality rates,
pregnancy rates, method failure rates, etc.

Some rates/coefficients held constant over time due to limited
data (i.e. unmet need, abortion rates)

Emphasise that all results are only estimates and don’t
measure real life
SLIDE 13
Types of results
SLIDE 14
Timeframe


Annual: snapshot of impact that happen in a given year
–
impact from services provided in that year
–
impact from LAPM clients who received services in past
years, but, are estimated to still be using the method
Service lifespan: impacts that will happen over multiple future
years, from services provided in a given year
–
Similar to CYP-based impacts from old model
–
Number of future years depends on mix of LAPMs
SLIDE 15
Timeframe example
2005
Annual impact
in 2008 =
impacts from
these users
2006
2007
2008
2009
2006
2007
2008
2009
2010
2007
2008
2009
2010
Service lifespan
impacts in 2006 =
impacts to women
who received
services in 2006
over multiple years
2011
SLIDE 16
Historic services?

Annual impacts can be looked at:
–
including historic services: includes services provided prior to the start
year of your trend
–
without historic services: does not include services provided prior to the
start year of your trend
Trends from 2007 - 2010
2005
2006
2007
2008
2009
2006
2007
2008
2009
2010
2007
2008
2009
2010
Include = full picture
Without = isolate impact of project
2011
SLIDE 17
2008
2009
2010
2011
2012
What to count (1)

Total: counts the full impact of your programme
–

Includes impacts to all clients, regardless of their previous use
Incremental: counts impacts for which your organisation is
responsible
–
or: impacts that would not exist without us, unless other providers
increased their service levels
–
Excludes impacts to clients who were “taken” from another provider
SLIDE 18
Total v incremental (1)
Not using any
modern method
Your clinic
14
7
Other
provider
Number of clients
doubled…
SLIDE 19
Total v incremental (2)
Not using any
modern method
Your
clinic
Other
provider
Total users = 14
Incremental users = 9
SLIDE 20
What to count (2)
Before:
41% market share
17 women
using FP
After:
73% market share
Despite
large
increase in
market
share,
almost no
change to
the “market”
19 women
using FP
SLIDE 21
What to count (3)

Growing the market = increasing CPR

But. . .

Don’t forget about population growth!
Must reach
more and
more
women just
to maintain
CPR. . .
SLIDE 22
What to count (4)


Total and incremental impacts do not take any of this into
account.
–
These estimates include impacts that were already being averted,
because women were already using FP
–
You could increase your total/incremental impact without having
any impact at the national level
National contribution (annual only): counts impacts which
contribute to a national reduction in burden
–
Only includes impacts to women increasing CPR
SLIDE 23
National contribution
Had no women been using FP = more unintended pregnancies = more unsafe abortions
Unsafe abortions continuing to be averted
Already
being
averted
“national contribution”– offset population growth, then reach
adopters
Actual number of unsafe abortions
2010
2011
SLIDE 24
Increasing CPR
1.
Maintain number
2.
maintain percent (due to
population growth- adopters only)
Accounting for historic:
Including- baseline = year before trend starts
Increase CPR
Maintain CPR
contribution
Without- baseline = 0
3.
increase CPR (adopters only)
Maintain
user numbers
SLIDE 25
Key assumptions
Several key assumptions for national contributions:
–
All other providers at least maintain their contributions
– If not, your increase will offset declines from other provides
–
Your client profile is accurate
–
There are no unexpected changes at national level
– i.e. sudden increase in MMR
SLIDE 26
Client profile

Adopters: women who were not using a modern FP method
before coming to your programme

Continuers: women who were already using a modern method,
and received the method from your programme

Provider changers: women who were already using a modern
method, and received the method from another provider
MSI uses exit interviews to get this information
SLIDE 27
Which timeframe?

Annual: useful to give a ‘snapshot’ of your impact– works best
to look an organisation as whole, or entire regions or country
programmes

Service lifespan: shows all the impacts that will come from an
investment– works best for donors and to look at impacts of
particular projects
SLIDE 28
Which type of impact?

Total: most simple (and largest!) result, gives overall picture of
the impact of our work; does not require client profile data
When possible, in addition to total you may wish to show:

Incremental: gives overall picture of the impact of your work,
with some consideration for who we are reaching (i.e. excludes
impacts taken from other providers)

National reduction: shows meaningful result about your role in
changing the national picture
SLIDE 29
When in doubt…


Use total service lifespan impacts
–
Easiest
–
Closest to old model
If you have client profile data– can get more ‘meaningful’
estimates (i.e. incremental, national reduction)– eventually
moving in this direction, but, not there yet. . .
SLIDE 30
Remember!

Always use the word “estimated”– these are modelled,
and not measures of real life

For service lifespan results, always use the future tense
(“will be averted”)– these impacts are spread over
multiple future years
SLIDE 31
Impact 2 demo and activity
SLIDE 32
Tip: Don’t forget to enable macros!
SLIDE 33
SLIDE 34
Tip: you can edit/change any yellow box!
SLIDE 35
Tip: print this out for a quick reference of data and sources
SLIDE 36
Tip: You must enter service data by method
SLIDE 37
Tip: a low default is set for the client profile, change if you have
this data
Tip: only need to enter your costs to see “cost per xxx”
SLIDE 38
Tip: all graphs and tables can be cut and paste into
presentations, reports, etc.
SLIDE 39
Exploring Impact 2
Can you create the following?

Graph of annual (including historic services) maternal deaths and child
deaths averted in Tanzania from 2005 to 2010.

Table of life-span demographic impacts (pregnancies, births, and
abortions averted) in Myanmar from 2007 to 2011

Bar chart of the number of using an implant or IUD (including historic
services) from Marie Stopes Cambodia in from 2001 to 2011

Pie chart of modern market share (including historic services) in
Pakistan in 2010
SLIDE 40
Using results
SLIDE 41
Using results in your work

Proposals to donors

Tracking progress over time

Setting realistic goals

Communicating with Governments and donors

Decision making– compare different situations
SLIDE 42
Helpful handouts
SLIDE 43
More information

Download Impact 2: http://www.mariestopes.org/Resources/Tools.aspx

Additional trainings being developed on:

–
How to do goal setting
–
Using Impact 2 for proposal writing
–
Using Impact 2 for planning
–
How to generate multi-country results
Other needs? Please get in touch with MSI’s Impact Analysis
Team: [email protected]
SLIDE 44