Transcript Evaluation of Health Education and Risk Reduction Program

Evidence-Based Public Health: A
Course in Chronic Disease
Prevention
MODULE 7:
Economic Evaluation
Kathy Gillespie
March 2013
“An economist is a man who
states the obvious in terms of
the incomprehensible.”
Alfred A. Knopf
2
Kathleen N. Gillespie, PhD
314-977-8147
[email protected]
Learning Objectives
Be able to answer the following questions:
1. Why is economic evaluation needed?
2. What is economic evaluation? Are
there different types?
3. When is an economic evaluation
necessary or useful?
4
Learning Objectives
More questions…
4. How is an economic evaluation done?
5. Who can help me with an economic
evaluation?
6. Where can I find information about
economic evaluations?
5
Where This Module Fits
The green
and blue
sections,
primarily
6
Next Module: Logic Model
EE helps
with costs
and
objectives
7
Why?
Why EE?
• Evidence-based practice requires knowing
– Does it work?
– At what cost?
• EE starts after efficacy has been
determined.
Want to promote interventions that work
at a reasonable cost, i.e. that provide
“good value for the money”
9
Examples
• Increasing Physical Activity: Evidence
shows that the following will work (The
Community Guide)
– Community wide campaigns
– School-based physical education
– Street-scale urban design and land use
policies and practices
– Social support interventions in community
settings
• Which to invest in? or a mix?
10
Examples, cont.
• Budget cuts: A department’s budget must
be cut by 5%. How to do this so as to
minimize the effect on the public’s health?
• Grant awards: A number of applications
have been received for a limited pool of
grant dollars. How to select the winners?
11
Need to Weigh Costs and
Benefits
• A scale can compare
apples and oranges
because both are
measured by weight
• EE provides the scale
– an agreed upon way
to measure health
benefits and costs
12
What?
What Is EE?
• EE examines the costs and benefits
of alternative programs to inform
decisions about the allocation of
scarce resources using an
established set of economic tools.
• There are four types of EE.
14
Types of Economic Evaluation
•
•
•
•
Cost-benefit (CBA)
Cost-utility (CUA)
Cost-effectiveness (CEA
Cost-minimization
The number of projects that can be compared
declines as we move down the list.
15
EE and Business
• EE methods are closely related to several
business methods
– Return on Investment, or ROI
– Make or buy decision
– A business plan
• Investment decision making
16
It Helps to Have a Map
• What is the
intervention?
• Who is the relevant
group?
• How long should the
group be followed?
• What can happen at
each time period?
• DECISION ANALYSIS
can provide the map.
17
Example Decision Tree
Dowding D and Thompson C, Using decision analysis to integrate evidence
into decision making, Evid Based Nursing, 2009 12: 102-104. 18
Components of Economic
Evaluation in Public Health
Economic
Inputs
Public
Health
Program
Costs
Direct Costs
Indirect Costs
Averted Treatment Costs
Health
Outputs
Benefits
YOLS
QALYS
Dollars
Always compare a new program to some alternative.
19
Components of Economic
Evaluation in Public Health
New Program A
Benefits A
Costs A
Choice
Direct
Indirect
Averted Treatment
Costs
Costs B
Program B
•Comparison program
•May be new or old
•Could be ‘doing nothing’
YOLS
QALYS
Dollars
Benefits B
20
The Product of an EE
Incremental
Costs
EE ratio =
Incremental
Benefits
21
The EE Ratio
• The EE ratio is often called the
Incremental Cost Effectiveness Ratio, or
ICER
• Emphasizes that we are comparing 2 or
more interventions
22
Dimensions of Economic Evaluation
Type
Inputs
Outputs
Perspective
CMA
Dollars
Natural Units Organization
ROI
Dollars
Dollars
Organization
CBA
Dollars
Dollars
Society
CEA
Dollars
Natural Units Society
CUA
Dollars
Utilities
Society
23
Every EE is Conducted from
a Particular Perspective
• Perspective (or viewpoint)
determines whose interests are
paramount in the evaluation
• Perspective determines what costs
and consequences are considered
• Perspective determines how the
results are interpreted
24
Determining the Perspective
•
•
•
•
Single provider’s practice
FQHC
Healthcare system
Insurer – public (Medicaid) or private
(BCBS)
• State agency
• Federal program (CDC, Medicare)
• Society
25
Example: Increasing Physical
Activity
• Community wide
campaigns
• Suppose a
campaign would cost
$150,000 and yield
an improvement of
1,000 qualityadjusted life years
• ICER = $150/QALY
• Social support
interventions in
community settings
• Suppose this would
cost $200,000 and
yield 2,000 qualityadjusted life years
• ICER = $100/QALY
26
Some Assumptions of Economics
• Resources are scarce.
• Human wants are unlimited, and more is
(almost) always preferred to less.
• If there is perfect competition, the market
will do an efficient job of allocating scarce
resources to maximize profits and
satisfaction. Note: society may not
consider this outcome equitable.
27
Some Assumptions of Economics
• If the market works
well, then there are
only 2 parties to each
trade – a buyer and a
seller – and they bear
all the costs and reap
all the benefits of the
trade.
28
When?
When Is EE Needed?
• EE needed if there is market failure
(inefficiency).
– Perfect competition not present.
– Could occur naturally or the market could be
‘broken’.
• EE helpful if there are concerns about
inequity.
30
Inefficiency Reasons for EE
• When the market fails, or is broken.
– Complex decisions with high stakes.
– Little or no price information available.
– High information costs to find out about the
product.
• Markets involving externalities or public
goods.
– Externalities are spillover effects from trades.
– Public goods are non-excludable.
31
When is EE needed? Really?
• Practically speaking, may be mandated or
requested by funding agency.
• Useful when comparing options internally.
• Useful when justifying programs externally.
32
Inequity Reasons for EE
• Social or government decision-making
processes.
– If the distribution of goods and services that
would emerge from the market is considered
inequitable, then society can legislate,
regulate, or tax/subsidize.
– EE measures the current or desired
outcomes.
33
Public Health Requires EE
• Provides services when the market fails
– WIC program, health clinics
• Complex decisions, high stakes - health
and life
• Provides services with external benefits
– Immunization programs, sanitation,
inspections
– Tobacco control programs
34
Public Health Requires EE
• Provides public goods and services
– Health monitoring and assessment
– Social marketing and public information
campaigns
• Access to health and equity are often
concerns
35
How?
The Product of an EE
Incremental
Costs
EE ratio =
Incremental
Benefits
37
Possible Outcomes of an EE
Aggregate Health Benefits
Quadrant IV
Saves money,
Improves health
Quadrant I
Costs money,
Improves health
Aggregate Costs
Saves money,
Worsens health
Quadrant III
Costs money,
Worsens health
Quadrant II
38
Measuring Costs
• All economic evaluation methods require
listing and measurement of costs.
• One tricky point.
– Mathematically, a benefit can be a negative
cost, and a cost can be a negative benefit.
39
Negative Costs and Benefits
• Whether some items are negative costs or
benefits depends on the method.
• Makes a difference because we are
constructing a ratio.
• General idea: all items falling on the
health system are costs, positive or
negative.
40
Types of Costs
• Direct (or program) costs
• Indirect costs
– Time and travel costs to participants
– Averted productivity losses (a negative cost)?
– Cost of treatment during gained life
expectancy ?
• Averted treatment costs (a negative cost)
41
Direct Costs
All costs of the intervention
• Labor
• Supplies
• Rent
• Utilities
• Costs of treating side effects of the
intervention, if any
42
Indirect Costs
Also called spillover costs.
• Time and travel costs to participants.
• Costs to parties outside controlling program or
agency.
• Averted productivity losses (a negative cost)?
• Cost of treatment during gained life expectancy?
43
Averted Productivity Losses
• The present value of future wages gained.
– The intervention increases the worker’s
longevity and/or reduces disability.
• Used in cost-benefit analyses.
• Not included in cost-utility analyses.
– Double counting of the benefit.
44
Costs of Treatment During
Gained Life Expectancy
• If the intervention saves a life, that person will
die of something else later on. What are his/her
expected total medical expenditures in the
added life years? A controversial cost.
• Pro: these are expenses that would be avoided
in the absence of the intervention, and therefore
should be counted.
• Con: health expenses should not be singled out
for inclusion.
45
Averted Treatment Costs
• The averted costs to society of treating
persons for the disease are subtracted
from total costs.
• Included in the cost side because they
impact the health care budget.
– A negative cost, not a benefit.
46
Measuring Benefits
• Benefits can be measured in many ways
• Different units of measurement
– Dollars
– Years of life saved
– Quality adjusted years of life gained
– Specific health outcome
• Different time frames for the outcome
– Intermediate outcome
– Final health outcomes
47
Measuring Benefits
Prevention poses special problems:
• Benefits may be:
– small and cumulative.
– complex and interrelated.
• Are difficult to measure or validate without
large samples of panel data observed over
long time periods.
• Consequently, often focus on intermediate
outcomes.
48
Measuring Benefits
Prevention poses special problems:
• Aim is often to change behavior among
more people.
– Increases costs in the short term.
• Community versus individual approach.
– Economies of scale possible.
• Provide walking trails; tax gasoline.
– Individual interventions often more costly.
49
How to Put it Together?
• Have costs and
benefits, how to
put them together?
• Depends on the
type of EE.
50
Definition
Cost-minimization Analysis
• Populations served must be identical
• Outputs for all alternatives must be
identical
• Timeframes must be the same
• Search for alternative that yields the
lowest total cost
51
CMA Example
• Deliver dental sealants to school children
• Choose staffing that minimizes program
cost per student
Cost per Student
30
25
Cost
20
15
10
5
0
1
2
3
4
Supervision Level
No supervision
General supervision
Indirect supervision
Direct supervision
Scherrer, CR, et al, Public Health Sealant Delivery Programs, Medical Decision Making, Nov-Dec 2007
52
Definition
Return on Investment (ROI)
or Business Case
• Answers question: Will an activity pay for
itself?
• Analysis of net discounted cash flow to entity
paying for an intervention
• Typically short-term (1-5 years)
• Perspective is key to determining costs
and consequences to include in analysis
53
Why develop a business case?
• A positive business case increases
the probability that a prevention
intervention will be sustained
• ROI analyses are most easily
understood by administrators and
budget officers
Brownson CA and Kilpatrick KE, Building a Business Case for Diabetes Self
Management: A Handbook for Program Managers, Robert Wood Johnson
Foundation, 2008 (available at www.diabetesinitiative.org/resources)
54
Return on Investment Example
• CareOregon (Portland): initiated case
management for high-risk Medicaid
enrollees with multiple comorbidities
• Study design: before-after design
compared to baseline
• Investment costs: $526,290
• Discounted savings: $6,423,776
• Return on investment: 12.21: 1
Greene SB, et al, Searching for a business case for quality in Medicaid
managed care, Health Care Management Review, 2008, 33(4), 350-360.
55
Definition
Cost-benefit Analysis
• Cost-benefit analysis values both the costs
and benefits of a program, project, or
treatment in monetary terms.
• Result of analysis
– The net benefit of the project (e.g. benefits
minus costs, or $12,000) OR
– The ratio of costs divided by benefits (e.g. ½).
56
Cost Benefit Example
• Intervention: Neighborhood-based
program to prevent teen pregnancy
• Program costs: $9,386 per participant per
year
• Effects: reduced teen pregnancy from
94/1000 to 40/1000
• Cost: $26,142 per birth averted
• Saved: $81,256 society costs/birth averted
Rosenthal MS, et al., Economic Evaluation of a Comprehensive Teenage
Pregnancy Prevention Program, Am J Prev Med 2009;37(6s1)
57
Example ICBR
ICB ratio = $26,142
= .322
$81,256
Smaller ratio is better
58
Cost-benefit Analysis
• Gold Standard for EE.
• Allows for economic comparison of widely
disparate publicly funded programs in
such areas as health, education, and the
environment.
• Problem: valuing a life in monetary terms.
59
Definition
Cost-effectiveness Analysis
Cost-effectiveness analysis measures the
benefits of a program in naturally
occurring health units, such as lives
saved.
Example of a study result: $10,000 per
life saved.
60
CEA Example
• Intervention: Smoking cessation
•
•
•
program in the workplace
Effect measured: Number of people
who quit smoking
ICER = $596 cost per additional quitter
This cost was less than “high intensity”
interventions by clinicians
Tanaka H et al. Effectiveness of low-intensity intra-workplace intervention
on smoking cessation in Japanese employees: a three year
interventionTrial, J. Occup Health 2006; 48(3):175-82.
61
Cost-effectiveness Analysis
• CEA formerly most common form of EE
conducted in health arena.
• Limited in its ability to report outcomes.
– Often, there are multiple outcomes.
• Limited in its ability to compare
interventions.
62
Three Projects Example
• A nursing program for newborns and their
parents costs $50,000 per year and serves 50
infants with high-risk conditions
• A screening program that visits local malls and
community centers costs $15,000 and provides
information to about 5,000 persons
• A vaccination program costs $100,000 and
provides vaccinations to 20,000 area children
63
Three Projects Example
and CEA
• Nurses for
infants
• Costeffectiveness
ratio is
$50,000/50
infants or
$1,000 per
infant
• Screening
program
• Costeffectiveness
ratio is
$15,000/5000 or
$3 per attendee
•
•
Vaccination
program
Costeffectiveness
ratio is
$100,000
/20,000 or $5
per child
64
Definition
Cost-utility Analysis
Cost-utility analysis compares the costs of
different programs, projects, or treatments
with their outcomes measured in “utility based
units” which are related to a person’s health
related quality of life.
Example of a study result: $10,000 per qualityadjusted life year, or $10,000/QALY.
65
Definition
Cost-utility Analysis
• Becoming the most common form of
analysis.
• Widely used in Britain and Canada.
• Allows comparison of many projects with
health-related outcomes.
• Often called cost-effectiveness analysis;
closely related.
66
CUA Formula
Cost utility
ratio
(direct costs
+ indirect costs
- averted treatment costs)
=
Quality adjusted
life years
Output of CUA is ‘cost per QALY’
67
Cost-utility Outcomes
• Outputs are measured in terms of a preferencebased outcome measure.
– Quality Adjusted Life Years (QALYS): the
number of years at full health that would be
valued equivalently to a given number of
years of life experienced with a disease or
disability.
– Other measures are available, this is the most
common.
68
Why Measure Quality of Life?
• Health care outcomes are
multidimensional.
- Length of life, or mortality.
- Quality of life, or morbidity.
• Allows for more than one disease or
•
health problem to be compared.
Considers the individual’s preference for
health outcomes.
69
How QALYS Are Measured
• One year of life in excellent health is assigned
a value of 1; death is given a value of 0.
• A health state is described and its utility or
quality elicited.
• The value assigned to quality of life is referred
to as health utility.
0
.5
1
70
An example of estimated health utility
values for diabetes
0
0.2
0.4
0.6
0.8
Diabetes (Diet &
Exercise)
0.69
Diabetes (Oral Agent)
0.67
Diabetes with
Neuropathy
0.60
Diabetes with Neuro and
High BP
Diabetes with Neuro,
High BP, and Stroke
Coffey et al. 2002 Diabetes Care
0.59
0.52
71
QALYs Gained from an intervention
with program
Q
y
lit
ua
j
ad
e
lif
without program
d
te
us
s
ar
ye
ed
in
ga
Health related quality of life
optimal
1
health
0
Duration (years)
death
without the
program
death with
the program
72
Problems With QALYs
• Subjective and difficult to measure.
• Whose QALYs should count? A
representative sample of the
population or the affected group?
• QALY scales will differ depending
upon factors such as age, gender
73
Stop Smoking Example
• After considering your community and its
public health problems, your organization
has identified smoking as a problem you
can address.
• Two interventions are being considered;
only one can be done.
74
Stop Smoking Example Using CUA
• Multicomponent
interventions that
include patient
telephone support
• Suppose this would
cost $150,000 and yield
1,000 quality-adjusted
life years
• Healthcare provider
reminder system
• Suppose this would
cost $200,000 and yield
2,000 quality-adjusted
life years
ICER for B vs. A =
$50,000/1000 QALYs
or $50/QALY
75
Cost Utility Example
• Intervention: Diabetes self-management
programs in primary care settings
• Program costs: $866 per participant per
year
• Effects: 87.5% benefited, A1c -.5%, total
cholesterol. -10%
• ICER: $39,563/QALY saved
Brownson CA, et al., Cost-effectiveness of Diabetes Self-management
Programs in Community Primary Care Settings, Diabetes Ed, v. 35, no.5,
2009
76
How Do I Know If It’s CostEffective?
• Are we almost there
yet?
• Once I have the ratio,
how do I know if it’s
too high, too low, or
just right?
77
Is It Worth It?
• Results can be used internally or
externally
– To rank programs internally
– To argue for external support
• Intermediate results, such as productivity
gains, can be highlighted for some
stakeholders
78
Using Results Internally
• For internal use
– Rank options from lowest to highest ratio.
– Start spending on lowest ratio, move on until
the money is exhausted
– What have we spent before?
79
Using Results Externally
• The ICER is
compared to a
threshold value
• Suggested U.S.
threshold is
$50,000 to
$100,000 per
QALY at minimum
80
Exercise
81
Two Other Important Features
• Before study is complete,
should consider
– Discounting
– Sensitivity analysis
82
Discounting
• Time value of money.
– A dollar in the future will be worth less than a
dollar in the present.
• Needed to compare present value and future
value of benefits from project.
• Recommend a discount rate of 3-5%.
• Discount rate chosen can affect results.
83
Discounting in Practice
• Suppose you want to find the present
value of $100 received in 10 years.
Several options for finding this.
• Tables in finance and accounting books.
• Excel or other spreadsheet programs.
• The internet: Google “present value
calculator” and several pop up.
84
Sensitivity Analysis
• EE is based on estimates and
assumptions – want to vary them and see
how robust the results are.
• Variables to test in sensitivity analysis
should include the “top 3” or “top 5”
“wobbliest” assumptions.
• Analysis should be redone varying the
assumptions.
85
EE Results Reporting
• The ICER(s) should be reported for the
intervention(s) studied for the base case.
• ICERs may be reported for subgroups of
the population.
• ICERs should be reported for different
assumptions (sensitivity analysis).
86
How is This Done in Practice?
• Back of the envelope EE
– Draw a decision tree, follow the costs and
benefits, form the ratio
– No economist required
• Primary data collection
– Often alongside an intervention
– Collect cost and benefit data
– Economist works with team
87
How is This Done in Practice?
• Synthetic EE
– Follow a hypothetical cohort of persons
– Use the literature to obtain cost and benefit
estimates
– Rely on modeling and computer analysis
– Economist leads the team
88
Who?
Who Does EE?
• Some economists are trained to do EE.
• Most EEs require a multi-disciplinary team.
• Few public health agencies have the
resources to have a staff economist – so
think collaboratively – local colleges.
90
Where?
Web Resources
• http://www.thecommunityguide.org/econ/defa
ult.htm The economics section of The
Community Guide, it includes systematic
reviews of EEs for recommended activities.
• http://www.tufts-nemc.org/cearegistry/ The
Cost Effectiveness Analysis Registry at Tufts
– New England Medical Center. Includes EE
results and QALY estimates for numerous
conditions.
92
Web Resources
• http://www3.interscience.wiley.com/cgibin/mrwhome/106568753/HOME The
Cochrane library, a reliable source of
information on the effects of interventions in
health care. Economic evaluations are
available at
• http://www.mrw.interscience.wiley.com/cochr
ane/cochrane_cleed_articles_fs.html or by
clicking from the web site above.
93
Web Resources
• http://www.york.ac.uk/inst/crd/ The Centre for
Reviews and Dissemination at the University of
York. Includes several searchable databases; of
particular interest is the NHS Economic Evaluation
Database (NHS EED), which provides article
summaries, similar to the Community Guide.
• http://www.nice.org.uk The National Institute for
Clinical Excellence. A more general database (the
prior website is reachable through this site, for
example) it includes RSS feed capabilities.
94
NHS – EED
• Provides summaries of EE articles
– Follows a standard format
– Summary is about 2 pages or less
– Can then click on full report, which may be
several pages
• Easily searchable
– Good accessible instructions for searching
95
Sample NHS EED Results
Topic
Hits
Smoking cessation
146
Diabetes prevention
279
Obesity
208
Obesity prevention
40
Physical activity
106
“Physical activity”
44
96
Reference Books
• Drummond MF, Sculpher MJ, Torrance GW,
O'Brien BJ, Stoddart GL. Methods for the
economic evaluation of health care
programmes. Third edition. Oxford: Oxford
University Press; 2005.
• Muennig P (Contributing Editor, Kahn K).
Designing and Conducting Cost-Effectiveness
Analysis in Health and Medicine. San Francisco:
Jossey-Bass, 2002. (2nd edition forthcoming).
97
Search Tips
• General strategy: search for the
intervention of interest AND economic
evaluation
• Often easier to run 2 searches – 1 for
intervention and 1 for economic
evaluation, and then combine them
98
Search Tips
• If you want any type of economic
evaluation, search for ‘cost-benefit’ OR
‘cost effectiveness’ OR ‘cost utility’
– “cost-effective” is a buzzword that will net a lot
of articles!
• Articles in intervention journals will be
more verbose on the EE and terse on the
intervention; articles in economic journals
will be the reverse.
99
Economic Journals With an EE
Focus
• Health Economics
• Journal of Health Economics
• Cost Effectiveness and Resource
Allocation
• Health Technology Assessment
• Applied Health Economics and Health
Policy
• Value in Health
100
Summary
•
•
•
Economic evaluation is the comparison of costs
and benefits to determine the most efficient
allocation of scarce resources.
Economic evaluations can use existing or new
information and can provide a reliable tool for
decision making among public health
professionals and policy makers.
Though relatively sophisticated, the underlying
logic and structure of an economic evaluation can
be understood.
101
Summary
• Several challenges (e.g., inconsistent quality,
methodological issues, difficulties in
implementation) should be kept in mind when
considering the use of economic evaluations.
• Economic evaluation will be increasingly used,
especially in times of limited public health
resources, and practitioners must be able to
understand them so that they can argue for setting
appropriate public health priorities.
102
“It is our choices … that
show what we truly are, far
more than our abilities.”
J.K. Rowling, Harry Potter and The
Chamber of Secrets, 1999
103