Transcript Economic tools to evaluate social science programs Bidisha Mandal

ECONOMIC TOOLS TO EVALUATE
SOCIAL SCIENCE PROGRAMS
BIDISHA MANDAL
SCHOOL OF ECONOMIC SCIENCES
WASHINGTON STATE UNIVERSITY
November 16, 2011
WSAC, 2011 Extension Directors Conference
Overview

Economic tools





Why evaluate?
What are economic tools, economic analyses?
When to evaluate?
How to evaluate?
Case studies from Health Extension




Spokane Public schools: Nutrition services intervention for middle
school students
Strengthening Families Program: Substance abuse prevention for
adolescents
Providence health care services: Transitional care model to
reduce preventable hospital readmissions
Supplemental Nutrition Assistance Program Education: University
of Idaho Extension
Magnitude of the Problem

Primary prevention
 Health

promotion
Secondary prevention
 Screening,

diagnosis, therapies
Tertiary prevention
 Treatment
to prevent or postpone complications
A Crucial Difference

Program effectiveness
 Outcome
oriented
 Directly links the intervention with health outcomes of
interest

Program evaluation
 Ways
to maximize the intended impact with available
resources, or
 Ways to obtain a particular impact with as little
resources as possible
Importance of Evaluation

Resource constraints
To eliminate or reduce waste
 Evidence of return on investment


Ideally…
Inform decisions
 Help make choices about future allocations


Example: Health-care system
Getting value for money is a widely accepted and
legitimate goal
 Quality medical care in part translates into potentially
expensive demands for new drugs and technology
 Pressure to improve efficiency, make trade-offs, and
develop incentive systems for patients and physicians while
holding down healthcare costs

Address Two Questions

What works?


What works best?


Multiple interventions could work
Identify the one intervention that provides the greatest bang for
the buck!
Example: Increased prevalence of diabetes among
adults

Strategies and reasonable alternatives
Physical activity – Fitness programs (worksite, community, less
TV/computer time)
 Diet – cooking programs, education program to change food
consumption behavior


What is the objective
Increased physical activity? Short-term
 Reduced risk of diabetes? Long-term

Economic Analysis

Evaluation is part of the program design and
planning
 There
are always competing use of resources
 Frame the study in order to consider opportunity costs for
each of our choices


Identify, measure, value and compare the costs and
consequences of alternative prevention strategies
Quantitative and analytic methods
 Cost-benefit
analysis
 Cost-effectiveness analysis
 Cost-utility analysis
Identify the Risks Factors




Define the target population for the intervention
Define the problem or question, and magnitude of
impact
Define the information needs of the target
population in reference to the program or
intervention
These steps will
 Influence
the types of benefits and costs to be included
 Help to determine which analytic method is most
appropriate
Identify the Intervention(s)





Indicate clearly the preventive strategies under
consideration, including baseline if any
Specify perspective of the program and analysis
Limit perspectives to those relevant to the study
Define relevant time frame in which program will be
delivered
Determine how far into the future costs and effects
that accrue from the intervention will be considered
Background on the Intervention

Can it work?



Will it work?





Efficacy
Degree to which intervention strategies can work under ideal conditions,
with carefully selected participants, and optimal resources.
Example: Randomized controlled trials
Effectiveness
How well these strategies actually work in community settings
Demonstrates real-world effectiveness under practical resource constraints
Effectiveness is likely to be lower than efficacy
What are the benefits and costs of the intervention?



Units of measurement
How do benefits compare with costs?
What additional benefits could be obtained with additional resources?
Identify the Methods



Determine the analytic methods for decision-making
The choice will depend on the policy question, the
outcomes of interest, and the availability of data
Determine whether analysis is to be marginal or
incremental
 Marginal
analysis: Examines the effect of expanding or
contracting an intervention
 Incremental analysis: Compares the effects of
alternative programs
Identify the Outcomes

Identify the relevant costs
 Program
costs
 Productivity losses

Identify the relevant outcomes
 Number


and nature of health outcomes
Specify the discount rate or time preference for
monetary and non-monetary costs
Identify sources of uncertainty and plan sensitivity
analysis
The Role of Discount Rate





Individuals generally weight costs and benefits in the near future more
heavily than in the distant future
This applies to the valuation of capital and investments and to health
outcomes
Societal preference is for health benefits received today versus health
benefits received in the future
Using an appropriate discount rate in an economic analysis allows
 Adjusting the value of receiving benefits today versus in the future or
of incurring costs today versus in the future
 Makes benefits and costs comparable over time
Discount rate is selected based on the study perspective
 Social or private or individual
The Role of Uncertainty

Precise estimates of costs and benefits/effects are
often not available
 Limited
literature
 Different population settings


Important to list all assumptions upon which
estimation is dependent
Perform sensitivity analyses
 How
will result of evaluation change if the assumptions
change?
Evaluation Tools



3 most commonly used techniques
 Cost-benefit analysis (CBA)
 Cost-effectiveness analysis (CEA)
 Cost-utility analysis (CUA)
Each method
 Allows comparison of different intervention strategies
 Calculates resources consumed and outputs generated
 Requires quite similar cost analysis
 Assessment of outcomes, both benefits and harms (negative
benefits)
Scope of analysis determines the appropriate analytic method
Cost-Benefit Analysis


All costs and benefits valued in dollars
Costs include
Cost of program
 Cost to participants - out-of-pocket expenses, productivity
losses, travel time, child care, intangible costs (pain,
suffering)


Benefits include
All types of beneficial and harmful health outcomes,
whether intended or not
 Have to be valued in monetary terms


CBA is well suited to comparisons with interventions that
include cross-sector considerations

Housing, education, transportation interventions
Cost-Effectiveness Analysis


Usually examines direct medical, non-medical, and
productivity costs
Compares costs with outcomes in standard health
units
 Example:

costs per case averted
Most suitable when comparing interventions that
have similar health outcomes
Cost-Utility Analysis

Modified version of CEA
 Compares
direct medical and non-medical costs with
health outcomes converted to a standard health unit,
often a quality adjusted life year (QALY) combining
both mortality and morbidity

Often used to compare health intervention which
have different type of health outcomes
CASE STUDIES
Spokane Public Schools:
Objective


Spokane public schools’ lunch program

With Doug Wordell, Ruth Bindler, Kenn Daratha, Sue Butkus

Intervention program included reducing vending machine beverages, limiting ala
carte offerings, and adding seasonal fruits and vegetables to student lunch
menus

Compare pre-program and post-program behavior
Objective


My involvement


If there were associations between an altered school food environment and
food choices of middle school students both in and outside of school
Retroactive, after program was delivered and surveys were conducted
My role

Analyze survey data

Determine food behavior change
Spokane Public Schools:
Method & Results

Results
 Healthful
modifications in the school food environment
associated with some positive food behaviors

The cost of conducting the intervention was
approximately $24,000/year
 Lost
ala carte sales, loss in vending machine sales
 More expenditure on produce

In this study, is it possible to show the benefits of
improved food choices outweigh the costs?
 NO
- Related data was not collected for CBA
Spokane Public Schools:
What did I learn?

Difference between outcomes and impacts
 In
this study, there is no way to link improvement in
behavior to improvement in health (short-term or longterm)
 Even if they are positively related, we have no
quantitative measure for the benefits

Economic analysis is not always possible
 Unless
evaluation is part of the program design
Strengthening Families Program:
Objective

Strengthening Families Program (SFP) for Parents and Youth
10-14 years







With Laura Hill, Robby Rosenman, Ron Mittelhammer
Voluntary, family-based intervention
Designed to discourage future substance abuse among adolescents and
youth
Compare pre-program and post-program behavior
Randomized clinical trials (RCT) have shown SFP to be costeffective and that benefits outweigh costs
How does SFP’s impact in community dissemination compare to
results from RCTs?
My involvement

Retroactive, after programs were delivered and surveys were conducted
Strengthening Families Program:
Method & Results

Community dissemination has many practical issues



CBA, CEA and CUA not possible


Do not have necessary data
But, have data to determine which factors in community
dissemination of the program are different



Variation in program delivery across counties, states
Data not recorded systematically or consistently
Attrition – high incompletion rates, some people come to the sessions but do
not respond to surveys
Self-selection – more motivated parents are more likely to attend SFP
Results

People who come to the sessions but do not respond to surveys have lower
self-assessment scores
Strengthening Families Program:
What did I learn?

Validity of assumptions
 Does
sample match population?
 Differential dropouts
 Are facilitators similar?
 Some sessions have additional orientation session
 Language of delivery
Providence Hospital Transitional Care:
Objective

Transitional care model




With Cindy Corbett
Innovative model to improve and synchronize hospital discharge
planning and deliver core transitional care intervention to patients at
high risk for potentially preventable readmissions
Secondary objective: document barriers and facilitators of successful
delivery in different hospital environments
My involvement


Contacted during the planning process
But, resource constrained


Unable to collect/record some necessary data for a complete CEA
Future studies will look at CUA
Providence Hospital Transitional Care:
Method & Results

Cost-effective analysis of transitional care model


Have necessary costs


Cost of transitional care nurse, inpatient pharmacy, pharmacotherapy
clinic, administrative costs, home health care costs, hospital care costs
Have some necessary effects (in $)



Compared to patients not receiving transitional care
Hospital care avoided due to lower re-hospitalizations
Revenues from inpatient pharmacy, pharmacotherapy clinic not recorded
Result



Decrease in re-hospitalizations and ED visits
Total savings over 4 months = $55,752.34 (for ~ 100 patients)
Savings likely to increase over time since some of the cost items were
fixed costs, and some revenues were unknown
Providence Hospital Transitional Care:
What did I learn?

Know your audience
 Who
is using the results?
 What is the study perspective

Results of this study to be used by hospital
administrators
 CBA
not appropriate
Supplemental Nutrition Assistance Program:
Objective


University of Idaho Extension, Nutrition and Food Safety

With Shelly Johnson, Joey Peutz and others

Follow Virginia Tech report for CBA (1996) to calculate the costs and benefits of
UI’s Supplemental Nutrition Assistance Program Education (delivered in Coeur
d’ALene)
My role


A complete CBA with sensitivity analysis
My involvement:

Quite proactive

Involved in program design – control and intervention

Introduce new questions to pre- and post-program surveys to improve CBA
Supplemental Nutrition Assistance Program:
Method & Results

Costs and benefits
 Collect
up-to-date information on costs of health
conditions/diseases averted due to improvement in
nutritional intake and health behavior
 Cost of program delivery (compared to control group)

Results
 Control
and intervention will take place in
February/March 2012
 Analysis results expected middle of next year
Supplemental Nutrition Assistance Program:
What did I learn?

Practical issues
 Participant
enrollment
 Differences in incentives to participants in control and
intervention groups
 Sample size for robust results
 Are cost data in literature suitable for Idaho
population?
Challenges





Time consuming process
Uncertain monetary values in CBA
Uncertain QALY values in CUA
Comparison of results under different situations
Validity of assumptions
Other questions? Contact me!
[email protected]
509-335-7553
Reference

Haddix, A.C., Teutsch, S.M., Corso, P.S. (2003). “Prevention Effectiveness: A
Guide to Decision Analysis and Economic Evaluation”. Second Edition,
Oxford University Press.