Transcript Slide 1

New Research in
Economic Modeling
and Simulation
Greg Samsa PhD
Organization of the talk
3
questions
 How
does your work contribute to
economic and comparative effectiveness
modeling?
 What’s new in economic modeling and
simulation?
 What’s changing in how these models are
applied?
Question 1
 How
does your work contribute to
economic and comparative
effectiveness modeling?
 Short answer: You provide the
inputs
Question 2
 What’s
new in economic modeling
and simulation?
 Short answer: Very complex
models are now computationally
feasible – the debate is shifting
from what is possible to what is
desired
Question 3
 What’s
changing in how these
models are applied?
 Short answer: Decision makers are
starting to take these models more
seriously, and to embed them
within more general strategies for
learning
Background
 The
previous speakers have
provided definitions and examples
of economic models
 I’ll discuss “complex decision and
cost-effectiveness models” –
sufficiently complex to require
simulation to implement
Observation 1
A
CEA model is a peculiar thing
 It
is a counting machine intended to clarify
trade-offs among things that users value
(e.g., survival, quality of life, costs)
 Inputs are obtained from different sources
– “the model” isn’t something that can be
directly observed
 The usual principles of validation don’t
apply
– I’ll discuss what makes a good model later
Observation 2
A
dirty little secret:
A
CEA model is no stronger than its
weakest link
 Advocates focus on a model’s strengths,
but the weaknesses are also of importance
A personal confession
 “The
Duke Stroke Policy Model
combines data from the best
sources – natural history from
Framingham, costs from national
claims, utilities from a large survey
developed specifically for this
purpose…”
All true, but…
 Its
conclusions depend on how
survival, quality of life and costs
vary by disability level
 These
parameters were derived by
extrapolating from small studies of
inconsistent quality
 The user is essentially relying on (a) the
face validity of the parameter estimates;
and (b) sensitivity analyses
N=?
 Natural
history n=5,000
 Costs n=500,000
 Utilities n=1,500
 Natural
history, costs, utilities by disease
state – n is variable (e.g., only 100
hemorrhagic strokes)
 Impact of disability level n=20, mostly
interview rather than observation
Question 1
 How
does your work contribute to
economic and cost-effectiveness
models?
Data sources for model inputs
 Traditional
efficacy trials
 Effectiveness trials
 Registries
 Administrative data
 Surveys
 Observational studies
 Literature reviews
Study planning
 Design
studies to improve
estimates of parameters that are:
 Important
(e.g., using sensitivity analysis)
 Currently estimated with bias or
imprecision
Implication
 CEA
models can not only organize
thinking, decision making and
communication about a topic, but
can also be used to help set an
agenda for research
Question 2
 What’s
new in economic modeling
and simulation?
Bayesian approach
A
general approach to CEA
modeling
 All
parameter estimates are based on prior
distributions
 Ideally, correlations among parameters are
considered
 Ideally, these distributions reflect the
impact of covariates
 The output – posterior distributions –
reflects the impact of uncertainty
Example output
 “Uncertainty
in all the model
parameters was addressed using
(a) prior distributions; and (b)
resampling – in >95% of
replications of the simulation the
ICER was <$20,000/QALY…”
Advantages
 This
is a general, intellectually
coherent way of modeling
 Now computationally feasible
 Europeans and analysts like it
 A more sophisticated treatment of
uncertainty than 1- and multi-way
sensitivity analysis
Disadvantages
 Possible
loss of transparency
 Parameter estimates might not be
possible / practical to obtain
 Easy for the model to take on a life
of its own
What makes a good model?
Model structure focuses on core of the
issue
 As simple as possible, but not too
simple
 Model is transparent
 Model inputs can be collected at the
required level of precision / quality

Opinion
 Are
more structurally, technically
and computationally complex
models such as Bayesian CEA
models “good”?
 My
opinion: sometimes
Question 3
 What’s
changing in how CEA
models are applied?
Back in the day
 “Your
health care organization
should place our acute stroke drug
on the formulary because its ICER
indicates that it is good value for
the money…”
Problems
The decision maker doesn’t have the same
societal perspective as the analyst
 The analysis ignores silos
 Even with discounting, lifetime impact is less
important to the decision maker than short
term impacts
 Unless accompanied by a back-of-the
envelope calculation, the result isn’t
transparent

Example
A
back of the envelope model
 Suppose
that an acute stroke drug keeps 2
people per 100 out of nursing homes. If
they survive 3 years at $50,000 per year,
the excess cost is $300,000 per 100
patients, or $3,000 per patient. So long as
it costs less than $3,000, an acute stroke
treatment that is even marginally effective
is likely to be cost-effective as well.
Current trends
 With
calculations becoming less
burdensome, it is easier to
produced customized models
• (e.g., including only costs of interest to
the decision maker)
 Model
results are embedded within
more realistic frameworks such as
comparative effectiveness
Ideal framework
 Transparent
 Includes
as many of the elements
of interest to the decision maker as
possible
 CEA model is descriptive, not
prescriptive
Example
 As
an example of a formal decision
making process intended to satisfy
these criteria, I’ll describe how the
oncology clinics at Duke
systematically learn
 CEA
models are one (albeit not the only)
tool that we use
Oncology modeling at Duke
 Rapid
learning cancer clinics
 Combine sound data collection
with an explicit mechanism for
learning
Data
A
data warehouse is used to
generate multiple views, typically
derived from linked files (e.g.,
cancer type, treatments, outcomes)
 The lynchpin is a data set of
patient-reported outcomes (derived
from the PCM)
Data quality
The PCM contains 70+ items on a 0-10
scale (e.g., level of nausea during last 7
days)
 Filled out in waiting room using etablets – migrating to web
 Results are reported to clinicians in real
time – for example, highlighting issues
to discuss during the visit

Incentives
 Patients:
confident that their
concerns won’t be overlooked
 Physicians: saves time in
performing a review of symptoms
 Principle:
(Sufficiently) valid data are
produced by design, not by accident
Formal learning structure
 Learning
from the databases
occurs within a formal PDCA cycle
 Relevant
stakeholders are represented
 The stakeholders determine the level of
accuracy / precision required to make
decisions
 The stakeholders determine study design
(e.g., interventional, observational)
Types of designs
 Observational
designs with
undirected machine learning
 Observational designs with prespecified hypotheses
 Pre-post designs with
interventions
 Randomized trials
Example
 Is
it “worth it” to refer patients with
high levels of psychological
distress to specialized counseling?
Inputs
Observational data – natural history of
outcomes by level of distress
 CEA model to estimate what level of
improvement would justify use of
specialized counseling resources
 Literature review on expected impact of
counseling
 Pre-post design to assess impact of
counseling in our setting

Criteria for learning
A
practice is worth changing if the
alternative is cost-effective
 We use CEA models that are
simple to moderately complex
Comment
 Our
goal is to systematically and
explicitly embed learning into our
usual procedures
Final thoughts

Health economics has always been
quantitative – now, it is becoming more
explicitly “statistical” as well
 A statistically-inspired literature on CEA is
rapidly developing – a distinguishing
characteristic is the ability to accommodate
increasingly complex models through
advances in computation
Final thoughts
 The
danger in this literature is that,
if its perspective is entirely
statistical, it can become divorced
from reality
 A particular area of promise lies in
integrating CEA modeling with
modern systematic approaches to
learning