Transcript Document

Economic evaluations alongside
clinical trials: what they contribute,
how they are performed and their
limitations
Dr. Stavros Petrou
Presentation to Nottingham CTU
19th June 2008
What is economic evaluation?
• Premise: scarce (health care) resources
• Aim: to maximise health gain with the available
resources
• Method: compare cost and consequences of
interventions
• Balance: about costs and consequences, inputs
and outputs
• Economic evaluation: explicit criteria for making
choices.
Definition of economic evaluation
• Definition of economic evaluation:
“The comparative analysis of alternative
courses of action in terms of both their costs
and their consequences” (Drummond et al, 1997)
• Requires:
– a comparison of two or more alternatives
– examination of both costs and
consequences
Types of economic evaluation
Is there good evidence on effectiveness of interventions being compared?
Is effectiveness of interventions equal?
NO
YES
NO
Costing study
YES
Cost minimization study
Can all outcomes be valued in monetary
terms ( e.g. willingness to pay)?
YES
Cost benefit analysis
NO
NO
Can outcomes be measured as
quality adjusted life years?
Cost-effectiveness analysis
YES
Cost-utility analysis
Economic evaluation alongside
trials
Two independent groups
Control group
Treatment group
Patient
(Cost, Effect)
Patient
(Cost, Effect)
1
( CC1, EC1 )
1
( C T1 , E T1 )
2
( CC2, EC2 )
2
( C T2 , E T2 )
3
( CC3, EC3 )
3
( C T3 , E T3 )
.
.
.
.
nC
( CCn, ECn )
nT
( C Tn , E Tn )
Mean:
( CC, EC )
Mean:
( CT, ET )
Economic evaluation alongside trials
Two independent groups
Control group
Mean:
Treatment group
( CC, EC )
Mean:
Incremental cost-effectiveness ratio
CT - CC
ET - EC
( CT, ET )
The cost-effectiveness plane
New treatment
more costly
NW
New treatment more effective
but more costly
Existing treatment
dominates
New treatment
less effective
New treatment
more effective
C
New treatment less costly
but less effective
SW
NE
New treatment
less costly
New treatment
dominates
SE
The cost-effectiveness plane
NW
New treatment
more costly
NE
Maximum acceptable ICER
New treatment
cost-ineffective
New treatment
less effective
New treatment
more effective
C
New treatment
cost-effective
SW
New treatment
less costly
SE
NICE: method of operation
• Preferred measure of cost-effectiveness:
– Quality-adjusted life year (QALY)
– Alternatives - e.g. cost per life year gained acceptable
• No absolute threshold for level of acceptability:
– no empirical basis for setting a value
– may in some circumstances want to ignore threshold
– A set threshold implies efficiency has absolute priority
over other objectives (e.g. fairness)
– many technology suppliers are monopolies; a
threshold would discourage price competition
Cost per QALY results from NICE
Those in bold: rejected
Source: N Devlin, D Parkin. Does NICE have a cost-effectiveness threshold and what other
factors influence its decisions? A binary choice analysis. Health Economics 2004: 13(5):437-52.
Why QALYs as a measure of
outcome?
• To use cost-effectiveness as a guide to decisionmaking, we need to compare the c-e of different
uses of resources
• Therefore we need an effectiveness measure
that can be used in a wide range of settings:
• Life-years gained
– but only where survival is main outcome
• Quality adjusted life years (QALYs)
– Composite of survival and quality of life
1.0
Health profile with intervention
Health profile
without intervention
0
1
2
3
4
Time in years t
5
6
The EuroQol EQ-5D
The following questions are designed to tell us about your state of health today. Please
look at each group of questions, and then tick the statement which best describes you
own health state today. You should make five ticks in all, one for each group.
Mobility
I have no problems walking about
I have some problems walking about
I am confined to bed
Self-care
I have no problems with self-care
I have some problems washing or dressing myself
I am unable to wash or dress myself
Usual activities
I have no problems performing my usual activities
I have some problems with performing my usual activities
I am unable to perform my usual activities
Pain/discomfort
I have no pain or discomfort
I have moderate pain or discomfort
I have extreme pain or discomfort
Anxiety/depression
I am not anxious or depressed
I am moderately anxious or depressed
I am extremely anxious or depressed
____
____
____
____
____
____
____
____
____
____
____
____
____
____
____
Tariffs for the EuroQol EQ-5D
Coefficient
Constant
0.081
Mobility
- Some problems
0.069
- Confined to bed
0.314
Self care
- Some problems
0.104
- Unable to wash/dress
0.214
Usual activities
- Some problems
0.036
- Unable to perform
0.094
Pain/discomfort
- Moderate
0.123
- Extreme
0.386
Anxiety/depression
- Moderate
0.071
- Extreme
0.236
N3 (Level 3 at least once)
0.269
Why randomised trials?
• Most treatments do not have large
effects; reliably detecting moderate
effects requires studies that
simultaneously avoid:
– moderate bias
• proper randomisation
• intention-to-treat analysis
• avoidance of inappropriate sub-group analysis
– moderate random error
• adequate size
Why economic assessment in
clinical trials?
• Many health economists advocate models using
lots of data sources: trial, non-trial, summary data
etc
• But...issues of bias and random error also affect
incremental resource use and health outcomes
• And, trials provide patient-level data, useful for:
– Dealing with patient heterogeneity
– Examining covariance, e.g. between costs and outcomes
– Building and validating models, eg to extrapolate
• Trials allow prospective measurement of resources
& outcomes of interest
• Incremental cost of economic evaluation alongside
trials is low
UK Collaborative ECMO Trial
• Pragmatic RCT
• 185 mature (35 weeks, 2kgs) infants with
severe respiratory failure (ox. index 40)
• Infants recruited from 55 centres in 1993-5
• Randomisation to ECMO: 1 of 5 specialist
centres, cannulated, ECMO support
• Randomisation to CM: conventional care
• Outcomes: survival without severe disability up
to 7 years of age
Design and analytical issues
• Costs: measurement and valuation
• Consequences: measurement and
valuation
• Analytical issues: within and beyond RCTs
Perspectives and types of costs
• Direct costs
– Health care system
– Other care inputs, e.g. social services
– Patient, family, carer expenses
• Informal care costs
– Opportunity cost of unpaid informal care
• Indirect costs
– Time off work, reduced productivity
– Early retirement
– Premature mortality
• Transfer payments
– Payments such as social security benefits that
redistribute output with no exchange of goods or
services
Three elements of cost
• Resource use (cost generating event)
– a day in hospital
– a GP consultation
• Unit cost
– cost per in-patient day / per hospitalisation
– cost per GP consultation / per GP minute
• Cost
– the product of resource use and unit costs
Measurement of resource use
• All significant resource inputs during first 7 years of life
• Trial data collection forms:
– transport (mode, distances)
– duration and intensity of neonatal care
• Observational research for infant death
• Postal questionnaires, validated by information from
hospital records:
– post-discharge hospital and social service
utilisation
• GP records:
– community service utilisation
Valuation of resource use
• Unit costs employed to value resource
use
• Neonatal care – top down methodology
• Readmissions - Reference cost schedules
• Community service utilisation - Published
cost data (previous studies, PSSRU, etc.)
• Drugs – BNF
• £, 2002-3 prices
Time horizon and discounting
• Should extend far enough into the future to
capture all costs and consequences of
interventions being evaluated
• ECMO Study – time horizon initially reflected that
of RCT
• Costs (and consequences) occurring beyond the
first year of life must be reduced to present
values
• Rationale for discounting
- Time preference
- Opportunity cost – market basis
Mean costs and mean cost differences
Cost category
ECMO (n=93)
CM (n=92)
Mean
(SD)
Mean
(SD)
Mean
difference
P value
Death
434
(665)
846
(714)
-412
<0.0001
Transport
2147
(3080)
296
(777)
1851
<0.0001
Initial hospitalisation
22996
(21618)
6143
(13144)
16853
<0.0001
Hospital readmissions
1518
(3868)
1127
(3063)
391
0.447
Outpatient hospital
care
970
(1592)
496
(1008)
474
0.017
Community care
1976
(2882)
1247
(2647)
792
0.075
Other costs
229
(1255)
74
(182)
155
0.241
Total costs
30270
(24380)
10229
(18356)
20041
<0.0001
Source: Petrou S, Bischof M, Bennett C, Elbourne D, Field D, McNally H. Cost-effectiveness of neonatal ECMO based
on seven year results from the UK Collaborative ECMO Trial. Pediatrics 2006; 117(5): 1640-1649.
Bootstrap mean cost differences
Mean cost difference
(ECMO – CM)
Bootstrap mean cost
difference (95% CI)
Death
-412
-406 (-614, -209)
Transport
1851
1849 (1251, 2497)
Initial hospitalisation
16853
16,826 (11,775, 22,044)
Hospital readmissions
391
371 (-603, 1334)
Outpatient hospital care
474
481 (97, 875)
Community care
792
730 (-62, 1549)
Other costs
155
156 (-13, 443)
Total costs
20041
20057 (13690, 26318)
Cost category
Source: Petrou S, Bischof M, Bennett C, Elbourne D, Field D, McNally H. Cost-effectiveness of neonatal ECMO
based on seven year results from the UK Collaborative ECMO Trial. Pediatrics 2006; 117(5): 1640-1649.
Measurement of outcomes at 7 years
• Survival period → life years gained
• Neurodevelopmental assessments performed by
developmental psychologist across 6 domains:
– cognitive ability
– neuromotor skills
– general health
- behaviour
- hearing
- vision
• Each domain defined as normal, impaired or
mild, moderate or severely disabled
• Overall status defined by highest degree of
impairment or disability in any domain
→ disability-free life years gained
• Limitations of QALYs in childhood context
ECMO
(n=93)
CM
(n=92)
28 (30.1%)
54 (58.7%)
Between discharge and one year
2 (2.2%)
0 (0.0%)
Between one years and four years
1 (1.1%)
0 (0.0%)
Between four years and seven years
0 (0.0%)
0 (0.0%)
31 (33.3%)
54 (58.7%)
Between discharge and one year
1 (1.1%)
1 (1.1%)
Between one year and four years
2 (2.2%)
2 (2.2%)
Between four years and seven years
3 (3.2%)
1 (1.1%)
Total
6 (6.5%)
4 (4.3%)
Severe disability
3 (3.2%)
0 (0.0%)
Moderate disability
9 (9.7%)
6 (6.5%)
Mild disability
13 (14.0%)
11 (12.0%)
Impairment only
21 (22.6%)
15 (16.3%)
No abnormal signs or disability
10 (10.8%)
2 (2.2%)
31/56 (55.4%)
17/34 (50.0%)
Deaths
Before discharge
Total
Loss to follow-up
Final assessment at seven years of age
Known survivors with no disability
Cost-effectiveness of neonatal ECMO
Incremental cost-effectiveness ratio = C / E
= £13,385 per life year gained
or
= £23,566 per disability-free life year gained
NB: Natural or physical unit of outcome, which
ignores full range of consequences.
Cost-effectiveness plane, life years gained
£40,000
ECMO less effective,
more costly
maximum ICER
ECMO more effective,
more costly
£30,000
Incremental costs
£20,000
£10,000
-4.0
-3.0
-2.0
-1.0
£0
0.0
1.0
2.0
3.0
-£10,000
-£20,000
ECMO less effective,
less costly
-£30,000
-£40,000
Incremental life years gained
ECMO more effective,
less costly
4.0
Cost-effectiveness plane, disability-free life years gained
£40,000
ECMO less effective, more
costly
maximum ICER
ECMO more effective,
more costly
£30,000
Incremental costs
£20,000
£10,000
-4.0
-3.0
-2.0
-1.0
£0
0.0
1.0
2.0
3.0
-£10,000
-£20,000
ECMO less effective, less
costly
-£30,000
-£40,000
Incremental disability-free life years gained
ECMO more effective,
less costly
4.0
Cost-effectiveness acceptability curves, probability that neonatal ECMO is cost-effective after seven years
1 0.98
0.9
Probability cost-effective
0.8
0.7 0.69
0.6
0.5
0.4
0.3
0.2
0.1
0
£0
£10,000
£20,000
£30,000
Willingness to pay threshold (£)
Life year gained
Disability-free life year gained
£40,000
£50,000
Mean net benefits of neonatal ECMO
Mean net benefits = Rc.E - C
Rc
Life year gained
Mean net
benefit
95% CI
Disability-free life year
gained
Mean net
benefit
95% CI
0
-20,130
(-26,027, -13,522)
-19,949
(-25,997, -13,375)
10,000
-5,299
(-14,475, 3,750)
-11,387
(-20,394, -244)
20,000
9,532
(-6,691, 25,351)
-2,825
(-18,377, 13,804)
30,000
24,362
(1,474, 47,314)
5,737
(-16,245, 29,393)
40,000
39,193
(8,926, 69,513)
14,299
(-14,863, 44,781)
50,000
54,024
(15,496, 91,567)
22,861
(-13,246, 60,319)
Are trial-based economic
evaluations sufficient?
• Under the right circumstances, Yes
• Like clinical evidence, economic evidence can stand
alone or be synthesised
• Requirements: reasonable comparators, adopts final
outcomes, collects data on a sufficiently broad set of
services, adequately powered, sufficiently long follow-up,
representative patient population
• Problems: Use of inappropriate statistical tests, lack of
power, failure to handle missing data, lack of intention-totreat analysis, follow-up too short, lack of transferability
Was the ECMO trial-based economic
evaluation sufficient?
•
•
•
•
•
•
•
•
•
Reasonable comparator
Representative patient population
Intention to treat analysis
Adequately powered
Appropriate perspective
Sufficiently long follow-up
Adopts final outcomes
Appropriate statistical tests
Handled missing data





x
?


Longer-term cost-effectiveness
•
•
Simple decision-analytic model
Assumptions:
–
–
–
–
restricted to first 18 years of life
survivors to 7 years survive to 18 years
disability status at age 7 does not vary
excess annual costs during years 4-7 continue
during years 8-18
 ICER: £7344 per life year gained
£11802 per disability-free life year gained
Vehicles for economic evaluation
• Prospective collection of data alongside
randomised controlled trials
– Least subject to bias, control over instruments, low
incremental cost. May need to supplement.
• Prospective collection of data alongside nonrandomised studies
– More subject to possible bias, control over instruments,
low cost.
• Retrospective analysis of available data
– Low control over design and data, subject to bias.
• Modelling study
– Data from different sources, combined in decisionanalytic models. Hard to validate. Useful and often
unavoidable adjunct to trials.
Limitations of trials as a vehicle
for economic evaluation
Trial limitations
NICE Examples
Inappropriate or partial
comparisons
Temozolomide (recurrent
malignant glioma)
More than one trial
Drugs for Alzheimer’s
Partial measurement
Riluzole (resource use)
Unrepresentative practice
Glycoproteins
Intermediate outcomes
Beta interferon (MS)
Limited follow-up
Implantable cardioverter
defibrillators
No trials
Liquid-based cytology
Modelling
• Hence modelling aims to address all these
questions and can be used instead of or as a
complement to trial evidence
–
–
–
–
–
Structure the economic question
Extrapolate beyond observed data
Links intermediate and final endpoints
Generalizes results to other settings/patient groups
Synthesises evidence and can create head-to-head
comparisons where RCTs don’t exist
– Can indicate the need for further research
Models should:
• Represent a simplification of the real world
• Encourage decision-makers to be explicit
• Reflect current clinical practice and use
appropriate comparators
• Be based on the best quality data available
• Cover the appropriate time period
• Include sensitivity analysis to explore uncertainty
of data inputs and model structure
• Be transparent and reproducible
• Have internal and external validity
The realities of research funding
• A large proportion of funding for economic
evaluation is attached to trials
• Three options available to analysts:
– Satisfy expectation of standard trial-based economic
evaluation and risk misleading results
– Refuse to collaborate
– Pragmatic collaboration:
• Seek opportunities to use modelling to inform trial
design
• Ensure sufficient budget for analysis which includes
synthesis and modelling
A pragmatic way forward?
Primary
research
(e.g. RCTs)
Synthesis and
modelling with
updated
evidence
Identify
decision
problems
Setting of
research
priorities
Synthesis and
modelling
given
available
evidence
Suggested checklist for assessing economic evaluations
1) Was question well-defined?
2) Were alternatives clearly described?
3) What evidence on effectiveness was used?
4) Were resources measured and valued fully & appropriately?
5) Was discounting necessary and was it performed?
6) Were incremental costs and outcomes analysed?
7) Was an adequate sensitivity analysis performed?
8) Are the results adequate to inform purchasing?
9) Are the conclusions justified?
10) Are the results applicable to the local population?
Source: Drummond MF, O’Brien, B, Stoddart GL, Torrance GW. Methods for the economic evaluation of
health care programmes. 2nd edition. Oxford: Oxford University Press, 1997.
Drummond MF, Jefferson T, et al. Guidelines for authors and peer reviewers of economic submissions to
the BMJ. BMJ 1996; 313:275-83.