Transcript Keith Abrams

Generalised Evidence Synthesis
Keith Abrams, Cosetta Minelli, Nicola Cooper &
Alex Sutton
Medical Statistics Group
Department of Health Sciences,
University of Leicester, UK
CHEBS Seminar
‘Focusing on the Key Challenges’
Nov 7, 2003
Outline
• Why Generalised Evidence Synthesis?
• Bias in observational evidence
• Example: Hormone Replacement
Therapy (HRT) & Breast Cancer
• Discussion
Why Generalised Evidence Synthesis?
• RCT evidence ‘gold standard’ for assessing
efficacy (internal validity)
• Generalisability of RCT evidence may be
difficult (external validity), e.g. CHD & women
• Paucity of RCT evidence, e.g. adverse events
• Difficult to conduct RCTs in some situations,
e.g. policy changes
• RCTs have yet to be conducted, but health
policy decisions have to be made
• Consider totality of evidence-base – (G)ES
beyond MA of RCTs
Assessment of Bias in
Observational Studies - 1
• Empirical evidence relating to potential extent
of bias in observational evidence (Deeks et al.
2003)
• Primary studies:
– Sacks et al. (1982) & Benson et al. (2000)
• Primary & Secondary studies (meta-analyses):
– Britton et al. (1998) & MacLehose et al. (2000)
• Secondary studies (meta-analyses):
– Kunz et al. (1998,2000), Concato et al. (2000) &
Ioannidis et al. (2001)
Assessment of Bias in
Observational Studies - 2
• Using a random effects meta-epidemiology
model (Sterne et al. 2002)
– Sacks et al. (1982) & Schultz et al. (1995) ~ 30%
– Ioannidis et al. (2001) ~ 50%
– MacLehose et al. (2000) ~ 100%
• Deeks et al. (2003) simulation study:
comparison of RCTs and historical/concurrent
observational studies
– Empirical assessment of bias – results similar to
previous meta-epidemiological studies
– Methods of case-mix adjustment, regression &
propensity scores fail to properly account for bias
Approaches to Evidence
Synthesis
• Treat sources separately, possibly
ignoring/downweighting some implicitly
• Bayesian approach & treat observational
evidence as prior for RCTs & explicit
consideration of bias:
– Power Transform Prior
– Bias Allowance Model
• Generalised Evidence Synthesis
Example – HRT
• HRT used for relief of menopausal symptoms
• Prevention of fractures, especially in women
with osteoporosis & low bone mineral density
• BUT concerns have been raised over
possible increased risk of Breast Cancer
HRT & Breast Cancer – RCT Evidence before
July 2002
RCT
Year
HRT
Nachtigall
1979
0
/ 84
4
/ 84
Christiansen
1980
2
/ 56
1
/ 259
Genant
1990
1
/ 116
1
/ 40
<
Gallagher
1991
0
/ 62
1
/ 20
<
Lufkin
1992
1
/ 39
1
/ 36
Aloia
1994
0
/ 31
1
/ 70
Maheux
1994
0
/ 30
1
/ 30
Munk-Jensen
1994
2
/ 100
0
/ 51
PEPI
1995
7
/ 701
1
/ 174
Speroff
1996
6
/ 1128
0
/ 137
Steele
1997
1
/ 37
0
/ 37
Hulley (HERS)
1998
32
/ 1380
25
/ 1383
Komulainen
1999
2
/ 232
1
/ 232
Alexandersen
2000
1
/ 150
1
/ 50
<
Angerer
2000
0
/ 215
1
/ 106
<
Herrington
2000
1
/ 204
0
/ 105
Gallagher
2001
0
/ 243
4
/ 246
Mosekilde
2001
2
/ 502
5
/ 504
Viscoli
2001
5
/ 337
5
/ 327
Pooled
Control
<
>
<
>
>
<
OR 0.97 95% CI 0.67 to 1.39
0.05
0.1
1
Odds Ratio (log scale)
Source: Torgerson et al. (2002)
5
10
20
HRT & Breast Cancer – Observational Evidence*
HRT
Case-Control (Hospital)
Morabia
Vessey
La Vecchia
Katsouyanni
Franceschi
1974
1982
1987
1990
1992
No HRT
80
47
119
42
151
/ 144
/ 51
/ 64
/ 70
/ 132
104
369
1496
404
1265
/
/
/
/
/
178
411
1386
770
1379
1976
1981
1981
1983
1983
1984
1988
1989
1989
808
437
86
39
136
157
604
132
117
/
/
/
/
/
/
/
/
/
932
542
84
86
109
122
720
148
134
714
335
275
226
400
519
1892
269
149
/
/
/
/
/
/
/
/
/
869
420
282
458
414
547
2297
277
161
1985
1985
1986
1988
1991
205
341
618
30
355
/
/
/
/
/
954
1418
2442
125
1338
243
370
714
306
178
/
/
/
/
/
976
1422
3084
1076
702
Pooled
Case-Control (Population)
Brinton
CASH
Hislop
Bain
Ewertz
Long Island
4 State Study
Yang/Gallagher
Stanford
Pooled
Cohort
Canadian NBSS
Schairer
Nurses Health
Netherlands Cohort
Iowa Womens Health
Pooled
ALL Observational
Observational
All
OR 1.18 95% CI 1.10 to 1.26
RCTs
RCTs
OR 0.97 95% CI 0.67 to 1.39
0.5
Odds Ratio (log scale)
Source: Lancet (1997)
*
Adjusted for possible confounders
1
2
Use of Observational Evidence in
Prior Distribution
Quasi RCTs
Cohort
Case-Control
Synthesis
Empirical Evidence
Bias
Prior
RCTs
Power Transform Prior
Following Ibrahim & Chen (2000)
P( | Data)  L( | RCTs )  L( | Obs )  P( )

•
•
•
•
0    1 is degree of downweighting
 = 0  total discounting
 = 1  accept at ‘face value’
Evaluate for a range of values of 
1.25
1.00
0.75
0.50
Odds Ratio (log scale)
1.50
Power Transform Prior – Results 1
0.0
0.1
0.2
0.3
0.4
0.5
Alpha
0.6
0.7
0.8
0.9
1.0
Bias Allowance Model
Following Spiegelhalter et al. 2003
MA RCTs : yi ~ N ( i , s ) &  i ~ N (  , )
2
i
2
MA Obs : z j ~ N ( v ) &  j ~ N ( ,  )
2
j, j
2
 ~  | z i.e. obs. form prior for 
     &  ~ N (0,  )
*
2
• * is unbiased true effect in observational studies
•  is bias associated with observational evidence
• 2 represents a priori beliefs regarding the possible
extent of the bias
Bias Allowance Model - Results
Belief/Source
Bias
2
‘Face Value’
0%
Total Discounting
OR
95% CrI
P(OR>1)
0
1.14
1.07 to 1.20
1.00
%

0.87
0.30 to 1.60
0.31
Sacks & Schultz
30%
0.02
1.08
0.85 to 1.37
0.72
Ioannidis
50%
0.08
1.00
0.68 to 1.45
0.50
100% 0.24
0.94
0.56 to 1.49
0.40
MacLehose
HRT & Breast Cancer: Evidence – July 2002
HERS II (JAMA July 3) [Follow-up of HERS]
– n = 2321 & 29 Breast Cancers
– OR 1.08 (95% CI: 0.52 to 2.25)
• WHI (JAMA July 17) [Stopped early]
– n= 16,608 & 290 Breast Cancers
– OR 1.28 (95% CI: 1.01 to 1.62)
• HERS II & WHI
– OR 1.26 (95% CI: 1.01 to 1.58)
• Revised Meta-Analysis of RCTs
– WHI 68% weight
– OR 1.20 (95% CI: 0.99 to 1.45)
Power Transform Prior – Results
1.25
Power Prior Posterior
0.75
1.00
HERS II & WHI
0.50
Odds Ratio (log scale)
1.50
30%
0.0
0.1
0.2
0.3
0.4
0.5
Alpha
0.6
0.7
0.8
0.9
1.0
Generalised Evidence Synthesis
• Modelling RCT & observational (3
types) evidence directly;
– Hierarchical Models (Prevost et al, 2000;Sutton
& Abrams, 2001)
– Confidence Profiling (Eddy et al, 1990)
• Overcomes whether RCTs should form
likelihood & observational studies prior
Generalised Evidence Synthesis
RCTs
Quasi RCTs
Cohort
Case-Control
Beliefs
Synthesis
Utilities
Costs
Decision
Model
Routine
Hierarchical Model
yij ~ N [ ij , s ]
2
ij
i  1, , n j & j  1, , J
 ij ~ N [ j ,  ]
2
j
 j ~ N [  , ]
2
 ~ [,],  ~ [,] &  ~ [,]
2
j
2
HRT: Hierarchical Model - Results
Independent
Hierarchical
OR
95% CrI
OR
95% CrI
RCT
0.89
0.39 to 1.52
1.02
0.76 to 1.27
Cohort
0.98
0.84 to 1.12
1.01
0.89 to 1.13
CC-P
1.06
0.96 to 1.06
1.05
0.97 to 1.15
CC-H
1.23
0.94 to 1.55
1.12
0.93 to 1.36
Overall
1.05*
0.98 to 1.13
1.05
0.87 to 1.24
* Ignores study-type
Hierarchical Model - Extensions
• Inclusion of empirical assessment of
(differential) bias with uncertainty, i.e.
distribution
• Bias Constraint, e.g. HRT
 RCT     Coh     CCP     CCH  
Discussion – 1
• Direct vs Indirect use of non-RCT evidence
– Direct: intervention effect, e.g. RR
– Indirect: other model parameters, e.g. correlation
between time points
• Allowing for bias/adjusting at study-level
– IPD if aggregate patient-level covariates are
important, e.g. age, prognostic score
– Quality – better instruments for non-RCTs &
sensitivity of results to instruments
Discussion – 2
• Subjective prior beliefs regarding relative
credibility (bias or relevance) of sources of
evidence
– Elicitation
• Bayesian methods provide …
– A flexible framework to consider inclusion of all
evidence, & …
– which is explicit & transparent, BUT …
– Require careful & critical application
References
Deeks JJ et al. Evaluating non-randomised intervention studies. HTA
2003;7(27).
Eddy DM et al. A Bayesian method for synthesizing evidence. The
Confidence Profile Method. IJTAHC 1990;6(1):31-55.
Ibrahim JG & Chen MH. Power prior distributions for regression models.
Stat. Sci. 2000 15(1):46-60.
Prevost TC et al. Hierarchical models in generalised synthesis of
evidence: an example based on studies of breast cancer. Stat Med
2000;19:3359-76.
Sterne JAC et al. Statistical methods for assessing the influence of study
characteristics on treatment effects in ‘meta-epidemiological’ research.
Stat. Med. 2002;21:1513-1524.
Spiegelhalter DJ, Abrams KR, Myles JP. Bayesian Approaches to
Clinical Trials & Health-care Evaluation. London: Wiley, 2003.
Sutton AJ & Abrams KR. Bayesian methods in meta-analysis and
evidence synthesis. SMMR 2001;10(4):277-303.