Transcript Meta-analysis - University of Oxford

How to read a Systematic Review:
The FAST tool
Find
Appraise
Synthesise
Transferability
Paul Glasziou
Centre for Evidence Based Medicine
University of Oxford
www.cebm.net
The best evidence for
different types of question
Level
Treatment
Prognosis
Diagnosis
I
Systematic
Review of …
Systematic
Review of …
Systematic
Review of …
II
Randomised
trial
Inception
Cohort
Cross
sectional
III
Are RCTs always needed for
treatment questions?
• Some immediate & dramatic
effects don’t need RCTs*
• Example:
• Child with nasal foreign body
 Dislodged with Parent Kiss method
 Case series of success 15/19
o Botma J Laryngol Otol 2000
* Glasziou, Chalmers, Rawlins, McCulloch BMJ 2007
What do you do?
For an acutely ill patient, you do a search
• You find 17 studies:
• some show significant results; most do not
The 17 studies : Forest Plot/Blobbogram
A.
B.
C.
D.
Which is the smallest study?
Which is the largest study?
How many are statistically
significant?
Which studies are “large enough”?
The 17 studies: of streptokinase for MI
How large should the study be?
What sample size is needed?
For E.H.D.* the usual mortality rate is 0%
What sample size is needed to detect a
reduction in mortality?
• 100
• 1,000
• 100,000
• 1,000,000
* Excessive happiness disorder
Sample Size: Café Rule 1
The 50:50 Rule (proportions)
50 events are needed in the control group:
(For an 80% chance of finding a 50% reduction)
Control Rate Number
Events
50
20%
Control#
(Rule 1)
250
Control#
(Fisher exact)
215
10%
50
500
463
5%
50
1000
962
Glasziou P, Doll H. Was the study big enough? Two cafe rules. Evid Based Med. 2006;11(3):69-70.
What sample size is needed?
• There is usually a 12% mortality rate
 You think your treatment will lower mortality
by 50%
• What sample size is needed?
What sample size is needed?
• There is usually a 12% mortality rate
 You think your treatment will lower mortality by 50%
• What sample size is needed?
• 12% means
 12/100 or 24/200 or 48/400
 and 50 per 417
• Control + Treatment Groups = 834 in total
Systematic Review or meta-analysis?
• A Systematic Review is a review of a clearly
formulated question that uses systematic and
explicit methods to identify, select and critically
appraise relevant research, and to collect and
analyse data from the studies that are included
in the review.
• Statistical methods (meta-analysis) may or may
not be used to analyze and summarize the
results of the included studies.
Is the review any good?
FAST appraisal
• Question – What is the PICO?
• Finding
 Did they find most studies?
• Appraisal
 Did they select good ones?
• Synthesis
 What to they all mean?
• Transferability of results
FIND
APPRAISE
SYNTHESISE
TRANSFERABLE
Why do I need to check the review?
Most reviews do not pass minimum criteria
A study of 158 reviews*
 Only 2 met all 10 criteria
 Median was only 1 of 10 criteria met
FAST tool = 4 criteria
* McAlister Annals of Intern Med 1999
FIND
APPRAISE
SYNTHESISE
TRANSFERABLE
What it the review question (PICO)?
•
•
•
•
Population
Intervention
Comparison
Outcome(s)
Do pedometers increase
activity and improve health?
• Find: what is your
search strategy?
 Databases?
 Terms?
 Other methods?
Do yourself then
Get neighbour’s help
FIND
APPRAISE
SYNTHESISE
TRANSFERABLE
FIND: Did they find all Studies?
• Check for existing systematic review?
• Good initial search
 Terms (text and MeSH)
 At least 2 Databases: MEDLINE, EMBASE, CINAHL, CCTR, ...
• Plus a Secondary search
 Check references of relevant papers & reviews and
 Find terms (words or MeSH terms) you didn’t use
 Search again! (snowballing)
FIND
APPRAISE
SYNTHESISE
TRANSFERABLE
Is finding all published studies enough?
• Negative studies less likely to be
published than ‘Positive’
• How does this happen?
• Follow-up of 737 studies at Johns Hopkins
(Dickersin, JAMA, 1992)
 Positive SUBMITTED more than negative
(2.5 times)
FIND
APPRAISE
SYNTHESISE
TRANSFERABLE
Registered vs Published Studies
Ovarian Cancer chemotherapy: single v combined
Published
No. studies
Registered
16
13
1.16
1.05
95% CI
1.06-1.27
0.98-1.12
P-Value
0.02
0.25
Survival ratio
Simes, J. Clin Oncol, 86, p1529
FIND
APPRAISE
SYNTHESISE
TRANSFERABLE
Registered vs Published Studies
Ovarian Cancer chemotherapy: single v combined
Published
No. studies
Registered
16
13
1.16
1.05
95% CI
1.06-1.27
0.98-1.12
P-Value
0.02
0.25
Survival ratio
Simes, J. Clin Oncol, 86, p1529
FIND
APPRAISE
SYNTHESISE
TRANSFERABLE
Which are biased? Which OK?
1.
2.
3.
4.
5.
All
All
All
All
All
studies published in BMJ, Lancet, JAMA or NEJM
publicly funded studies
studies with more than 100 patients
studies conducted in the Northern Hemisphere
studies registered studies
FIND
APPRAISE
SYNTHESISE
TRANSFERABLE
Publication Bias: Solution
• All trials registered at inception,
o The National Clinical Trials Registry: Cancer Trials
o National Institutes of Health Inventory of Clinical
Trials and Studies
o International Registry of Perinatal Trials
• Meta-Registry of trial Registries
 www.controlled-trials.com
FIND
APPRAISE
SYNTHESISE
TRANSFERABLE
APPRAISE & select studies
Did they select only the
good quality studies?
FIND
APPRAISE
SYNTHESISE
Miscalculating Number
Needed to Treat (NNT)
TRANSFERABLE
1. EFFICACY: Would the death penalty for
miscalculating an NNT decrease miscalculation?
Yes/No
2. ATTITUDE: Should we introduce the death
penalty for miscalculation of NNT?
Yes/No
FIND
APPRAISE
SYNTHESISE
TRANSFERABLE
Selective Criticism of Evidence
Biased appraisal increases polarization
Death Penalty: beliefs and contradictory studies
In Favour
2
1
Against
0
Deterrent Efficacy
Attitude
Proponents
Opponents
-1
-2
Lord et al, J Pers Soc Psy, 1979, p2098
FIND
APPRAISE
SYNTHESISE
TRANSFERABLE
Selective Criticism of Evidence
28 reviewers assessed one “study”
results randomly positive or negative
“Positive” “Negative”
Relevance
5.2
4.9
Methods
4.2
2.4
Presentation
4.3
2.6
Summary
3.2
1.8
(Cog Ther Res, 1977, p161-75)
FIND
APPRAISE
SYNTHESISE
TRANSFERABLE
Assessment: How can you avoid biased
selection of studies?
• Assessment and selection should be:
Standardized “Objective” OR
Blinded to Results
* assessment of quality blind to study outcome
FIND
APPRAISE
SYNTHESISE
Synthesis: pooling the results
TRANSFERABLE
FIND
APPRAISE
SYNTHESISE
TRANSFERABLE
Meta-analysis (Forest) plot
Graphical and tabular summary of studies?
FIND
APPRAISE
SYNTHESISE
TRANSFERABLE
Transferable? Use in my patients
Is the AVERAGE effect similar across studies?
• If NO, then WHY?
 Study methods (RAMbo - biases)
 PICO (Patients, Intervention, …)
• If YES, then 2 questions
 Effect in different individuals?
 Which version of treatment?
FIND
APPRAISE
SYNTHESISE
TRANSFERABLE
Meta-analysis (Forest) plot
Are the results similar across studies? 3 tests
1. Eyeball” test – do they look they same?
2. Test of “Null hypothesis” of no variation (p-value)
3. Proportion of variation not due to chance (I2)
Epidemiology of Reviews
Conclusion
EBM and Systematic Review
•
•
EBM (quick & dirty)
Steps
•
•
1. Ask Question
2. Search
3. Appraise
1.
2.
3.
4.
5.
4. Apply
•
•
•
Time: 90 seconds
< 20 articles
This patient survives!
Systematic Review
Steps
•
•
•
Ask Question
Search ++++ x 2
Appraise x 2
Synthesize
Apply
Time: 6 months, team
< 2,000 articles
This patient is dead
Find a systematic review!! (and appraise it FAST)
Pros and cons of systematic
reviews
• Advantages
 Larger numbers & power
 Robustness across PICOs
• Disadvantages
 May conclude small biases
are real effects
Is the review any good?
FAST appraisal
• Question – What is the PICO?
• Finding
 Did they find most studies?
• Appraisal
 Did they select good ones?
• Synthesis
 What to they all mean?
• Transferability of results
Combined results
FIND
APPRAISE
SYNTHESISE
TRANSFERABLE
Using review results:
what do I do with my patient?
• STUDY: meta-analysis of behavioural
interventions for insomnia adults
 “.. confirms the efficacy of behavioral
interventions for person with chronic insomnia.“
• PROBLEM: No regimens for ‘behavioural
intervention’ described
 Author asked: “what specific treatment regime
(or regimes) would you recommend based on
your review?”
 Author response: “It was found that cognitive,
behavioral and relaxation therapies all in general
lead to similar improvements in sleep outcomes--although cognitive approaches might have been
a bit better. The references for these studies are
found in the article. “
Rx
“Behavioural
Intervention”
Summary: systematic reviews
• Advantages
 Larger numbers & power
 Robustness across PICOs
• Disadvantages
 May conclude small biases
are real effects
The results:
Are studies similar?
• What are the overall results?
• Similarity of results
 Heterogeneity statistic
• Similarity of question (PICO)
 Your judgement!
Traditional cf systematic
reviews
• Traditional




Many questions
No search methods
No inclusion criteria
No combining studies
• Systematic




One question
Explicit search
Explicit inclusion criteria
Combine study results
(meta-analysis)
Which are
(i) statistically significant * and
(ii) Clinically significant +?
(a*+)
(b+)
(c*)
Minimum clinical
Important difference
No difference
(d)
• Which studies (presented as Odds Ratio):
 Are not statistically significant?
 Have < 50 patients in the control arm?
Sample Size: Cafe Rule 2 - continuous
the 17/(SD squared) Rule
For continuous outcomes, number per arm is
17 / (“worthwhile difference” measured in SDs)
Difference
in mmHg
16
Difference
(in SDs)
Number per
arm
1
17
8
½
68
4
¼
272
In a small randomized double-blind trial of a new treatment for
acute myocardial infarction, the mortality in the treated group
was half that in the control group, but the difference was not
significant. We can conclude that:
a) The treatment is useless
b) there is no point in continuing to develop the treatment
c) the reduction in mortality is so great that we should
introduce the treatment immediately
d) we should keep adding cases to the trial until the Normal
test for comparison of two proportions is significant
e) we should carry out a new trial of much greater size
The best evidence for
different types of question
Level
Treatment
Prognosis
Diagnosis
I
Systematic
Review of …
Systematic
Review of …
Systematic
Review of …
II
Randomised
trial
Inception
Cohort
Cross
sectional
III
Has the systematic reviewer
done a good job?
FIND
APPRAISE
SYNTHESISE
TRANSFERABLE
Are the studies equivalent?
• Are variations in results between studies
consistent with chance?
(Test of homogeneity: has low power)
• If NO, then WHY?
 Variation in study methods (RAMbo - biases)
 Variation in PICO (Patients, Intervention, …)
Is bed rest ever helpful?
A systematic review of trials*
• 10 trials of bed rest after spinal
puncture
 no change in headache with bed rest
 Increase in back pain
• Protocols in UK neurology units 80% still recommend bed rest after
LP
Serpell M, BMJ 1998;316:1709–10
•*Allen,
…evidence
of harm available for 17
Glasziou, Del Mar. Lancet, 1999