Transcript LSHTM - Cochrane Collaboration

Systematic reviews and meta-analyses of health research
6–10 September 2010
From Evidence to Recommendations:
the GRADE methodology
David Tovey
Editor in Chief, The Cochrane Library
16/07/2015
1
The bottom line.....
Who?
What need?
Your reader
What action?
16/07/2015
What outcome?
2
Session Plan
•
•
•
•
•
•
•
•
What does reader want from an SR?
Grading the evidence: different approaches
Background about GRADE
Quality of evidence exercise
GRADE in practice
Is GRADE perfect?
Going from evidence to recommendations
Q&As
16/07/2015
3
Session Plan
•
•
•
•
•
•
•
•
What does reader want from an SR?
Grading the evidence: different approaches
Background about GRADE
Quality of evidence exercise
GRADE in practice
Is GRADE perfect?
Going from evidence to recommendations
Q&As
16/07/2015
4
Evidence-based healthcare decisions
Population values
and preferences
(Clinical) state and
circumstances
Expertise
Research evidence
16/07/2015
Haynes et al.52002
What do we mean by “research
evidence”?
• Clarity about:
– The question (PICO)
• For each comparison and important outcome:
– Is there an effect?
– What is the direction of the effect?
– What is the magnitude of the effect?
– How confident are we in the effect estimate?
• What is the link between RoB and results?
16/07/2015
6
Risk of bias?
16/07/2015
7
Session Plan
•
•
•
•
•
•
•
•
What does reader want from an SR?
Grading the evidence: different approaches
Background about GRADE
Quality of evidence exercise
GRADE in practice
Is GRADE perfect?
Going from evidence to recommendations
Q&As
16/07/2015
8
Which grading system to use?
• Many available
–
–
–
–
–
Australian National and MRC
Oxford Centre for Evidence-Based Medicine (CEBM)
Scottish Intercollegiate Guidelines (SIGN)
US Preventive Services Task Force
US professional organizations
• AHA/ACC, ACCP, AAP, Endocrine Society, etc.
• Cause of confusion, dismay
16/07/2015
9
Different approaches
16/07/2015
10
Different approaches
Level of
evidence
16/07/2015
Source of evidence
Ia
Ib
Systematic reviews (SR)
RCTs
II
Cohort studies
Grades of
recommend.
A
B
III
Case-control studies
IV
Case series
C
V
Expert opinion
D
11
Which hierarchy?
Recommendation for use of oral
anticoagulation in patients with atrial
fibrillation and rheumatic mitral valve disease
Evidence
B
A
IV
16/07/2015
Recommendation
Class I
1
C
Organization
AHA
ACCP
SIGN
12
Hierarchy of evidence
based on quality
STUDY DESIGN
RCTs
BIAS




Cohort Studies and Case
Control Studies
Case Reports and Case
Series, Non-systematic
Observations
Expert Opinion
16/07/2015
13
BMJ 2003
16/07/2015
BMJ, 2003
14
Limitations of existing systems
• Confuse quality of evidence with strength of
recommendations
• Inflexible
• Study-, not outcomes-based
• Focus on primary benefit, and not all important
outcomes related to a specific question
16/07/2015
15
Session Plan
•
•
•
•
•
•
•
•
What does reader want from an SR?
Grading the evidence: different approaches
Background about GRADE
Quality of evidence exercise
GRADE in practice
Is GRADE perfect?
Going from evidence to recommendations
Q&As
16/07/2015
16
GRADE Working Group
Grades of Recommendation Assessment,
Development and Evaluation
• Aim: to develop a common, transparent and sensible
system for grading the quality of evidence and the
strength of recommendations
• International group of guideline developers,
methodologists & clinicians from around the world
(>100 contributors) – since 2000
16/07/2015
17 2005,
CMAJ 2003, BMJ 2004, BMC 2004, BMC
AJRCCM 2006, Chest 2006, BMJ 2008
GRADE: Quality of evidence
The extent to which one can be
confident that an estimate of
effect or association is correct
16/07/2015
18
Quality of evidence across studies
Outcome #1
Quality: High
Outcome #2
Quality: Moderate
Outcome #3
III
V
II
IB
Old system
16/07/2015
Quality: Low
GRADE
19
Session Plan
•
•
•
•
•
•
•
•
What does reader want from an SR?
Grading the evidence: different approaches
Background about GRADE
Quality of evidence exercise
GRADE in practice
Is GRADE perfect?
Going from evidence to recommendations
Q&As
16/07/2015
20
Quality of Evidence
• List 5 characteristics that would increase your
confidence in the estimate of effect
or
• List 5 characteristics that would decrease your
confidence in the estimate of effect
16/07/2015
21
Session Plan
•
•
•
•
•
•
•
•
What does reader want from an SR?
Grading the evidence: different approaches
Background about GRADE
Quality of evidence exercise
GRADE in practice
Is GRADE perfect?
Going from evidence to recommendations
Q&As
16/07/2015
22
Components determining quality
• RCTs start high
• Observational studies start low
What lowers quality of evidence? 5 factors:
Methodological
limitations
16/07/2015
Inconsistency
of results
Indirectness
of evidence
Imprecision
of results
Publication
bias
23
Methodological
limitations
Inconsistency
of results
Indirectness
of evidence
Imprecision
of results
Publication
bias
Assessment of detailed design and execution
(risk of bias)
For RCTs:
– Lack of allocation concealment
– No true ‘intention to treat’ principle
– Inadequate blinding
– Loss to follow-up
– Early stopping for benefit
16/07/2015
24
Cochrane risk of bias graph in
RevMan 5
16/07/2015
25
Methodological
limitations
Inconsistency
of results
Indirectness
of evidence
Imprecision
of results
Publication
bias
• Judgement
– variation in size of effect
– overlap in confidence intervals
– statistical significance of heterogeneity
– I2 (or 2)
• Look for explanation for inconsistency
– patients, intervention, comparator, outcome,
methods
16/07/2015
26
Heterogeneity
Neurological or vascular complications or death within 30 days of endovascular
treatment (stent, balloon angioplasty) vs. surgical carotid endarterectomy (CEA)
16/07/2015
Methodological
limitations
Inconsistency
of results
Indirectness
of evidence
Imprecision
of results
Publication
bias
• Indirect comparisons
– Interested in head-to-head comparison
– Drug A versus drug B – but what if not studied?
• Differences in
– Patients (early cirrhosis vs. end-stage cirrhosis)
– Interventions (CRC screening: flex. sig. vs. colonoscopy)
– Comparator (e.g. differences in dose)
– Outcomes (non-steroidal safety: ulcer on endoscopy vs.
symptomatic ulcer complications)
16/07/2015
28
Methodological
limitations
Inconsistency
of results
Indirectness
of evidence
Imprecision
of results
Publication
bias
Any stroke (or death) within 30 days of endovascular treatment (stent, balloon
angioplasty) vs. surgical carotid endarterectomy (CEA)
16/07/2015
29
Methodological
limitations
16/07/2015
Inconsistency
of results
Indirectness
of evidence
Imprecision
of results
Publication
bias
30
Methodological
limitations
Inconsistency
of results
Indirectness
of evidence
Imprecision
Publication
of results
bias
All phase II and III
licensing trials for
antidepressant drugs
between 1987 and
2004.
74 trials – 23 were
not published.
16/07/2015
31
Factors that increase confidence
• Large effect
• Dose response
• All plausible confounding would reduce a
demonstrated effect
16/07/2015
32
Relative risk reduction:
….> 99.9% (1/100,000)
US Parachute Association reported
821 injuries and 18 deaths out of 2.2
million jumps in 2007
16/07/2015
BMJ 2003
33
16/07/2015
34
16/07/2015
35
Acetylcysteine (Keays 1991)
• Randomized trial fulminant hepatic failure after
paracetamol before acetylcysteine
• Loading dose and infusion of acetylcysteine until
resolution or death vs. no acetylcysteine
• Treatment began 33 to 96 hours after OD
• Validity
– concealment: envelopes
– blinding – none
• 12/25 (48%) acetylcysteine survived vs. 5/25 (20%) of
controls
– (RR dying 0.65, 95% CI 0.43 to 0.99; p=0.037)
16/07/2015
36
Quality of evidence
• RCT starts high
– no study limitations
– imprecise (CI too wide)
– directness (possibly indirect)
– consistency (no other RCT)
– publication bias – no suggestion
• Overall moderate quality
16/07/2015
37
Acetylcysteine for paracetamol OD
observational studies
• Prescott BMJ 1979
– severe liver damage AST or ALT > 1,000
– 1/62 treated within 10 hours (2%)
– retrospective series 33/57 (58%)
• Smilkstein NEJM 1988
–
–
–
–
2,540 patients
same criteria for damage
32/537 (6.1%) treated in 10 hours
247/935 (26.4%) treated 10 to 24 hours
16/07/2015
38
Quality assessment ― summary
Quality of Study
evidence design
High
Randomized
trial
Moderate
Low
Very low
Observation
al study
Lower if …
Higher if …
Study limitations
Large effect (e.g. RR 0.5)
Very large effect (e.g. RR 0.2)
Inconsistency
Evidence of dose-response
gradient
Indirectness
All plausible confounding
would reduce a
demonstrated effect
Imprecision
Publication bias
16/07/2015
39
Conceptualizing quality
High
Moderate
Low
Very low
16/07/2015




We are moderately confident in the estimate of effect:

The true effect is likely to be close to the estimate of

effect, but possibility to be substantially different.


Our confidence in the effect is limited: The true effect

may be substantially different from the estimate of the


effect.



We have very little confidence in the effect estimate: The
true effect is likely to be substantially different from the

estimate of effect.

We are very confident that the true effect lies close to
that of the estimate of the effect.
40
Adoption of GRADE
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
World Health Organization
British Medical Journal
American Thoracic Society
American College of Physicians
American Endocrine Society
Society of Critical Care Medicine
Cochrane Collaboration
BMJ Clinical Evidence
UpToDate
American College of Chest Physicians
European Soc of Thoracic Surgeons
KDIGO
Polish Institute for EBM
EBM Guidelines Finland
Society of Vascular Surgery
Society of Pediatric Endocrinology
European Respiratory Society
Surviving Sepsis Campaign
Allergic Rhinitis in Asthma Guidelines
Society of Vascular Surgery
Infectious Diseases Society of America
16/07/2015
•
•
•
•
•
•
•
•
•
•
•
•
•
National Institute for Health and Clinical
Excellence (NICE)
Agency for Health Care Research and Quality
(AHRQ)
Swedish National Board of Health and Welfare
Canadian Agency for Drugs and Technology in
Health
Ontario MOH Medical Advisory Secretariat
Agenzia Sanitaria Regionale, Bologna, Italy
German Agency for Quality in Medicine
Evidence-based Nursing Sudtirol, Alta Adige,
Italy
Norwegian Knowledge Centre for the Health
Services
University of Pennsylvania Health System Center
for EB Practice
Journal of Infection in Developing Countries –
International
Japanese Society of Oral and Maxillofacial
Radiology
Emergency Medical Services for Children
National Resource Center
41
GRADEPro
16/07/2015
42
Key benefit: The GRADE evidence profile
16/07/2015
43
Another view: Summary of findings table
16/07/2015
44
Key benefits of GRADE in SR
• For healthcare professionals/consumers
–
–
–
–
Rely on standardized quality of evidence ratings
Better understand the effect measures
Likely to increase SR value and use
Better information improves shared decision-making
• For guideline authors
– Utilize full GRADE evidence profiles
• For policy-makers
– Inform the process of determining net benefits (which
may include resource use considerations, values and
preferences)
16/07/2015
45
Session Plan
•
•
•
•
•
•
•
•
What does reader want from an SR?
Grading the evidence: different approaches
Background about GRADE
Quality of evidence exercise
GRADE in practice
Is GRADE perfect?
Going from evidence to recommendations
Q&As
16/07/2015
46
GRADE’s limitations
• Evidence rating for alternative management
strategies, not risk or prognosis per se.
• Does not eliminate disagreements in
interpreting the evidence – judgements on
thresholds continue to be necessary
• Requires some training in methodology to
be applied optimally
16/07/2015
47
What GRADE isn’t
• Not another ‘risk of bias’ tool
• Not a quantitative system (no scoring
required)
• Not eliminate COI, but able to minimize
• Not ‘expensive’
– Builds on well-established principles of EBM
– Some degree of training is needed for any system
– Proportionally adds minimal amount of extra time
to a systematic review
16/07/2015
48
Session Plan
•
•
•
•
•
•
•
•
What does reader want from an SR?
Grading the evidence: different approaches
Background about GRADE
Quality of evidence exercise
GRADE in practice
Is GRADE perfect?
Going from evidence to recommendations
Q&As
16/07/2015
49
Grading recommendations
• High-quality evidence = Strong recommendation?
• Think of 5 factors that would strengthen a
recommendation to treat?
or
• Think of 5 factors that would weaken a
recommendation to treat?
16/07/2015
50
Why Grade recommendations?
• Strong recommendations
– strong methods
– large precise effect
– few downsides of therapy
• Weak recommendations
–
–
–
–
16/07/2015
weak methods
imprecise estimate
small effect
substantial downsides
51
Critical
Outcome
Critical
Outcome
Important
Outcome
Low
High
Moderate
Low
Very low
Summary of findings
& estimate of effect
for each outcome
Systematic review
Grade down
Outcome
Grade up
P
I
C
O
RCT start high,
obs. data start
low
1. Risk of bias
2. Inconsistency
3. Indirectness
4. Imprecision
5. Publication
bias
1. Large effect
2. Dose
response
3. Confounders
Guideline development
Formulate recommendations:
• For or against (direction)
• Strong or weak (strength)
By considering:
 Quality of evidence
 Balance benefits/harms
 Values and preferences
Revise if necessary by considering:
 Resource use (cost)
16/07/2015
Rate
overall quality of evidence
across outcomes based on
lowest quality
of critical outcomes
•
•
•
•
“We recommend using…”
“We suggest using…”
“We recommend against using…”
“We suggest against using…”
52
An ideal world…
SR
RCTs
Moving towards standards
CPG
PRISMA 
CONSORT 
AMSTAR 
Obs.
Harms/safety?
Baseline risk?
Values/prefs?
16/07/2015
STROBE 
53
Conclusion
Gaining acceptance as international standard because
GRADE adds value:
1. GRADE has criteria for evidence assessment across
a range of questions and outcomes
2. Sensible, transparent, systematic
3. Balance between simplicity and methodological
rigor
16/07/2015
54
Session Plan
•
•
•
•
•
•
•
•
What does reader want from an SR?
Grading the evidence: different approaches
Background about GRADE
Quality of evidence exercise
GRADE in practice
Is GRADE perfect?
Going from evidence to recommendations
Q&As
16/07/2015
55
Any questions?
Thanks to Gordon Guyatt, Holger Schunemann and Yngve Falck-Ytter
for assistance for providing permission to reproduce slides
16/07/2015
56
GRADE exercise
DRAFT - not for circulation
57
DRAFT - not for circulation
58
DRAFT - not for circulation
59
DRAFT - not for circulation
60
DRAFT - not for circulation
61
DRAFT - not for circulation
62
DRAFT - not for circulation
63
DRAFT - not for circulation
64
DRAFT - not for circulation
65