Transcript Methodology for Guideline Development for the 7th ACCP

GRADE – An introduction and workshop
Holger Schünemann
Yngve Falck-Ytter
NICE, London December 11, 2006
Why bother about grading?



People draw conclusions about the
– quality of evidence
– strength of recommendations
Systematic and explicit approaches can help
– protect against errors
– resolve disagreements
– facilitate critical appraisal
– communicate information
However, there is wide variation in currently
used approaches and grading can be
misused or misunderstood
Which grading system?
Recommendation for use of oral
anticoagulation in patients with atrial
fibrillation and rheumatic mitral valve disease
Evidence
 B
 C+
 IV
Recommendation
Class I
1
C
Organization
 AHA
 ACCP
 SIGN
Grading Systems
Current profusion: can there be consensus?
GRADE
Grades of Recommendation
Assessment, Development and
Evaluation
About GRADE
o
o
o
o
o
Working group since 2000
Researchers/guideline developers with interest
in methodology
Aim: to develop a common system for
grading the quality of evidence and the
strength of recommendations that is sensible
and to explore the range of interventions and
contexts for which it might be useful*
Evaluation of existing systems and reliability*
Adopted by ATS, ACCP, ACP, WHO,
Cochrane
*Grade Working Group. CMAJ 2003, BMJ 2004, BMC 2004, BMC 2005
GRADE Working Group
David Atkins, chief medical officera
Dana Best, assistant professorb
Peter A Briss, chiefc
Martin Eccles, professord
Yngve Falck-Ytter, associate directore
Signe Flottorp, researcherf
Gordon H Guyatt, professorg
Robin T Harbour, quality and information director h
Margaret C Haugh, methodologisti
David Henry, professorj
Suzanne Hill, senior lecturerj
Roman Jaeschke, clinical professork
Gillian Leng, guidelines programme directorl
Alessandro Liberati, professorm
Nicola Magrini, directorn
James Mason, professord
Philippa Middleton, honorary research fellowo
Jacek Mrukowicz, executive directorp
Dianne O’Connell, senior epidemiologistq
Andrew D Oxman, directorf
Bob Phillips, associate fellowr
Holger J Schünemann, associate professorg,s
Tessa Tan-Torres Edejer, medical officer/scientistt
Helena Varonen, associate editoru
Gunn E Vist, researcherf
John W Williams Jr, associate professorv
Stephanie Zaza, project directorw
a) Agency for Healthcare Research and Quality, USA
b) Children's National Medical Center, USA
c) Centers for Disease Control and Prevention, USA
d) University of Newcastle upon Tyne, UK
e) German Cochrane Centre, Germany
f) Norwegian Centre for Health Services, Norway
g) McMaster University, Canada
h) Scottish Intercollegiate Guidelines Network, UK
i) Fédération Nationale des Centres de Lutte Contre le
Cancer, France
j) University of Newcastle, Australia
k) McMaster University, Canada
l) National Institute for Clinical Excellence, UK
m) Università di Modena e Reggio Emilia, Italy
n) Centro per la Valutazione della Efficacia della
Assistenza Sanitaria, Italy
o) Australasian Cochrane Centre, Australia
p) Polish Institute for Evidence Based Medicine,
Poland
q) The Cancer Council, Australia
r) Centre for Evidence-based Medicine, UK
s) National Cancer Institute, Italy
t) World Health Organisation, Switzerland
u) Finnish Medical Society Duodecim, Finland
v) Duke University Medical Center, USA
w) Centers for Disease Control and Prevention, USA
Guideline development process
Prioritise Problems, establish panel

Systematic Review

Evidence Profile

Relative importance of outcomes

Overall quality of evidence

Benefit – downside evaluation

Strength of recommendation
GRADE
Guideline development process
Prioritise Problems, establish panel

Systematic Review

Summary
Evidence Profile
of Findings

Relative importance of outcomes

GRADE
Overall quality of evidence

Benefit – downside evaluation

Strength of recommendation
GRADE
Quality of evidence
The extent to which one can be confident that
an estimate of effect or association is correct.
Although the degree of confidence is a
continuum, we suggest using four categories:
–
–
–
–
High
Moderate
Low
Very low
Judgements about the quality of
evidence
The quality of the evidence (i.e. our confidence)
depends on:




study design (e.g. RCT, case-control study)
study quality/limitations (protection against bias; e.g.
concealment of allocation, blinding, follow-up)
consistency of results
directness of the evidence including the
– populations (those of interest versus similar; for example,
older, sicker or more co-morbidity)
– interventions (those of interest versus similar; for example,
drugs within the same class)
– outcomes (important versus surrogate outcomes)
– comparison (A - C versus A - B & C - B)
Moving quality down
• poor (RCT) design, implementation
→randomization, blinding, concealment,
follow-up, intention to treat principle, early
stopping for benefit
• inconsistency
• Indirect evidence
→patients, interventions, outcomes
→A vs B, but have A to C, B to C
• sparse or imprecise data
• reporting bias
Moving quality up
Observational studies – high or moderate quality?



For example, plausible explanatory
factors that were not adjusted for in studies
comparing mortality
Strong association
rates of for-profit and not-for-profit
→ strong association: hospitals
RR > 2 orwould
RR < have
0.5
→ very strongreduced
association:
RR > 5 or
RR <Thus,
0.2 the
the observed
effect.
evidence showing that for-profit
Dose response relationship
hospitals
have
higher risk
mortality
is
– bleeding risk
associated
with aincreasing
INR of
(blood
thinning
with warfarin)
more convincing
Plausible confounders would have reduced the effect
Quality of
evidence
High
Study design
Lower if
Higher if
Randomised trial
Study quality:
-1 Serious
limitations
-2 Very serious
limitations
Strong
association:
+1 Strong, no
plausible
confounders,
consistent and
direct evidence
+2 Very strong, no
major threats to
validity and
direct
evidence
Moderate
Low
Observational
study
Very low
Any other evidence
-1 Important
inconsistency
Directness:
-1 Some
uncertainty
-2 Major
uncertainty
-1 Sparse data
-1 High probability
of Reporting
bias
+1 Evidence of a
Dose response
gradient
+1 All plausible
confounders
would have
reduced the
effect
Categories of quality




High: Further research is very unlikely to
change our confidence in the estimate of
effect.
Moderate: Further research is likely to
have an important impact on our
confidence in the estimate of effect and
may change the estimate.
Low: Further research is very likely to
have an important impact on our
confidence in the estimate of effect and is
likely to change the estimate.
Very low: Any estimate of effect is very
uncertain.












Judgements about the overall
quality of evidence

Most systems not explicit

Options:
–
–
–
–


Benefits
Primary outcome
Highest
Lowest
Based on lowest of all the critical outcomes
Beyond the scope of a systematic review
Levels of evidence: SIGN
Grading of recommendations
Some problems






Oversimplified hierarchy based on study design
– Inadequate consideration of other factors
– Distinction between study design, quality of evidence
and strength of recommendation blurred
Systematic reviews included in the hierarchy
– rather than viewed as the basis for making judgements
Expert opinion included in the hierarchy
– rather than explicitly considering the evidence
underlying expert opinions
Balance between desirable and undesirable effects
– Not reflected in the grade
– Not considered transparently
Inadequate consideration of other factors that affect
confidence in a recommendation
Grading misused when recommendation not separated
from the quality of the evidence
Example
WHO Avian Influenza guidelines - key
clinical questions:
Population: H5N1 infected individuals
Intervention: Neuraminidase Inhibitors, M2
Inhibitors, other pharmacological agents
Comparison: no therapy/alternative
management
Outcomes: ?
Example
WHO Avian Influenza guidelines - key
clinical questions:
Population: H5N1 infected individuals
Intervention: Neuraminidase Inhibitors, M2
Inhibitors, other pharmacological agents
Comparison: no therapy/alternative
management
Outcomes: Mortality?, Hospitalizations?
Resource use?, Adverse outcomes?
Clinical Question Refinement
Survey of panel members
Outcome definition:
• List of potential outcomes circulated
• Feedback from panel
• Concealed rating of importance
• Consultation with Cochrane Consumers
network
Outcomes/endpoints
Judgment about the relative importance for each
endpoint (scale from 9 to 1):
• 7 – 9: the endpoint is critical for decision making.
• 4 – 6: the endpoint is important but not critical.
• 1 – 3: the endpoint is not important.
Only critical (and important) outcomes included
Treatment: mortality, duration of hospitalization,
incidence of lower respiratory tract complications,
antiviral drug resistance and serious adverse events.
Evidence Summary and
Quality Ratings
• Draft summaries sent to panel members for
review and identification of gaps
• Restricted additional evidence at meeting
• Evidence profiles using GRADE methodology
and GRADEpro software (v1.12)
Evidence Profiles
Oseltamivir for treatment of H5N1 infection:
Summary of findings
Quality assessment
No of studies
(Ref)
Design
Limitations
Consistency
No of patients
Other
considerations
Directness
Effect
Oseltamivir
Placebo
Relative
(95% CI)
Absolute
(95% CI)
Quality
Importance
Healthy adults:
Mortality
0
Hospitalisation (Hospitalisations from influenza – influenza cases only)
-
-
-
-
-
5
(TJ 06)
Imprecise or
sparse data (-1)
-
-
OR 0.22
(0.02 to 2.16)
-

Very low
6
-
-
-
-
-
-
7
2/982
(0.2%)
9/662
(1.4%)
RR 0.149
(0.03 to 0.69)
-

Very low
8
Randomised
trial
No limitations One trial only
-
Major
uncertainty
(-2)1
9
Duration of hospitalization
0
LRTI (Pneumonia - influenza cases only)
5
(TJ 06)
Randomised
trial
-
No limitations One trial only
-
Major
uncertainty
(-2)1
Imprecise or
sparse data (-1)2
Duration of disease (Time to alleviation of symptoms/median time to resolution of symptoms – influenza cases only)
Randomised
53
No limitations4 Important
trials
inconsistency
(TJ 06)
(DT 03)
(-1)5
Viral shedding (Mean nasal titre of excreted virus at 24h)
26
(TJ 06)
Randomised
trials
No limitations
-7
Major
uncertainty
(-2)1
-
-
-
HR 1.303
(1.13 to 1.50)
-

Very low
5
Major
uncertainty
(-2)1
None
-
-
-
WMD -0.738
(-0.99 to -0.47)

Low
4
-
-
-
-
-
-
4
-
-
-
-
-
-
7
-
-
-
-
-
-
7
Imprecise or
sparse data (-1)14
-
-
OR range15
(0.56 to 1.80)
-

Low
-
-
-
-
-
-
Outbreak control
0
Resistance
-
-
-
-
0
Serious adverse effects (Mention of significant or serious adverse effects)
09
Minor adverse effects
311
(TJ 06)
10
-
-
-
(number and seriousness of adverse effects)
Randomised
trials
No limitations
-12
Some
uncertainty
(-1)13
Cost of drugs
0
-
-
-
-
4
Evidence Summary
Summary of findings
• No clinical trial of oseltamivir for treatment of H5N1
patients.
• 4 systematic reviews and health technology assessments
(HTA) reporting on 5 studies of oseltamivir in seasonal
influenza.
• Healthy adults, high risk adults or children for treatment of
seasonal influenza
• Duration of treatment up to 5 days
• Several countries in the northern and southern hemispheres (no
resource poor countries)
• 3 published case series describing H5N1 patients treated
with oseltamivir.
• Many in vitro and animal studies.
• < 400 cases; mortality > 50% worldwide
Case scenario
A 13 year old girl who lives in rural Indonesia
presented with flu symptoms and developed
severe respiratory distress over the course of
the last 2 days. She required intubation. The
history reveals that she shares her living
quarters with her parents and her three
siblings. At night the family’s chicken stock
shares this room too and several chicken
had died unexpectedly a few days before the
girl fell sick.
Who would recommend oseltamivir
for this patient or similar patients?



YES (pink)
No (green)
Recommendations
Recommendation: In patients with confirmed or
strongly suspected infection with avian influenza A
(H5N1) virus, clinicians should administer oseltamivir
treatment as soon as possible (????? recommendation,
very low quality evidence).
Recommendations
Recommendation: In patients with confirmed or
strongly suspected infection with avian influenza A
(H5N1) virus, clinicians should administer oseltamivir
treatment as soon as possible (strong recommendation,
very low quality evidence).
Comparison of GRADE and other systems











Explicit definitions
Explicit, sequential judgements
Components of quality defined
Quality by outcome and overall quality
Relative importance of outcomes
Balance between health benefits and harms
Balance between incremental health benefits
and costs
Evidence profiles
International collaboration
Consistent judgements?
Communication?
GRADE Profiler