Transcript Globalize the evidence – individualize Decisions

Holger Schünemann, MD, PhD
Chair, Department of Clinical Epidemiology & Biostatistics
Professor of Clinical Epidemiology, Biostatistics and
Medicine
McMaster University, Hamilton, Ontario, Canada
THE GRADE SYSTEM
1. Formulating questions
Guidelines are a way of answering questions
about clinical, communication, organisational
or policy interventions, in the hope of improving
health care or health policy.
It is therefore helpful to structure a guideline in
terms of answerable questions.
WHO Guideline Handbook, 2008
Different types of questions
Background Questions
Definition: e.g. What is Human Papilloma Virus
(HPV) infection?
Mechanism: e.g. How does HPV cause cancer?
Foreground Questions
Efficacy: e.g. What is the efficacy of an HPV
vaccine?
Recommendations/decisions: e.g. e.g. Should we
use HPV vaccine?
Different types of questions
Background Questions
Definition: e.g. What is Human Papilloma Virus
(HPV) infection?
Mechanism: e.g. How does HPV cause cancer?
Actionable items
Foreground Questions
Efficacy: e.g. What is the efficacy of an HPV
vaccine?
Recommendations/decisions: e.g. e.g. Should we
use HPV vaccine?
2. Choosing outcomes
 Every decision comes with desirable and
undesirable consequences
 Developing recommendations must include a
consideration of desirable and undesirable
consequences
Desirable and undesirable
consequences
 desirable effects
 lower mortality
 improvement in quality of life, fewer hospitalizations
 reduction in the burden of treatment
 reduced resource expenditure
 undesirable consequences
 deleterious impact on morbidity, mortality or quality
of life, increased resource expenditure
Limitations of older
systems & approaches
 confuse quality of evidence with strength of
recommendations
 lack well-articulated conceptual framework
 criteria not comprehensive or transparent
 focus on single outcomes
Grades of Recommendation Assessment,
Development and Evaluation
GRADE
WORKING GROUP
CMAJ 2003, BMJ 2004, BMC 2004, BMC
2005, AJRCCM 2006, Chest 2006, BMJ
2008, Lancet ID 2007, PLOS Medicine 2007
GRADE Working Group
David Atkins, chief medical officera
a) Agency for Healthcare Research and Quality, USA
Dana Best, assistant professorb
b) Children's National Medical Center, USA
Martin Eccles, professord
c) Centers for Disease Control and Prevention, USA
Francoise Cluzeau, lecturerx
d) University of Newcastle upon Tyne, UK
Yngve Falck-Ytter,
Signe
associate directore
e) German Cochrane Centre, Germany
Flottorp, researcherf
Gordon H
f) Norwegian Centre for Health Services, Norway
Guyatt, professorg
Robin T Harbour, quality
g) McMaster University, Canada
and information director h
Margaret C Haugh, methodologisti
David Henry,
i) Fédération Nationale des Centres de Lutte Contre le Cancer, France
professorj
Suzanne Hill, senior
j) University of Newcastle, Australia
lecturerj
Roman Jaeschke, clinical
h) Scottish Intercollegiate Guidelines Network, UK
k) McMaster University, Canada
professork
l) National Institute for Clinical Excellence, UK
Regina Kunx, Associate Professor
m) Università di Modena e Reggio Emilia, Italy
Gillian Leng, guidelines programme directorl
n) Centro per la Valutazione della Efficacia della Assistenza Sanitaria, Italy
Alessandro Liberati, professorm
o) Australasian Cochrane Centre, Australia
Nicola
Magrini, directorn
p) Polish Institute for Evidence Based Medicine, Poland
James Mason, professord
q) The Cancer Council, Australia
Philippa Middleton, honorary research
Jacek Mrukowicz, executive
Dianne O’Connell, senior
fellowo
directorp
epidemiologistq
Andrew D Oxman, directorf
Bob Phillips, associate
fellowr
r) Centre for Evidence-based Medicine, UK
s) National Cancer Institute, Italy
t) World Health Organisation, Switzerland
u) Finnish Medical Society Duodecim, Finland
v) Duke University Medical Center, USA
Holger J Schünemann, professorg,s
w) Centers for Disease Control and Prevention, USA
Tessa Tan-Torres Edejer, medical officert
x) University of London, UK
David Tovey, Editory
Y) BMJ Clinical Evidence, UK
Jane Thomas, Lecturer, UK
Helena Varonen, associate editoru
Gunn E Vist, researcherf
John W Williams Jr, professorv
Stephanie Zaza, project directorw
The GRADE approach
Clear separation of 2 issues:
1) 4 categories of quality of evidence: very low, low,
moderate, or high quality?



methodological quality of evidence
likelihood of systematic deviation from truth
by outcome
2) Recommendation: 2 grades – conditional or
strong (for or against)?
 Quality of evidence only one factor
*www.GradeWorkingGroup.org
Determinants of quality
 RCTs start high
 observational studies start low
 5 factors lower the quality of evidence





limitations in detailed design and execution
inconsistency
indirectness
reporting bias
imprecision
 3 factors can increase the quality of evidence
Example: Limitations in Design
and Execution
 Limitations – observational studies
 Failure to develop and apply appropriate eligibility criteria 




under- or over-matching in case-control studies
Selection of exposed and unexposed in cohort studies from
different populations
Flawed measurement of both exposure and outcome (e.g. recall
bias in CC studies)
Differential surveillance for outcome in exposed and unexposed
in cohort studies
Failure to adequately measure/control for confounding
Failure to match for prognostic factors and/or adjustment in
statistical analysis
Quality assessment criteria
Quality of
evidence
High
Study design
Lower if
Higher if
Randomised trial
Study quality:
Serious
limitations
Very serious
limitations
Strong association:
Strong, no
plausible
confounders
Very strong,
no major
threats to
validity
Moderate
Low
Very low
Observational
study
Important
inconsistency
Directness:
Some
uncertainty
Major
uncertainty
Sparse or
imprecise data
High probability
of reporting bias
Evidence of a
Dose response
gradient
All plausible
confounders
would have
reduced the
effect
From Evidence to
Recommendation
14
Categories of
recommendations
Although the degree of confidence is a
continuum, we suggest using two categories:
strong and weak/conditional.
 Strong recommendation: the panel is
confident that the desirable effects of
adherence to a recommendation outweigh the
undesirable effects.
 Conditional recommendation: the panel
concludes that the desirable effects of
adherence to a recommendation probably
outweigh the undesirable effects, but is not
confident.
Recommend
 
Suggest
 
Determinants of the strength
of recommendation
Factors that can strengthen a Comment
recommendation
Quality of the evidence
The higher the quality of evidence, the
more likely is a strong
recommendation.
Balance between desirable
The larger the difference between the
and undesirable effects
desirable and undesirable
consequences, the more likely a strong
recommendation warranted. The
smaller the net benefit and the lower
certainty for that benefit, the more likely
weak recommendation warranted.
Values and preferences
The greater the variability in values and
preferences, or uncertainty in values
and preferences, the more likely weak
recommendation warranted.
Costs (resource allocation)
The higher the costs of an intervention
– that is, the more resources
consumed – the less likely is a strong
recommendation warranted
Judgements about the
strength of a recommendation
 No precise threshold for going from a strong to a weak
recommendation
 The presence of important concerns about one or more
of these factors make a weak recommendation more
likely.
 Panels should consider all of these factors and make the
reasons for their judgements explicit.
 Recommendations should specify the perspective that
is taken (e.g. individual patient, health system) and which
outcomes were considered (including which, if any
costs).
Finally: There are no RCTs!
 We will do a consensus statement/guideline
(and not use rigorous methods)
 Do you think that those using the
recommendations would like to be informed
about the basis (explanation) for a
recommendation if they were asked (by their
patients)?
 I suspect the answer is “yes”
18
There are no RCTs! Cont’d
 Another reason for using structured
approaches: any form of recommendation
needs agreement/consensus – whether based
on high or lower quality evidence (voting as a
forced form of consensus)
 Ergo: Consensus statement is a misnomer in
regards to differentiating from guideline
 The level of detail depends on other aspects:
 Funds, time, greater interest, higher priority
 Transparency is key
19
Conclusions
 Clinical practice guidelines should be based on the




best available evidence
GRADE provides a structure approach to improve
communication – official WHO system
Criteria for evidence assessment across questions and
outcomes
Criteria for moving from evidence to
recommendations
Transparent, systematic
 four categories of quality of evidence
 two grades for strength of recommendations
 Transparency in decision making and judgments is
key
What can raise quality?
2. dose response relation
 (higher INR – increased bleeding)
 childhood lymphoblastic leukemia




risk for CNS malignancies 15 years after cranial irradiation
no radiation: 1% (95% CI 0% to 2.1%)
12 Gy: 1.6% (95% CI 0% to 3.4%)
18 Gy: 3.3% (95% CI 0.9% to 5.6%)
3. all plausible confounding may be working to reduce the
demonstrated effect or increase the effect if no effect was
observed
All plausible confounding
would result in an underestimate of
the treatment effect
 Higher death rates in private for-profit versus
private not-for-profit hospitals
 patients in the not-for-profit hospitals likely sicker
than those in the for-profit hospitals
 for-profit hospitals are likely to admit a larger
proportion of well-insured patients than not-forprofit hospitals (and thus have more resources
with a spill over effect)
All plausible biases
would result in an overestimate of
effect



Hypoglycaemic drug phenformin causes
lactic acidosis
The related agent metformin is under
suspicion for the same toxicity.
Large observational studies have failed to
demonstrate an association
 Clinicians would be more alert to lactic acidosis in
the presence of the agent
Implications of a strong
recommendation
 Patients: Most people in your situation would
want the recommended course of action and
only a small proportion would not
 Clinicians: Most patients should receive the
recommended course of action
 Policy makers: The recommendation can be
adapted as a policy in most situations
Implications of a
weak/conditional recommendation
 Patients: The majority of people in your
situation would want the recommended
course of action, but many would not
 Clinicians: Be prepared to help patients to
make a decision that is consistent with their
own values
 Policy makers: There is a need for substantial
debate and involvement of stakeholders
Should oseltamivir be used for
treatment of patients
hospitalised with avian
influenza (H5N1)?
Should oseltamivir be used for treatment of patients
hospitalised with avian influenza (H5N1)?
Summary of findings
Transmission: No human to human transmission
Patient or population: Hospitalised, clinical and serologically confirmed cases of
avian influenza
Oseltamivir for Avian Flu
Summary of findings:
• No clinical trial of oseltamivir for treatment of H5N1 patients.
• 4 systematic reviews and health technology assessments (HTA)
reporting on 5 studies of oseltamivir in seasonal influenza.
• Hospitalization: OR 0.22 (0.02 – 2.16)
• Pneumonia: OR 0.15 (0.03 - 0.69)
• 3 published case series.
• Many in vitro and animal studies.
• No alternative that is more promising at present.
• Cost: ~ Euro 50 per treatment course
What would you recommend?
Factors that can strengthen a
recommendation
Quality of the evidence
Balance between desirable
and undesirable effects
Values and preferences
Costs (resource allocation)
Comment
The higher the quality of evidence, the
more likely is a strong
recommendation.
The larger the difference between the
desirable and undesirable
consequences, the more likely a strong
recommendation warranted. The
smaller the net benefit and the lower
certainty for that benefit, the more likely
weak recommendation warranted.
The greater the variability in values and
preferences, or uncertainty in values
and preferences, the more likely weak
recommendation warranted.
The higher the costs of an intervention
– that is, the more resources
consumed – the less likely is a strong
recommendation warranted
Strong recommendation: the panel is confident that the desirable effects
of adherence to a recommendation outweigh the undesirable effects.
Weak recommendation: the panel concludes that the desirable effects of
adherence to a recommendation probably outweigh the undesirable effects,
but is not confident.
Judgments about the strength of a recommendation oseltamivir for treatment of patients hospitalised
with avian influenza (H5N1)
Factors
Comments
Balance between desirable
and undesirable effects
“The benefits are uncertain, but
potentially large.”
Quality of the evidence
“The quality of the evidence is
very low.”
Values and preferences
“All patients and care providers
would accept treatment for
H5N1 disease.” No alternative
Costs (resource use)
“The cost is not high for
treatment of sporadic cases.”
Who would recommend oseltamivir for these
patients (no other alternative)?


YES (green card)
No (pink card)
Example: Oseltamivir for
Avian Flu
Recommendation: In patients with confirmed or
strongly suspected infection with avian influenza A
(H5N1) virus, clinicians should administer oseltamivir
treatment as soon as possible (????? recommendation,
very low quality evidence).
Schunemann et al., The Lancet ID, 2007
Example: Oseltamivir for
Avian Flu
Recommendation: In patients with confirmed or
strongly suspected infection with avian influenza A
(H5N1) virus, clinicians should administer oseltamivir
treatment as soon as possible (strong recommendation,
very low quality evidence).
Values and Preferences
Remarks: This recommendation places a high value on
the prevention of death in an illness with a high case
fatality. It places relatively low values on adverse
reactions, the development of resistance and costs of
Schunemann et al., The Lancet ID, 2007
treatment.
Other explanations
Remarks: Despite the lack of controlled treatment data
for H5N1, this is a strong recommendation, in part,
because there is a lack of known effective alternative
pharmacological interventions at this time.
The panel voted on whether this recommendation
should be strong or weak and there was one abstention
and one dissenting vote.
Schunemann et al., The Lancet ID, 2007