Transcript Globalize the evidence – individualize Decisions

Holger Schünemann, MD, PhD
Professor
Utrecht, NL
September 18 - 19, 2008
GRADE FROM EVIDENCE TO
RECOMMENDATIONS
1
Evidence based clinical
decisions
Patient values
and preferences
Clinical state and
circumstances
Expertise
Research evidence
Equal for all
Haynes et al. 2002
The GRADE approach
Separation of 2 issues:
1) 4 categories of quality of evidence: very low, low,
moderate, or high quality?



methodological quality of evidence
likelihood of bias
by outcome and across outcomes
2) Recommendation: 2 grades - weak or strong (for
or against)?
 Quality of evidence only one factor
www.GradeWorkingGroup.org
Grades of recommendation:
Strength of recommendations
Strong recommendations
 high quality methods with large precise effect
 benefits much greater than downsides, or downsides
much greater than benefits
 we recommend
 Grade 1
Weak/conditional recommendations
 lower quality methods
 benefits not clearly greater or smaller than downsides
 values and preferences uncertain or very variable
 we suggest
 Grade 2
Case scenario
A 13 year old girl who lives in rural Indonesia presented
with flu symptoms and developed severe respiratory
distress over the course of the last 2 days. She
required intubation. The history reveals that she
shares her living quarters with her parents and her
three siblings. At night the family’s chicken stock
shares this room too and several chicken had died
unexpectedly a few days before the girl fell sick.
Relevant clinical question?
Example from a not so common disease
Clinical question:
Population:
Avian Flu/influenza A (H5N1) patients
Intervention: Oseltamivir (or Zanamivir)
Comparison: No pharmacological intervention
Outcomes:
Mortality, hospitalizations,
resource use, adverse outcomes,
antimicrobial resistance
Schunemann et al. The Lancet ID, 2007
Methods – WHO Rapid Advice Guidelines
for management of Avian Flu
 Applied findings of a recent systematic evaluation of
guideline development for WHO/ACHR
 Group composition (including panel of 13 voting
members):





clinicians who treated influenza A(H5N1) patients
infectious disease experts
basic scientists
public health officers
methodologists
 Independent scientific reviewers:
 Identified systematic reviews, recent RCTs, case series,
animal studies related to H5N1 infection
Evidence Profile
Oseltamivir for treatment of H5N1 infection:
Summary of findings
Quality assessment
No of studies
(Ref)
Design
Limitations
Consistency
No of patients
Other
considerations
Directness
Effect
Oseltamivir
Placebo
Relative
(95% CI)
Absolute
(95% CI)
Quality
Importance
Healthy adults:
Mortality
0
Hospitalisation (Hospitalisations from influenza – influenza cases only)
-
-
-
-
-
5
(TJ 06)
Imprecise or
sparse data (-1)
-
-
OR 0.22
(0.02 to 2.16)
-

Very low
6
-
-
-
-
-
-
7
2/982
(0.2%)
9/662
(1.4%)
RR 0.149
(0.03 to 0.69)
-

Very low
8
Randomised
trial
No limitations One trial only
-
Major
uncertainty
(-2)1
9
Duration of hospitalization
0
LRTI (Pneumonia - influenza cases only)
5
(TJ 06)
Randomised
trial
-
No limitations One trial only
-
Major
uncertainty
(-2)1
Imprecise or
sparse data (-1)2
Duration of disease (Time to alleviation of symptoms/median time to resolution of symptoms – influenza cases only)
Randomised
53
No limitations4 Important
trials
inconsistency
(TJ 06)
(DT 03)
(-1)5
Viral shedding (Mean nasal titre of excreted virus at 24h)
26
(TJ 06)
Randomised
trials
No limitations
-7
Major
uncertainty
(-2)1
-
-
-
HR 1.303
(1.13 to 1.50)
-

Very low
5
Major
uncertainty
(-2)1
None
-
-
-
WMD -0.738
(-0.99 to -0.47)

Low
4
-
-
-
-
-
-
4
-
-
-
-
-
-
7
-
-
-
-
-
-
7
Imprecise or
sparse data (-1)14
-
-
OR range15
(0.56 to 1.80)
-

Low
-
-
-
-
-
-
Outbreak control
0
Resistance
-
-
-
-
0
Serious adverse effects (Mention of significant or serious adverse effects)
09
Minor adverse effects
311
(TJ 06)
10
-
-
-
(number and seriousness of adverse effects)
Randomised
trials
No limitations
-12
Some
uncertainty
(-1)13
Cost of drugs
0
-
-
-
-
4
Oseltamivir for Girl with
Avian Flu
Summary of findings:
 No clinical trial of oseltamivir for treatment of
H5N1 patients.
 4 systematic reviews and health technology
assessments (HTA) reporting on 5 studies of
oseltamivir in seasonal influenza.
 Hospitalization: OR 0.22 (0.02 – 2.16)
 Pneumonia: OR 0.15 (0.03 - 0.69)




3 published case series.
Many in vitro and animal studies.
No alternative that is more promising at present.
Cost: ~ Euro 40$ per treatment course
Determinants of the strength
of recommendation
Factors that can strengthen a Comment
recommendation
Quality of the evidence
The higher the quality of evidence, the
more likely is a strong
recommendation.
Balance between desirable
The larger the difference between the
and undesirable effects
desirable and undesirable
consequences, the more likely a strong
recommendation warranted. The
smaller the net benefit and the lower
certainty for that benefit, the more likely
weak recommendation warranted.
Values and preferences
The greater the variability in values and
preferences, or uncertainty in values
and preferences, the more likely weak
recommendation warranted.
Costs (resource allocation)
The higher the costs of an intervention
– that is, the more resources
consumed – the less likely is a strong
recommendation warranted
Example: Oseltamivir for
Avian Flu
Recommendation: In patients with confirmed or
strongly suspected infection with avian influenza A
(H5N1) virus, clinicians should administer
oseltamivir treatment as soon as possible (?????
recommendation, very low quality evidence).
Schunemann et al. The Lancet ID, 2007
Example: Oseltamivir for
Avian Flu
Recommendation: In patients with confirmed or
strongly suspected infection with avian influenza A
(H5N1) virus, clinicians should administer
oseltamivir treatment as soon as possible (strong
recommendation, very low quality evidence).
Values and Preferences
Remarks: This recommendation places a high
value on the prevention of death in an illness
with a high case fatality. It places relatively low
values on adverse reactions, the development
of resistance and costs of treatment.
Schunemann et al. The Lancet ID, 2007
Other explanations
Remarks: Despite the lack of controlled treatment
data for H5N1, this is a strong recommendation, in
part, because there is a lack of known effective
alternative pharmacological interventions at this
time.
The panel voted on whether this recommendation
should be strong or weak and there was one
abstention and one dissenting vote.
ACCP: Acute coronary syndrome
Would a recommendation ignoring V & P be
possible?
For all patients presenting with NSTE ACS, without
a clear allergy to aspirin, we recommend
immediate aspirin, 75 to 325 mg po, and then daily,
75 to 162 mg po (strong recommendation, high
quality evidence).
Value sensitive
recommendation
 Idiopathic deep venous thrombosis (DVT) is potentially life
threatening condition
 Patients usually receive blood thinners for one year
 Continuing therapy will reduce a patients absolute risk for
recurrent DVT by 7% per year for several years
 The burdens include:
 taking a blood thinner daily
 keeping dietary intake of vitamin K constant
 blood tests to monitor the intensity of anticoagulation
 living with increased risk of minor and major bleeding
Value sensitive
recommendations
→ Patients who are very averse to a recurrent DVT would
consider the benefits of avoiding DVT worth the downsides of
taking warfarin. Other patients are likely to consider the benefit
not worth the harms and burden.
For patients with idiopathic DVT, without elevated bleeding risk,
we suggest long term warfarin therapy (weak recommendation,
high quality evidence).
Values and preferences: this recommendation ascribes a low
value to bleeding complications and burden from therapy and a
high value to avoiding DVTs
Quality assessment criteria
Quality of
evidence
High
Study design
Lower if
Higher if
Randomised trial
Study quality:
Serious
limitations
Very serious
limitations
Strong association:
Strong, no
plausible
confounders
Very strong,
no major
threats to
validity
Moderate
Low
Observational
study
Very low
Any other
evidence
Important
inconsistency
Directness:
Some
uncertainty
Major
uncertainty
Sparse or
imprecise data
High probability
of reporting bias
Evidence of a
Dose response
gradient
All plausible
confounders
would have
reduced the
effect
Strength of recommendation
 “The strength of a recommendation reflects
the extent to which we can, across the range
of patients for whom the recommendations
are intended, be confident that desirable
effects of a management strategy outweigh
undesirable effects.”
 Strong or weak
Quality of evidence &
strength of recommendation
 Linked but no automatism
 Other factors beyond the quality of evidence
influence our confidence that adherence to a
recommendation causes more benefit than harm
 Systems/approaches failed to make this explicit
 GRADE separates quality of evidence from
strength of recommendation
Implications of
a strong recommendation
 Patients: Most people in this situation would want
the recommended course of action and only a small
proportion would not
 Clinicians: Most patients should receive the
recommended course of action
 Policy makers: The recommendation can be
adapted as a policy in most situations
Implications of
a weak recommendation
 Patients: The majority of people in this situation
would want the recommended course of action,
but many would not
 Clinicians: Be prepared to help patients to make a
decision that is consistent with their own
values/decision aids and shared decision making
 Policy makers: There is a need for substantial
debate and involvement of stakeholders
Respiratory disease guidelines ?
Factors determining strength
of recommendation
Factors that can strengthen a Comment
recommendation
Quality of the evidence
The higher the quality of evidence, the
more likely is a strong
recommendation.
Balance between desirable and The larger the difference between the
undesirable effects
desirable and undesirable
consequences, the more likely a strong
recommendation warranted. The
smaller the net benefit and the lower
the certainty for that benefit, the more
likely is a weak recommendation.
Values and preferences
The greater the variability in values and
preferences, or uncertainty in values
and preferences, the more likely weak
recommendation warranted.
Costs (resource allocation)
The higher the costs of an intervention
– that is, the more resources
consumed – the less likely is a strong
recommendation warranted
Values & Preferences
 Patients’ perspectives, beliefs, expectations,
and goals for health and life.
 Underlying processes used in considering the
benefits, harms, costs, and inconveniences
patients will experience with each
management option and the resulting
preferences for each option.
Relative importance of
outcomes and management
approaches
 Guideline panels should be explicit about the relative
value they place on the range of relevant patientimportant outcomes. If values and preferences vary
widely, a strong recommendation becomes less likely
 Example: Patients vary widely in their view of how
aversive they find the risk of a stroke versus the risk of
a gastrointestinal bleed when deciding about oral
anticoagulation for atrial fibrillation.
Desirable and undesirable
effects
 desirable effects
 Mortality
 improvement in quality of life, fewer hospitalizations
 reduction in the burden of treatment
 reduced resource expenditure
 undesirable consequences
 deleterious impact on morbidity, mortality or quality
of life, increased resource expenditure
Conclusion
 clinicians, policy makers need summaries
 quality of evidence
 strength of recommendations
 explicit rules
 transparent, informative
 GRADE





four categories of quality of evidence
two grades for strength of recommendations
transparent, systematic by and across outcomes
applicable to diagnosis
wide adoption
28
Design and Execution
 limitations
 lack of concealment
 intention to treat principle violated
 inadequate blinding
 loss to follow-up
 early stopping for benefit
 Example: RCT suggests that danaparoid sodium is of
benefit in treating HIT complicated by thrombosis
 Key outcome: clinicians’ assessment of when the
thromboembolism had resolved
 Not blinded – subjective judgement
Consistency of results
 consistency of results
 if inconsistency, look for explanation
 patients, intervention, outcome, methods
 unexplained inconsistency downgrade quality
 Bleeding in thrombosis-prophylaxed hospitalized
patients
 seven RCTs
 4 lower, 3 higher risk
Example: Bleeding in the
hospital
Dentali et al. Ann Int Med, 2007
Directness of Evidence
 indirect comparisons
 interested in A versus B
 have A versus C and B versus C
 formoterol versus salmeterol versus tiotropium
 differences in
 patients (mild versus severe COPD)
 interventions (all inhaled steroids)
 outcomes (long-term health-related quality of life, short –
term functional capacity, laboratory exercise, spirometry)
Reporting Bias &
Imprecision
 reporting bias
 reporting of studies
 publication bias
 number of small studies
 reporting of outcomes
 small sample size
 small number of events
 wide confidence intervals
 uncertainty about magnitude of effect
Example: Bleeding in the
hospital
Dentali et al. Ann Int Med, 2007
What can raise quality?
 large magnitude can upgrade (RRR 50%)
 very large two levels (RRR 80%)
 common criteria
 everyone used to do badly
 almost everyone does well
 Oral anticoagulation for mechanical heart valves
 dose response relation
(higher INR – increased bleeding)
Questions for you
 Are systematic reviews for every recommendation
in your guidelines a reality/possibility?
 What about cost – how do you deal with cost and
how should we deal with it?
Content
 Study design – bias
 Levels/quality of evidence - GRADE
 Guidelines/Recommendations
Content
 Study design – bias
 Levels/quality of evidence - GRADE
 Guidelines/Recommendations
Confidence in evidence
 There always is evidence
 “When there is a question there is evidence”
 Better research  greater confidence in the
evidence and decisions
 Evidence alone is never sufficient to make a
clinical decision
Evidence based clinical
decisions
Patient values
and preferences
Clinical state and
circumstances
Expertise
Research evidence
Equal for all
Haynes et al. 2002
About GRADE
 Since 2000
 Researchers/guideline developers with interest
in methodology
 Aim: to develop a common, transparent and
sensible system for grading the quality of
evidence and the strength of
recommendations
 Evaluation of existing systems
GRADE Evidence Profiles
The GRADE approach
Clear separation of 2 issues:
1) 4 categories of quality of evidence: very low, low,
moderate, or high quality?



methodological quality of evidence
likelihood of bias
by outcome and across outcomes
2) Recommendation: 2 grades - weak or strong (for
or against)?
 Quality of evidence only one factor
*www.GradeWorking-Group.org
Determinants of quality
 RCTs start high
 observational studies start low
 what can lower quality?
1.
2.
3.
4.
5.
detailed design and execution
inconsistency
indirectness
reporting bias
imprecision
45
The GRADE approach
Separation of 2 issues:
1) 4 categories of quality of evidence: very low, low,
moderate, or high quality?



methodological quality of evidence
likelihood of bias
by outcome and across outcomes
2) Recommendation: 2 grades - weak or strong (for
or against)?
 Quality of evidence only one factor
www.GradeWorkingGroup.org
Grades of recommendation:
Strength of recommendations
Strong recommendations
 high quality methods with large precise effect
 benefits much greater than downsides, or downsides
much greater than benefits
 we recommend
 Grade 1
Weak/conditional recommendations
 lower quality methods
 benefits not clearly greater or smaller than downsides
 values and preferences uncertain or very variable
 we suggest
 Grade 2
Case scenario
A 13 year old girl who lives in rural Indonesia presented
with flu symptoms and developed severe respiratory
distress over the course of the last 2 days. She
required intubation. The history reveals that she
shares her living quarters with her parents and her
three siblings. At night the family’s chicken stock
shares this room too and several chicken had died
unexpectedly a few days before the girl fell sick.
Relevant clinical question?
Example from a not so common disease
Clinical question:
Population:
Avian Flu/influenza A (H5N1) patients
Intervention: Oseltamivir (or Zanamivir)
Comparison: No pharmacological intervention
Outcomes:
Mortality, hospitalizations,
resource use, adverse outcomes,
antimicrobial resistance
Schunemann et al. The Lancet ID, 2007
Methods – WHO Rapid Advice Guidelines
for management of Avian Flu
 Applied findings of a recent systematic evaluation of
guideline development for WHO/ACHR
 Group composition (including panel of 13 voting
members):





clinicians who treated influenza A(H5N1) patients
infectious disease experts
basic scientists
public health officers
methodologists
 Independent scientific reviewers:
 Identified systematic reviews, recent RCTs, case series,
animal studies related to H5N1 infection
Evidence Profile
Oseltamivir for treatment of H5N1 infection:
Summary of findings
Quality assessment
No of studies
(Ref)
Design
Limitations
Consistency
No of patients
Other
considerations
Directness
Effect
Oseltamivir
Placebo
Relative
(95% CI)
Absolute
(95% CI)
Quality
Importance
Healthy adults:
Mortality
0
Hospitalisation (Hospitalisations from influenza – influenza cases only)
-
-
-
-
-
5
(TJ 06)
Imprecise or
sparse data (-1)
-
-
OR 0.22
(0.02 to 2.16)
-

Very low
6
-
-
-
-
-
-
7
2/982
(0.2%)
9/662
(1.4%)
RR 0.149
(0.03 to 0.69)
-

Very low
8
Randomised
trial
No limitations One trial only
-
Major
uncertainty
(-2)1
9
Duration of hospitalization
0
LRTI (Pneumonia - influenza cases only)
5
(TJ 06)
Randomised
trial
-
No limitations One trial only
-
Major
uncertainty
(-2)1
Imprecise or
sparse data (-1)2
Duration of disease (Time to alleviation of symptoms/median time to resolution of symptoms – influenza cases only)
Randomised
53
No limitations4 Important
trials
inconsistency
(TJ 06)
(DT 03)
(-1)5
Viral shedding (Mean nasal titre of excreted virus at 24h)
26
(TJ 06)
Randomised
trials
No limitations
-7
Major
uncertainty
(-2)1
-
-
-
HR 1.303
(1.13 to 1.50)
-

Very low
5
Major
uncertainty
(-2)1
None
-
-
-
WMD -0.738
(-0.99 to -0.47)

Low
4
-
-
-
-
-
-
4
-
-
-
-
-
-
7
-
-
-
-
-
-
7
Imprecise or
sparse data (-1)14
-
-
OR range15
(0.56 to 1.80)
-

Low
-
-
-
-
-
-
Outbreak control
0
Resistance
-
-
-
-
0
Serious adverse effects (Mention of significant or serious adverse effects)
09
Minor adverse effects
311
(TJ 06)
10
-
-
-
(number and seriousness of adverse effects)
Randomised
trials
No limitations
-12
Some
uncertainty
(-1)13
Cost of drugs
0
-
-
-
-
4
Oseltamivir for Girl with
Avian Flu
Summary of findings:
 No clinical trial of oseltamivir for treatment of
H5N1 patients.
 4 systematic reviews and health technology
assessments (HTA) reporting on 5 studies of
oseltamivir in seasonal influenza.
 Hospitalization: OR 0.22 (0.02 – 2.16)
 Pneumonia: OR 0.15 (0.03 - 0.69)




3 published case series.
Many in vitro and animal studies.
No alternative that is more promising at present.
Cost: ~ Euro 40$ per treatment course
Determinants of the strength
of recommendation
Factors that can strengthen a Comment
recommendation
Quality of the evidence
The higher the quality of evidence, the
more likely is a strong
recommendation.
Balance between desirable
The larger the difference between the
and undesirable effects
desirable and undesirable
consequences, the more likely a strong
recommendation warranted. The
smaller the net benefit and the lower
certainty for that benefit, the more likely
weak recommendation warranted.
Values and preferences
The greater the variability in values and
preferences, or uncertainty in values
and preferences, the more likely weak
recommendation warranted.
Costs (resource allocation)
The higher the costs of an intervention
– that is, the more resources
consumed – the less likely is a strong
recommendation warranted
Example: Oseltamivir for
Avian Flu
Recommendation: In patients with confirmed or
strongly suspected infection with avian influenza A
(H5N1) virus, clinicians should administer
oseltamivir treatment as soon as possible (?????
recommendation, very low quality evidence).
Schunemann et al. The Lancet ID, 2007
Example: Oseltamivir for
Avian Flu
Recommendation: In patients with confirmed or
strongly suspected infection with avian influenza A
(H5N1) virus, clinicians should administer
oseltamivir treatment as soon as possible (strong
recommendation, very low quality evidence).
Values and Preferences
Remarks: This recommendation places a high
value on the prevention of death in an illness
with a high case fatality. It places relatively low
values on adverse reactions, the development
of resistance and costs of treatment.
Schunemann et al. The Lancet ID, 2007
Other explanations
Remarks: Despite the lack of controlled treatment
data for H5N1, this is a strong recommendation, in
part, because there is a lack of known effective
alternative pharmacological interventions at this
time.
The panel voted on whether this recommendation
should be strong or weak and there was one
abstention and one dissenting vote.
ACCP: Acute coronary syndrome
Would a recommendation ignoring V & P be
possible?
For all patients presenting with NSTE ACS, without
a clear allergy to aspirin, we recommend
immediate aspirin, 75 to 325 mg po, and then daily,
75 to 162 mg po (strong recommendation, high
quality evidence).
Value sensitive
recommendation
 Idiopathic deep venous thrombosis (DVT) is potentially life
threatening condition
 Patients usually receive blood thinners for one year
 Continuing therapy will reduce a patients absolute risk for
recurrent DVT by 7% per year for several years
 The burdens include:
 taking a blood thinner daily
 keeping dietary intake of vitamin K constant
 blood tests to monitor the intensity of anticoagulation
 living with increased risk of minor and major bleeding
Value sensitive
recommendations
→ Patients who are very averse to a recurrent DVT would
consider the benefits of avoiding DVT worth the downsides of
taking warfarin. Other patients are likely to consider the benefit
not worth the harms and burden.
 For patients with idiopathic DVT, without elevated bleeding
risk, we suggest long term warfarin therapy (weak
recommendation, high quality evidence).
Values and preferences: this recommendation ascribes a low
value to bleeding complications and burden from therapy and a
high value to avoiding DVTs
Background to GRADEpro
People faced with a health care decision need summaries,
including information about the quality of evidence
 Detailed summaries: providing guidance for large
audiences, e.g. guideline panels – detailed profiles
 Usable shorter summaries: users of systematic reviews,
e.g. clinicians
How do I create a summary of
findings table or evidence profile
 GRADEpro – software to create SoF
 Import data from RevMan 5 into GRADEpro
 Create table – author makes suggestions
about information to present and GRADEs
the evidence
 Export table from GRADEpro and import into
RevMan 5
Creating a new GRADEpro
file
Profile groups
Profiles
Profiles: Questions
Importing a RevMan 5 file
of a systematic review
Imported data from RevMan 5 file:
• outcomes
• meta-analyses results
• bibliographic information
Managing outcomes to include a
maximum of 7
Entering/editing information for dichotomous
outcomes
Entering/editing information to grade the quality of the
evidence
77
2. Consistency of results
 Look for explanation for inconsistency
 patients, intervention, comparator, outcome, methods
 Judgment
 variation in size of effect
 overlap in confidence intervals
 statistical significance of heterogeneity
 I2
3. Directness of Evidence
 indirect comparisons
 interested in A versus B
 have A versus C and B versus C
 differences in
 patients
 interventions
 outcomes
Directness of Evidence
Table 5. Sources of likely indirectness of evidence
Source of indirectness Question of interest
Indirect comparison
Early emergency department
systemic corticosteroids to
treat acute exacerbations in
adult patients with asthma
Differences in
Anti-leukotrienes plus inhaled
populations
glucocorticosteroids vs.
inhaled glucocorticosteroids
alone to prevent asthma
exacerbations and nighttime
symptoms in patients with
chronic asthma and allergic
rhinitis.
Differences in
Avoidance of pet allergens in
intervention
non-allergic infants or
preschool children to prevent
development of allergy.
Differences in
outcomes of interest
Intranasal glucocorticosteroids
vs. oral H1-antihistamines in
children with seasonal allergic
rhinitis.
Example
Both oral and intravenous routes are
effective but there is no direct
comparison of these two routes of
administration in adults.
Trials that measured asthma
exacerbations and nighttime symptoms
did not include patients with allergic
rhinitis.
Available studies used multifaceted
interventions directed at multiple
potential risk factors in addition to pet
avoidance.
In the available study parents were rating
the symptoms and quality of life of their
teenage children, instead the children
themselves.
4. Reporting Bias
 reporting bias
 reporting of studies
 publication bias
 number of small studies
 reporting of outcomes
Example: a systematic review of topical treatments for seasonal allergic
conjunctivitis showed that patients using topical sodium cromoglycate
were more likely to perceive benefit than those using placebo. However,
only small trials reported clinically and statistically significant benefits of
active treatment, while a larger trial showed a much smaller and a
statistically not significant effect (Owen 2004 [53]). These findings suggest
that smaller studies demonstrating smaller effects might not have been
published.
5. Imprecision
 small sample size
 small number of events
 wide confidence intervals
 uncertainty about magnitude of effect