Transcript Overview of GRADE

1
Webcast 25
National Center
for the Dissemination
of Disability Research
Jan Brożek, MD, PhD
Clinical Epidemiology & Biostatistics
McMaster University, Hamilton, Ontario, Canada
Rating the Quality of Evidence and Strength of
Recommendations
for Disability and Rehabilitation Research
GRADE
2010
APR
14
1234567
8901234
5678901
2345678
2
D
Cochrane Collaboration
isclosure
GRADE working group
3
P
Why you should care?
lan
What is GRADE?
4
?
yesterday
today
tomorrow
5
GRADE
Working Group
• to develop a common, transparent and
sensible system for grading the quality of
evidence and the strength of
recommendations
• since 2000
• guideline developers, clinicians and
methodologists around the world
6


















World Health Organization
Allergic Rhinitis in Asthma Guidelines (ARIA)
American Thoracic Society
American College of Physicians
European Respiratory Society
European Society of Thoracic Surgeons
Centers for Disease Control and Prevention
British Medical Journal
Infectious Disease Society of America
American College of Chest Physicians
UpToDate©
NICE
Scottish Intercollegiate Guideline Network (SIGN)
Cochrane Collaboration
Clinical Evidence
Agency for Health Care Research and Quality
Partner of GIN
Over 40 major organizations *
* www.gradeworkinggroup.org
7
Guyatt GH et al.
BMJ 2008;336:294-6
8
9
O
question (in whom, what, in place of ...?)
utline
problem
guideline panel
outcomes (what for?)
evidence
systematic review
estimates of effects
quality of evidence
evidence table
solution
balance benefits & harms
values and preferences
recommendation and its strength
10
O
question (in whom, what, in place of ...?)
utline
problem
guideline panel
outcomes (what for?)
evidence
systematic review
estimates of effects
quality of evidence
evidence table
solution
balance benefits & harms
values and preferences
recommendation and its strength
Formulate
question
Select
outcomes
Rate
importance
Systematic Review
(outcomes across studies)
Evidence Profile
(GRADEpro)
1
2
Pooled estimate of effect for each outcome
Quality of evidence for each outcome
11
PICO
Critical
Outcome2
Critical
Outcome3
Important
Outcome4
Not
Formulate recommendations
For or against an action
Strong or weak (strength)
consider:
Balance benefits/downsides
Quality of evidence
Values and preferences
Resource use (cost)
recommendation
Guideline panel
RCT
observational
1.
2.
3.
4.
5.
1. large effect
2. dose-response
3. antagonistic bias
risk of bias
inconsistency
indirectness
imprecision
publication bias
Rate overall quality of evidence
across outcomes
Wording
 “We recommend…” | “Clinicians should…”
 “We suggest…” | “Clinicians might…”



start
rate down
action
Outcome1
rate up
High
 Moderate
High
| Moderate |Low
Low| Very
Very low
low
unambiguous
clear implications for action
transparent (values & preferences statement)
high
low
12
patient
intervention
13
Q
uestions
• What are the phenomena associated with the
problem?
• What is frequency of the problem?
• What causes the problem?
(aetiology)
• Does this person have the problem?
(diagnosis)
• What if one gets the problem?
(prognosis)
• What can we do about the problem?
(treatment)
• … etc.
14
Q
injection immunotherapy
be used
in adults with allergic rhinitis?
uestion
Should
15
Population
Intervention
(what?)
Comparison
(instead of what else?)
Outcomes
(what for?)
uestion
(in whom?)
Q
Formulate
question
Select
outcomes
Rate
importance
Systematic Review
(outcomes across studies)
Evidence Profile
(GRADEpro)
1
2
Pooled estimate of effect for each outcome
Quality of evidence for each outcome
16
PICO
Critical
Outcome2
Critical
Outcome3
Important
Outcome4
Not
Formulate recommendations
For or against an action
Strong or weak (strength)
consider:
Balance benefits/downsides
Quality of evidence
Values and preferences
Resource use (cost)
recommendation
Guideline panel
RCT
observational
1.
2.
3.
4.
5.
1. large effect
2. dose-response
3. antagonistic bias
risk of bias
inconsistency
indirectness
imprecision
publication bias
Rate overall quality of evidence
across outcomes
Wording
 “We recommend…” | “Clinicians should…”
 “We suggest…” | “Clinicians might…”



start
rate down
action
Outcome1
rate up
High
 Moderate
High
| Moderate |Low
Low| Very
Very low
low
unambiguous
clear implications for action
transparent (values & preferences statement)
high
low
17
Important to patients
symptoms
quality of life
adverse effects
cost
• anterior rhinomanometry
• eosinophil count
utcomes
•
•
•
•
O
Formulate
question
Select
outcomes
Rate
importance
Systematic Review
(outcomes across studies)
Evidence Profile
(GRADEpro)
1
2
Pooled estimate of effect for each outcome
Quality of evidence for each outcome
18
PICO
Critical
Outcome2
Critical
Outcome3
Important
Outcome4
Not
Formulate recommendations
For or against an action
Strong or weak (strength)
consider:
Balance benefits/downsides
Quality of evidence
Values and preferences
Resource use (cost)
recommendation
Guideline panel
RCT
observational
1.
2.
3.
4.
5.
1. large effect
2. dose-response
3. antagonistic bias
risk of bias
inconsistency
indirectness
imprecision
publication bias
Rate overall quality of evidence
across outcomes
Wording
 “We recommend…” | “Clinicians should…”
 “We suggest…” | “Clinicians might…”



start
rate down
action
Outcome1
rate up
High
 Moderate
High
| Moderate |Low
Low| Very
Very low
low
unambiguous
clear implications for action
transparent (values & preferences statement)
high
low
19
9
8
critical
7
6
5
important, but not critical
4
3
2
1
informative, but not important
Formulate
question
Select
outcomes
Rate
importance
Systematic Review
(outcomes across studies)
Evidence Profile
(GRADEpro)
1
2
Pooled estimate of effect for each outcome
Quality of evidence for each outcome
20
PICO
Critical
Outcome2
Critical
Outcome3
Important
Outcome4
Not
Formulate recommendations
For or against an action
Strong or weak (strength)
consider:
Balance benefits/downsides
Quality of evidence
Values and preferences
Resource use (cost)
recommendation
Guideline panel
RCT
observational
1.
2.
3.
4.
5.
1. large effect
2. dose-response
3. antagonistic bias
risk of bias
inconsistency
indirectness
imprecision
publication bias
Rate overall quality of evidence
across outcomes
Wording
 “We recommend…” | “Clinicians should…”
 “We suggest…” | “Clinicians might…”



start
rate down
action
Outcome1
rate up
High
 Moderate
High
| Moderate |Low
Low| Very
Very low
low
unambiguous
clear implications for action
transparent (values & preferences statement)
high
low
21
s
0,01
0,1
treatment better
1
10
control better
ystematic review
RR
22
s
ystematic review
Formulate
question
Select
outcomes
Rate
importance
Systematic Review
(outcomes across studies)
Evidence Profile
(GRADEpro)
1
2
Pooled estimate of effect for each outcome
Quality of evidence for each outcome
23
PICO
Critical
Outcome2
Critical
Outcome3
Important
Outcome4
Not
Formulate recommendations
For or against an action
Strong or weak (strength)
consider:
Balance benefits/downsides
Quality of evidence
Values and preferences
Resource use (cost)
recommendation
Guideline panel
RCT
observational
1.
2.
3.
4.
5.
1. large effect
2. dose-response
3. antagonistic bias
risk of bias
inconsistency
indirectness
imprecision
publication bias
Rate overall quality of evidence
across outcomes
Wording
 “We recommend…” | “Clinicians should…”
 “We suggest…” | “Clinicians might…”



start
rate down
action
Outcome1
rate up
High
 Moderate
High
| Moderate |Low
Low| Very
Very low
low
unambiguous
clear implications for action
transparent (values & preferences statement)
high
low
24
Question 32
E
vidence profile
Date: 2007-09-06
Question: Should subcutaneous allergen specific immunotherapy vs placebo be used in adults with allergic rhinitis?
Bibliography: Calderon M., Alves B., Jacobson M., Hurwitz B., Sheikh A., Durham S. Allergen injection immunotherapy for seasonal allergic rhinitis. Cochrane database of systematic reviews (Online),
2007:CD001936.
Summary of findings
Quality assessment
No of patients
Effect
Importance
Quality
No of
Other
subcutaneous allergen
Relative
Design
Limitations
Inconsistency
Indirectness
Imprecision
placebo
Absolute
studies
considerations
specific immunotherapy
(95% CI)
Symptoms (recorded in diary) (follow-up 3 days to 3 years1; Better indicated by less)
152
randomised no serious
no serious
no serious
no serious
none
SMD -0.73 (
597
466
CRITICAL
trial
limitations
inconsistency 3
indirectness
imprecision
0.97 to -0.5)
HIGH
1
Medication score (follow-up 3 days to 3 years ; Better indicated by less)
13
randomised no serious
no serious
serious4
no serious
none
SMD -0.57 (
549
414
IMPORTANT
trial
limitations
inconsistency 3
imprecision
0.33 to -0.82) MODERATE
1
symptom and medication score (follow-up 3 days to 3 years ; Better indicated by less)
8
randomised no serious
no serious
serious4
no serious
none
SMD -0.48 (
320
297
IMPORTANT
trial
limitations
inconsistency
imprecision
0.29 to -0.67) MODERATE
Nasal symptoms (follow-up 3 days to 3 years1; Better indicated by less)
96
randomised no serious
serious3
no serious
no serious
none
SMD -1.59 (
396
276
CRITICAL
trial
limitations
indirectness
imprecision
0.89 to -2.29)
HIGH
1
Bronchial symptoms (follow-up 3 days to 3 years ; Better indicated by less)
57
randomised no serious
no serious
no serious
serious5
none
SMD -0.59 (
266
163
IMPORTANT
trial
limitations
inconsistency 3
indirectness
0.11 to -1.06)
LOW
1
Ocular symptoms (follow-up 3 days to 3 years ; Better indicated by less)
38
randomised no serious
no serious
no serious
no serious
none
SMD -1.80 (
226
119
CRITICAL
trial
limitations
inconsistency
indirectness
imprecision
0.31 to -3.28)
HIGH
1
Global improvement (follow-up 3 days to 3 years ; Better indicated by less)
99
randomised no serious
–10
no serious
–10
none
0
0
not pooled9
IMPORTANT
trial
limitations
indirectness
1
Quality of life (follow-up 3 days to 3 years ; Better indicated by less)
5
randomised no serious
no serious
no serious
no serious
none
SMD -0.52 (
332
239
CRITICAL
trial
limitations
inconsistency
indirectness
imprecision
0.34 to -0.69)
HIGH
Adverse events (local reaction not requiring treatment) (follow-up 3 days to 3 years1)
24
randomised no serious
–10
no serious
–10
none
not
–/90711
–/69712
not pooled
IMPORTANT
trial
limitations
indirectness
pooled
1
Adverse events (local reaction requiring treatment) (follow-up 3 days to 3 years )
7
randomised no serious
no serious
no serious
serious13
none
not

–/20814
–/18615
not pooled
IMPORTANT
trial
limitations
inconsistency
indirectness
pooled
MODERATE
16
Adverse events (early mild systemic reaction [< 30 minutes]) (follow-up 3 days to 3 years; assessed with: mild systemic reactions )
17
randomised no serious
–10
no serious
–10
none
not
–/70617
–/56618
not pooled
CRITICAL
trial
limitations
indirectness
pooled
1
19
Adverse events (early severe systemic reaction [< 30 minutes]) (follow-up 3 days to 3 years ; assessed with: non life-threatening systemic reactions )
13
randomised no serious
–10
no serious
–10
none
not
–/61520
–/46321
not pooled
CRITICAL
trial
limitations
indirectness
pooled
25
Question 32
E
vidence profile
Date: 2007-09-06
Question: Should subcutaneous allergen specific immunotherapy vs placebo be used in adults with allergic rhinitis?
Bibliography: Calderon M., Alves B., Jacobson M., Hurwitz B., Sheikh A., Durham S. Allergen injection immunotherapy for seasonal allergic rhinitis. Cochrane database of systematic reviews (Online),
2007:CD001936.
Summary of findings
Quality assessment
No of patients
Effect
Importance
Quality
No of
Other
subcutaneous allergen
Relative
Design
Limitations
Inconsistency
Indirectness
Imprecision
placebo
Absolute
studies
considerations
specific immunotherapy
(95% CI)
Symptoms (recorded in diary) (follow-up 3 days to 3 years1; Better indicated by less)
152
randomised no serious
no serious
no serious
no serious
none
SMD -0.73 (
597
466
CRITICAL
trial
limitations
inconsistency 3
indirectness
imprecision
0.97 to -0.5)
HIGH
1
Medication score (follow-up 3 days to 3 years ; Better indicated by less)
13
randomised no serious
no serious
serious4
no serious
none
SMD -0.57 (
549
414
IMPORTANT
trial
limitations
inconsistency 3
imprecision
0.33 to -0.82) MODERATE
1
symptom and medication score (follow-up 3 days to 3 years ; Better indicated by less)
8
randomised no serious
no serious
serious4
no serious
none
SMD -0.48 (
320
297
IMPORTANT
trial
limitations
inconsistency
imprecision
0.29 to -0.67) MODERATE
Nasal symptoms (follow-up 3 days to 3 years1; Better indicated by less)
96
randomised no serious
serious3
no serious
no serious
none
SMD -1.59 (
396
276
CRITICAL
trial
limitations
indirectness
imprecision
0.89 to -2.29)
HIGH
1
Bronchial symptoms (follow-up 3 days to 3 years ; Better indicated by less)
57
randomised no serious
no serious
no serious
serious5
none
SMD -0.59 (
266
163
IMPORTANT
trial
limitations
inconsistency 3
indirectness
0.11 to -1.06)
LOW
1
Ocular symptoms (follow-up 3 days to 3 years ; Better indicated by less)
38
randomised no serious
no serious
no serious
no serious
none
SMD -1.80 (
226
119
CRITICAL
trial
limitations
inconsistency
indirectness
imprecision
0.31 to -3.28)
HIGH
1
Global improvement (follow-up 3 days to 3 years ; Better indicated by less)
99
randomised no serious
–10
no serious
–10
none
0
0
not pooled9
IMPORTANT
trial
limitations
indirectness
1
Quality of life (follow-up 3 days to 3 years ; Better indicated by less)
5
randomised no serious
no serious
no serious
no serious
none
SMD -0.52 (
332
239
CRITICAL
trial
limitations
inconsistency
indirectness
imprecision
0.34 to -0.69)
HIGH
Adverse events (local reaction not requiring treatment) (follow-up 3 days to 3 years1)
24
randomised no serious
–10
no serious
–10
none
not
–/90711
–/69712
not pooled
IMPORTANT
trial
limitations
indirectness
pooled
1
Adverse events (local reaction requiring treatment) (follow-up 3 days to 3 years )
7
randomised no serious
no serious
no serious
serious13
none
not

–/20814
–/18615
not pooled
IMPORTANT
trial
limitations
inconsistency
indirectness
pooled
MODERATE
16
Adverse events (early mild systemic reaction [< 30 minutes]) (follow-up 3 days to 3 years; assessed with: mild systemic reactions )
17
randomised no serious
–10
no serious
–10
none
not
–/70617
–/56618
not pooled
CRITICAL
trial
limitations
indirectness
pooled
1
19
Adverse events (early severe systemic reaction [< 30 minutes]) (follow-up 3 days to 3 years ; assessed with: non life-threatening systemic reactions )
13
randomised no serious
–10
no serious
–10
none
not
–/61520
–/46321
not pooled
CRITICAL
trial
limitations
indirectness
pooled
26
27
Formulate
question
Select
outcomes
Rate
importance
Systematic Review
(outcomes across studies)
Evidence Profile
(GRADEpro)
1
2
Pooled estimate of effect for each outcome
Quality of evidence for each outcome
28
PICO
Critical
Outcome2
Critical
Outcome3
Important
Outcome4
Not
Formulate recommendations
For or against an action
Strong or weak (strength)
consider:
Balance benefits/downsides
Quality of evidence
Values and preferences
Resource use (cost)
recommendation
Guideline panel
RCT
observational
1.
2.
3.
4.
5.
1. large effect
2. dose-response
3. antagonistic bias
risk of bias
inconsistency
indirectness
imprecision
publication bias
Rate overall quality of evidence
across outcomes
Wording
 “We recommend…” | “Clinicians should…”
 “We suggest…” | “Clinicians might…”



start
rate down
action
Outcome1
rate up
High
 Moderate
High
| Moderate |Low
Low| Very
Very low
low
unambiguous
clear implications for action
transparent (values & preferences statement)
high
low
29
„I figure there is a 40% chance of showers, and
a 10% chance we know what we’re talking about”
Formulate
question
Select
outcomes
Rate
importance
Systematic Review
(outcomes across studies)
Evidence Profile
(GRADEpro)
1
2
Pooled estimate of effect for each outcome
Quality of evidence for each outcome
30
PICO
Critical
Outcome2
Critical
Outcome3
Important
Outcome4
Not
Formulate recommendations
For or against an action
Strong or weak (strength)
consider:
Balance benefits/downsides
Quality of evidence
Values and preferences
Resource use (cost)
recommendation
Guideline panel
RCT
observational
1.
2.
3.
4.
5.
1. large effect
2. dose-response
3. antagonistic bias
risk of bias
inconsistency
indirectness
imprecision
publication bias
Rate overall quality of evidence
across outcomes
Wording
 “We recommend…” | “Clinicians should…”
 “We suggest…” | “Clinicians might…”



start
rate down
action
Outcome1
rate up
High
 Moderate
High
| Moderate |Low
Low| Very
Very low
low
unambiguous
clear implications for action
transparent (values & preferences statement)
high
low
31
risk of bias
• lack of concealment
• intention to treat principle violated
• inadequate blinding
• loss to follow-up
• early stopping for benefit
• selective outcome reporting
32
Regular treatment with salmeterol
for chronic asthma: serious adverse
events (Review)
Cates CJ, Cates MJ
33
risk of bias
overall judgment is required
34
inconsistency
/unexplained/
35
indirectness
• differences in
– populations/patients
– interventions
– outcomes
• indirect comparisons
– interested in A versus B
– have A versus C and B versus C
36
indirect comparison
Source of
Question of interest
indirectness
Indirect comparison Early emergency
department systemic
corticosteroids to treat
acute exacerbations in
adult patients with asthma
Example
Both oral and intravenous
routes are effective but there is
no direct comparison of these
two routes of administration in
adults.
37
different populations
Source of
indirectness
Question of interest
Example
Differences in
populations
Anti-leukotrienes plus
inhaled steroids vs. inhaled
steroids alone to prevent
asthma exacerbations and
nighttime symptoms in
patients with chronic asthma
and allergic rhinitis.
Trials that measured asthma
exacerbations and nighttime
symptoms did not include
patients with allergic rhinitis.
38
different intervention
Source of
indirectness
Question of interest
Example
Differences in
intervention
Avoidance of pet
allergens in non-allergic
infants or preschool
children to prevent
development of allergy.
Available studies used
multifaceted interventions
directed at multiple potential risk
factors in addition to pet
avoidance.
39
different outcomes
Source of
indirectness
Question of interest
Example
Differences in
outcomes of
interest
Intranasal steroids vs.
oral antihistamines in
children with seasonal
allergic rhinitis
In the available study parents
were rating the symptoms and
quality of life of their teenage
children, instead the children
themselves
40
imprecision
• few events (small sample size)
• wide confidence intervals
fluoroquinolone prophylaxis in
neutropenia
infection-related mortality
41
42
publication bias ?
Standard Error
0
Asymetrical
1
no interest in
publishing
or being
published
0.4
2
3
0.1
0.3
0.6 1
3
10
Odds ratio
42
Formulate
question
Select
outcomes
Rate
importance
Systematic Review
(outcomes across studies)
Evidence Profile
(GRADEpro)
1
2
Pooled estimate of effect for each outcome
Quality of evidence for each outcome
43
PICO
Critical
Outcome2
Critical
Outcome3
Important
Outcome4
Not
Formulate recommendations
For or against an action
Strong or weak (strength)
consider:
Balance benefits/downsides
Quality of evidence
Values and preferences
Resource use (cost)
recommendation
Guideline panel
RCT
observational
1.
2.
3.
4.
5.
1. large effect
2. dose-response
3. antagonistic bias
risk of bias
inconsistency
indirectness
imprecision
publication bias
Rate overall quality of evidence
across outcomes
Wording
 “We recommend…” | “Clinicians should…”
 “We suggest…” | “Clinicians might…”



start
rate down
action
Outcome1
rate up
High
 Moderate
High
| Moderate |Low
Low| Very
Very low
low
unambiguous
clear implications for action
transparent (values & preferences statement)
high
low
44
large effect
• common criteria
– everyone used to do badly
– almost everyone does well
• Example
–
–
–
–
–
oral anticoagulation with mechanical heart valves
insulin for diabetic ketoacidosis
hip replacement for severe osteoarthritis
adrenaline for anaphylactic shock
parachutes when jumping from airplanes
45
dose-response
• higher INR – increased bleeding
• childhood lymphoblastic leukemia
– risk for CNS malignancies 15 years after cranial
irradiation
– no radiation: 1% (95% CI 0% to 2.1%)
– 12 Gy: 1.6% (95% CI 0% to 3.4%)
– 18 Gy: 3.3% (95% CI 0.9% to 5.6%)
46
“antagonistic bias”
all plausible confounding would reduce the
demonstrated effect
or increase the effect if no effect was observed
47
Question 32
E
vidence profile
Date: 2007-09-06
Question: Should subcutaneous allergen specific immunotherapy vs placebo be used in adults with allergic rhinitis?
Bibliography: Calderon M., Alves B., Jacobson M., Hurwitz B., Sheikh A., Durham S. Allergen injection immunotherapy for seasonal allergic rhinitis. Cochrane database of systematic reviews (Online),
2007:CD001936.
Summary of findings
Quality assessment
No of patients
Effect
Importance
Quality
No of
Other
subcutaneous allergen
Relative
Design
Limitations
Inconsistency
Indirectness
Imprecision
placebo
Absolute
studies
considerations
specific immunotherapy
(95% CI)
Symptoms (recorded in diary) (follow-up 3 days to 3 years1; Better indicated by less)
152
randomised no serious
no serious
no serious
no serious
none
SMD -0.73 (
597
466
CRITICAL
trial
limitations
inconsistency 3
indirectness
imprecision
0.97 to -0.5)
HIGH
1
Medication score (follow-up 3 days to 3 years ; Better indicated by less)
13
randomised no serious
no serious
serious4
no serious
none
SMD -0.57 (
549
414
IMPORTANT
trial
limitations
inconsistency 3
imprecision
0.33 to -0.82) MODERATE
1
symptom and medication score (follow-up 3 days to 3 years ; Better indicated by less)
8
randomised no serious
no serious
serious4
no serious
none
SMD -0.48 (
320
297
IMPORTANT
trial
limitations
inconsistency
imprecision
0.29 to -0.67) MODERATE
Nasal symptoms (follow-up 3 days to 3 years1; Better indicated by less)
96
randomised no serious
serious3
no serious
no serious
none
SMD -1.59 (
396
276
CRITICAL
trial
limitations
indirectness
imprecision
0.89 to -2.29)
HIGH
1
Bronchial symptoms (follow-up 3 days to 3 years ; Better indicated by less)
57
randomised no serious
no serious
no serious
serious5
none
SMD -0.59 (
266
163
IMPORTANT
trial
limitations
inconsistency 3
indirectness
0.11 to -1.06)
LOW
1
Ocular symptoms (follow-up 3 days to 3 years ; Better indicated by less)
38
randomised no serious
no serious
no serious
no serious
none
SMD -1.80 (
226
119
CRITICAL
trial
limitations
inconsistency
indirectness
imprecision
0.31 to -3.28)
HIGH
1
Global improvement (follow-up 3 days to 3 years ; Better indicated by less)
99
randomised no serious
–10
no serious
–10
none
0
0
not pooled9
IMPORTANT
trial
limitations
indirectness
1
Quality of life (follow-up 3 days to 3 years ; Better indicated by less)
5
randomised no serious
no serious
no serious
no serious
none
SMD -0.52 (
332
239
CRITICAL
trial
limitations
inconsistency
indirectness
imprecision
0.34 to -0.69)
HIGH
Adverse events (local reaction not requiring treatment) (follow-up 3 days to 3 years1)
24
randomised no serious
–10
no serious
–10
none
not
–/90711
–/69712
not pooled
IMPORTANT
trial
limitations
indirectness
pooled
1
Adverse events (local reaction requiring treatment) (follow-up 3 days to 3 years )
7
randomised no serious
no serious
no serious
serious13
none
not

–/20814
–/18615
not pooled
IMPORTANT
trial
limitations
inconsistency
indirectness
pooled
MODERATE
16
Adverse events (early mild systemic reaction [< 30 minutes]) (follow-up 3 days to 3 years; assessed with: mild systemic reactions )
17
randomised no serious
–10
no serious
–10
none
not
–/70617
–/56618
not pooled
CRITICAL
trial
limitations
indirectness
pooled
1
19
Adverse events (early severe systemic reaction [< 30 minutes]) (follow-up 3 days to 3 years ; assessed with: non life-threatening systemic reactions )
13
randomised no serious
–10
no serious
–10
none
not
–/61520
–/46321
not pooled
CRITICAL
trial
limitations
indirectness
pooled
Formulate
question
Select
outcomes
Rate
importance
Systematic Review
(outcomes across studies)
Evidence Profile
(GRADEpro)
1
2
Pooled estimate of effect for each outcome
Quality of evidence for each outcome
48
PICO
Critical
Outcome2
Critical
Outcome3
Important
Outcome4
Not
Formulate recommendations
For or against an action
Strong or weak (strength)
consider:
Balance benefits/downsides
Quality of evidence
Values and preferences
Resource use (cost)
recommendation
Guideline panel
RCT
observational
1.
2.
3.
4.
5.
1. large effect
2. dose-response
3. antagonistic bias
risk of bias
inconsistency
indirectness
imprecision
publication bias
Rate overall quality of evidence
across outcomes
Wording
 “We recommend…” | “Clinicians should…”
 “We suggest…” | “Clinicians might…”



start
rate down
action
Outcome1
rate up
High
 Moderate
High
| Moderate |Low
Low| Very
Very low
low
unambiguous
clear implications for action
transparent (values & preferences statement)
high
low
49
Question 32
E
vidence profile
Date: 2007-09-06
Question: Should subcutaneous allergen specific immunotherapy vs placebo be used in adults with allergic rhinitis?
Bibliography: Calderon M., Alves B., Jacobson M., Hurwitz B., Sheikh A., Durham S. Allergen injection immunotherapy for seasonal allergic rhinitis. Cochrane database of systematic reviews (Online),
2007:CD001936.
Summary of findings
Quality assessment
No of patients
Effect
Importance
Quality
No of
Other
subcutaneous allergen
Relative
Design
Limitations
Inconsistency
Indirectness
Imprecision
placebo
Absolute
studies
considerations
specific immunotherapy
(95% CI)
Symptoms (recorded in diary) (follow-up 3 days to 3 years1; Better indicated by less)
152
randomised no serious
no serious
no serious
no serious
none
SMD -0.73 (
597
466
CRITICAL
trial
limitations
inconsistency 3
indirectness
imprecision
0.97 to -0.5)
HIGH
1
Medication score (follow-up 3 days to 3 years ; Better indicated by less)
13
randomised no serious
no serious
serious4
no serious
none
SMD -0.57 (
549
414
IMPORTANT
trial
limitations
inconsistency 3
imprecision
0.33 to -0.82) MODERATE
1
symptom and medication score (follow-up 3 days to 3 years ; Better indicated by less)
8
randomised no serious
no serious
serious4
no serious
none
SMD -0.48 (
320
297
IMPORTANT
trial
limitations
inconsistency
imprecision
0.29 to -0.67) MODERATE
Nasal symptoms (follow-up 3 days to 3 years1; Better indicated by less)
96
randomised no serious
serious3
no serious
no serious
none
SMD -1.59 (
396
276
CRITICAL
trial
limitations
indirectness
imprecision
0.89 to -2.29)
HIGH
1
Bronchial symptoms (follow-up 3 days to 3 years ; Better indicated by less)
57
randomised no serious
no serious
no serious
serious5
none
SMD -0.59 (
266
163
IMPORTANT
trial
limitations
inconsistency 3
indirectness
0.11 to -1.06)
LOW
1
Ocular symptoms (follow-up 3 days to 3 years ; Better indicated by less)
38
randomised no serious
no serious
no serious
no serious
none
SMD -1.80 (
226
119
CRITICAL
trial
limitations
inconsistency
indirectness
imprecision
0.31 to -3.28)
HIGH
1
Global improvement (follow-up 3 days to 3 years ; Better indicated by less)
99
randomised no serious
–10
no serious
–10
none
0
0
not pooled9
IMPORTANT
trial
limitations
indirectness
1
Quality of life (follow-up 3 days to 3 years ; Better indicated by less)
5
randomised no serious
no serious
no serious
no serious
none
SMD -0.52 (
332
239
CRITICAL
trial
limitations
inconsistency
indirectness
imprecision
0.34 to -0.69)
HIGH
Adverse events (local reaction not requiring treatment) (follow-up 3 days to 3 years1)
24
randomised no serious
–10
no serious
–10
none
not
–/90711
–/69712
not pooled
IMPORTANT
trial
limitations
indirectness
pooled
1
Adverse events (local reaction requiring treatment) (follow-up 3 days to 3 years )
7
randomised no serious
no serious
no serious
serious13
none
not

–/20814
–/18615
not pooled
IMPORTANT
trial
limitations
inconsistency
indirectness
pooled
MODERATE
16
Adverse events (early mild systemic reaction [< 30 minutes]) (follow-up 3 days to 3 years; assessed with: mild systemic reactions )
17
randomised no serious
–10
no serious
–10
none
not
–/70617
–/56618
not pooled
CRITICAL
trial
limitations
indirectness
pooled
1
19
Adverse events (early severe systemic reaction [< 30 minutes]) (follow-up 3 days to 3 years ; assessed with: non life-threatening systemic reactions )
13
randomised no serious
–10
no serious
–10
none
not
–/61520
–/46321
not pooled
CRITICAL
trial
limitations
indirectness
pooled
Formulate
question
Select
outcomes
Rate
importance
Systematic Review
(outcomes across studies)
Evidence Profile
(GRADEpro)
1
2
Pooled estimate of effect for each outcome
Quality of evidence for each outcome
50
PICO
Critical
Outcome2
Critical
Outcome3
Important
Outcome4
Not
Formulate recommendations
For or against an action
Strong or weak (strength)
consider:
Balance benefits/downsides
Quality of evidence
Values and preferences
Resource use (cost)
recommendation
Guideline panel
RCT
observational
1.
2.
3.
4.
5.
1. large effect
2. dose-response
3. antagonistic bias
risk of bias
inconsistency
indirectness
imprecision
publication bias
Rate overall quality of evidence
across outcomes
Wording
 “We recommend…” | “Clinicians should…”
 “We suggest…” | “Clinicians might…”



start
rate down
action
Outcome1
rate up
High
 Moderate
High
| Moderate |Low
Low| Very
Very low
low
unambiguous
clear implications for action
transparent (values & preferences statement)
high
low
51
B
alance
„I can cure your back problem, but there’s a
risk that you’ll be left with nothing to talk
about.”
52
Q
uality
53
quality of evidence
=
/
strength of recommendation
54
V
alues
„Because I like it better than the old one, that’s
why!”
55
56
S
not confident
strong
weak
(conditional)
trength
confident
57
Recommendation
We suggest subcutaneous specific immunotherapy in
patients with seasonal (weak recommendation | high quality
evidence) and perennial allergic rhinitis due to house
dust mites (weak recommendation | low quality evidence).
Underlying values and preferences
This recommendation places a relatively high value on
relieving the symptoms of allergic rhinitis, and a
relatively low value on adverse effects and on resource
expenditure.
58
quality of evidence
strength of recommendation
Formulate
question
Select
outcomes
Rate
importance
Systematic Review
(outcomes across studies)
Evidence Profile
(GRADEpro)
1
2
Pooled estimate of effect for each outcome
Quality of evidence for each outcome
59
PICO
Critical
Outcome2
Critical
Outcome3
Important
Outcome4
Not
Formulate recommendations
For or against an action
Strong or weak (strength)
consider:
Balance benefits/downsides
Quality of evidence
Values and preferences
Resource use (cost)
recommendation
Guideline panel
RCT
observational
1.
2.
3.
4.
5.
1. large effect
2. dose-response
3. antagonistic bias
risk of bias
inconsistency
indirectness
imprecision
publication bias
Rate overall quality of evidence
across outcomes
Wording
 “We recommend…” | “Clinicians should…”
 “We suggest…” | “Clinicians might…”



start
rate down
action
Outcome1
rate up
High
 Moderate
High
| Moderate |Low
Low| Very
Very low
low
unambiguous
clear implications for action
transparent (values & preferences statement)
high
low
60
strong recommendation
Patients Most people in your situation would
want the recommended course of action and
only a small proportion would not
Clinicians Most patients should receive the
recommended course of action
Policy makers The recommendation can be
adapted as a policy in most situations
61
weak recommendation
Patients The majority of people in your
situation would want the recommended course
of action, but many would not
Clinicians Be prepared to help patients to
make a decision that is consistent with their
own values
Policy makers There is a need for substantial
debate and involvement of stakeholders
Formulate
question
Select
outcomes
Rate
importance
Systematic Review
(outcomes across studies)
Evidence Profile
(GRADEpro)
1
2
Pooled estimate of effect for each outcome
Quality of evidence for each outcome
62
PICO
Critical
Outcome2
Critical
Outcome3
Important
Outcome4
Not
Formulate recommendations
For or against an action
Strong or weak (strength)
consider:
Balance benefits/downsides
Quality of evidence
Values and preferences
Resource use (cost)
recommendation
Guideline panel
RCT
observational
1.
2.
3.
4.
5.
1. large effect
2. dose-response
3. antagonistic bias
risk of bias
inconsistency
indirectness
imprecision
publication bias
Rate overall quality of evidence
across outcomes
Wording
 “We recommend…” | “Clinicians should…”
 “We suggest…” | “Clinicians might…”



start
rate down
action
Outcome1
rate up
High
 Moderate
High
| Moderate |Low
Low| Very
Very low
low
unambiguous
clear implications for action
transparent (values & preferences statement)
high
low
63
Jan Brożek, MD, PhD
Department of Clinical Epidemiology & Biostatistics
McMaster University, Hamilton, Ontario, Canada
< [email protected] >