Society of General International Medicine

32 nd Annual Meeting, May 14 th 2009 Elie A. Akl, MD, MPH, PhD David Atkins, MD, MPH Eric Bass, MD, MPH Yngve Falck-Ytter, MD Stephanie Chang, MD, MPH

GRADING QUALITY OF EVIDENCE THE GRADE APPROACH

1

Session outline      Introductions, objectives Overview of the GRADE approach Applying the GRADE approach Wrap-up Session evaluation (5 min) (25 min) (45 min) (10 min) (5 min)

Disclosure  Presenters are members of the GRADE working group and have received honoraria related to this work that were deposited into research accounts  No conflict of interest related to pharmaceutical industry

Objectives

Learning objectives     To enumerate GRADE categories for quality of evidence To list the GRADE factors that affect the quality of evidence To apply the GRADE approach to a specific body of evidence To discuss the strengths and limitations of the GRADE approach

Overview of the GRADE approach

G

rades of

R

ecommendation

A

ssessment,

D

evelopment and

E

valuation

GRADE WORKING GROUP

CMAJ 2003, BMJ 2004, BMC 2004, BMC 2005, AJRCCM 2006, Chest 2006, BMJ 2008

GRADE definition of Quality of Evidence “Extent to which confidence in estimate of effect adequate to support decision”

GRADE rating of outcomes  GRADE rates the quality of evidence for each outcome separately  The type of evidence may be different for different outcomes  Different audiences are likely to have varying perspective on the importance of outcomes  GRADE considers desirable and undesirable outcomes and rates their relative importance 9

GRADE rating of outcomes  Desirable outcomes  lower mortality  reduced hospital stay  reduced duration of disease  reduced resource expenditure  Undesirable outcomes  adverse reactions  the development of resistance  costs of treatment

GRADE rating of outcomes

9 6 5 4 3 8 7 2 1

Critical for decision making Important, but not critical for decision making Of low importance

GRADE rating of outcomes  Ranking outcomes by their relative importance can help to focus attention on those outcomes that are considered most important  Outcome choice should be based on what is important, and not what was measured 12

GRADE uses a comprehensive and transparent conceptual framework for rating the quality of evidence 13

GRADE levels of Evidence  High:  Moderate:  Low:  Very low:

GRADE levels of Evidence  High: considerable confidence in estimate of effect  Moderate: further research likely to have impact on confidence in estimate, may change estimate  Low: further research is very likely to impact on confidence, likely to change the estimate  Very low: any estimate of effect is very uncertain

Determinants of quality  Quality starts high for evidence from RCTs  Quality starts low for evidence from observational studies  5 factors lower the quality of evidence  3 factors can increase the quality of evidence

Factors that lower quality 1.

2.

3.

4.

5.

Study limitations (in design and execution) Inconsistency Indirectness Reporting bias Imprecision 17

Factors that lower quality 1.

      Study limitations (in design and execution) Inappropriate randomization Lack of concealment Intention to treat principle violated Inadequate blinding Loss to follow-up Early stopping for benefit

Factors that lower quality  From Cates , CDSR 2008 CDSR 2008

Factors that lower quality Overall judgment required

Factors that lower quality 2.

 Inconsistency   Assess for inconsistency (Heterogeneity)  variation in size of effect  overlap in confidence intervals statistical significance of heterogeneity I 2  If inconsistency  look for explanation  patients, intervention, outcome, methods  If unexplained inconsistency  downgrade quality

Factors that lower quality 2.

Inconsistency Heparin or vitamin K antagonists for survival in patients with cancer: Akl E, Barba M, Rohilla S, Terrenato I, Sperati F, Schünemann HJ. “

Anticoagulation for the long term treatment of venous thromboembolism in patients with cancer

”. Cochrane Database Syst Rev. 2008 Apr 16;(2):CD006650.

Factors that lower quality 2.

Inconsistency Non-steroidal drug use and risk of pancreatic cancer: Capurso G, Sch ünemann HJ, Terrenato I, Moretti A, Koch M, Muti P, Capurso L, Delle Fave G.

Meta-analysis: the use of non-steroidal anti-inflammatory drugs and pancreatic cancer risk for different exposure categories.

Aliment Pharmacol Ther. 2007 Oct 15;26(8):1089-99.

Factors that lower quality 3.

 Indirectness of Evidence Differences in populations/patients  mild versus severe COPD   Differences in interventions all inhaled steroids, new vs. old   Differences in outcomes important vs. surrogate;

Factors that lower quality 3.

 Indirectness of Evidence indirect comparisons  interested in A versus B  have A versus C and B versus C Alendronate Risedronate Placebo

Factors that lower quality 4.

Publication bias  Number of small studies  Faster and multiple publication of “positive” trials  Fewer and slower publication of “negative” trials

I.V. Mg in acute myocardial infarction

Publication bias

ISIS-4

Lancet 1995

Meta-analysis

Yusuf S.Circulation 1993

Egger M, Smith DS. BMJ 1995;310:752-54 27

0

Funnel plot Symmetrical: No publication bias

1 2 3 0.1

0.3

0.6

1

Odds ratio Egger M, Cochrane Colloquium Lyon 2001

3 10

28

0

Funnel plot Asymmetrical: Publication bias?

1 2 3 0.1

0.3

0.6

1

Odds ratio Egger M, Cochrane Colloquium Lyon 2001

3 10

29

I.V. Mg in acute myocardial infarction

Publication bias

ISIS-4

Lancet 1995

Meta-analysis

Yusuf S.Circulation 1993

Egger M, Smith DS. BMJ 1995;310:752-54 30

Meta analysis confirme d by mega trials Egger M, Smith DS. BMJ 1995;310:752-54 31

Factors that lower quality 5.

 Imprecision small sample size  small number of events  wide confidence intervals  uncertainty about magnitude of effect  how to decide if CI too wide?

 grade down one level?

 grade down two levels?

Factors that raise quality 1.

Large magnitude of effect 2.

Dose response relation 3.

All plausible confounding may be working to reduce the demonstrated effect or increase the effect if no effect was observed 33

Factors that raise quality 1.

 Large magnitude of effect large (RRR 50%) can raise by one level  very large (RRR 80%) can raise by two levels  common criteria  everyone used to do badly  almost everyone does well  Examples    oral anticoagulation for mechanical heart valves insulin for diabetic ketoacidosis hip replacement for severe osteoarthritis

Factors that raise quality 2.

Dose response relation  higher INR – increased bleeding  childhood lymphoblastic leukemia  risk for CNS malignancies 15 years after cranial irradiation  no radiation: 1% (95% CI 0% to 2.1%)  12 Gy: 1.6% (95% CI 0% to 3.4%)  18 Gy: 3.3% (95% CI 0.9% to 5.6%)

Factors that raise quality 3. All plausible confounding may be working to reduce the demonstrated effect or increase the effect if no effect was observed

Factors that raise quality  Example 1: higher death rates in private for profit versus private not-for-profit hospitals  patients in the not-for-profit hospitals likely sicker than those in the for-profit hospitals  for-profit hospitals are likely to admit a larger proportion of well-insured patients than not-for-profit hospitals (and thus have more resources with a spill over effect)

Factors that raise quality  Example 2: hypoglycaemic drug phenformin causes lactic acidosis  The related agent metformin is under suspicion for the same toxicity.  Large observational studies have failed to demonstrate an association  Clinicians would be more alert to lactic acidosis in the presence of the agent

Summary of GRADE framework for rating the quality of evidence 39

Quality of evidence High Moderate Low Very low Study design

Randomised trial Observational study

Lower if Higher if

Study quality: Serious limitations Very serious limitations Important inconsistency Directness: Some uncertainty Major uncertainty

Sparse or

imprecise data High probability of reporting bias

Strong association:

Strong, no plausible confounders Very strong, no major threats to validity Evidence of a Dose response

gradient

All plausible confounders would have reduced the effect

Evidence Profiles and Summary

of Findings (SoF) Tables

summarize the rating of the quality of evidence across selected outcomes 41

42

43

Applying the GRADE approach Exercise: parenteral anticoagulation for prolonging the survival of patients with cancer

Wrap-up

46

Advantages of GRADE   Developed by a widely representative group of international guideline developers Clear separation between quality of evidence and strength of recommendations   Explicit evaluation of the importance of outcomes Explicit, comprehensive criteria for downgrading and upgrading quality of evidence ratings 47

Advantages of GRADE   Transparent process of moving from evidence to recommendations Explicit acknowledgment of values and preferences   Clear, pragmatic interpretation of strong versus weak recommendations for clinicians, patients, and policy makers Useful for systematic reviews and health technology assessments, as well as guidelines 48

Disadvantages of GRADE  Involves a number of judgments that might affect its reliability  Requires expertise/training 49

Session evaluation

Thank you!