Methodology for Guideline Development for the 7th ACCP

Download Report

Transcript Methodology for Guideline Development for the 7th ACCP

Grading evidence and recommendations

The GRADE initiative Holger Schünemann, MD, PhD

Associate Professor Italian National Cancer Institute Regina Elena, Rome

When to make a recommendation?



never

 patient values differ  just lay out benefits and risks 

when evidence strong enough

 when very weak, too uncertain 

clinicians need guidance

• intense study demands decision

Two examples from Clinical Evidence

Aspirin for acute myocardial infarction  “One systematic review in people with acute myocardial infarction has found that aspirin reduces mortality, reinfarction, and stroke at 1 month compared with placebo.”

Two examples from Clinical Evidence

Systemic thrombolysis for acute stroke  “One systematic review in people with confirmed ischaemic stroke found that thrombolysis reduced the risk of the composite outcome of death or dependency ...  However, it increased the risk of death from intracranial haemorrhage ...  The excess in deaths was offset by fewer people being alive but dependent 6 months after stroke onset, and the net effect was a reduction in people who were dead or dependent. ... Results of the reviews may not extrapolate to people with the mildest or most severe strokes.”

Two examples from Clinical Evidence

Systemic thrombolysis for acute stroke  “We found little evidence about which people are most and least likely to benefit from thrombolysis. A subgroup analysis suggested that thrombolysis may be more beneficial if given within 3 hours of symptom onset, but the duration of the “therapeutic time window” could not be determined reliably”

What do users want from recommendations?

• Users are looking for different things • Just tell me what to do (recommendation) • What to do, and on strong or weak grounds • recommendation and grade • Recommend, grade, evidence summary, values • systematic review, value statement • Evidence from individual studies

How can we judge the extent of our confidence that adherence to a recommendation will do more good than harm?

Why Grade Recommendations?

 strong recommendations – high quality methods – large precise effect – few down sides of therapy  weak recommendations – low quality methods – imprecise estimate – small effect – substantial down sides



Why bother about grading?

People draw conclusions about the – quality of evidence – strength of recommendations  Systematic and explicit approaches can help – protect against errors – resolve disagreements – facilitate critical appraisal – communicate information

Why bother about grading?

 Alternate practitioner behavior  Strong: apply uniformly – just do it  Weak: think about it – examine evidence yourself – consider patient circumstances – explore with the patient  However, there is wide variation in currently used approaches

Which grading system?

Evidence Recommendation  II-2 B   C+ 1 Strong Strongly recommended Organization  USPSTF   ACCP GCPS

Still not confused?

Recommendation for use of oral anticoagulation in patients with atrial fibrillation and rheumatic mitral valve disease Evidence Recommendation  B Class I   C+ IV 1 C Organization  AHA   ACCP SIGN

Grading System

Current profusion: can there be consensus?

GRADE

G rades of R ecommendation A ssessment, D evelopment and E valuation

What is GRADE*?

o o o o o o Began as informal working group in 2000 Researchers/guideline developers with interest in methodology Aim: to develop a common system for grading the quality of evidence and the strength of recommendations that is sensible and to explore the range of interventions and contexts for which it might be useful* 13 meetings (~10 – 35 attendants) Evaluation of existing systems and reliability* Workshops at Cochrane Colloquia, WHO, GIN and various conferences since 2000 *Grade Working Group. CMAJ 2003, BMJ 2004, BMC 2004, BMC 2005

BMJ 2004; 328:1490

GRADE Working Group David Atkins, chief medical officer a Dana Best, assistant professor b Martin Eccles, professor d Francoise Cluzeau, lecturer x Yngve Falck-Ytter, associate director e Signe Flottorp, researcher f Gordon H Guyatt, professor g Robin T Harbour, quality and information director h Margaret C Haugh, methodologist i David Henry, professor j Suzanne Hill, senior lecturer j Roman Jaeschke, clinical professor k Regina Kunx, Associate Professor Gillian Leng, guidelines programme director l Alessandro Liberati, professor m Nicola Magrini, director n James Mason, professor d Philippa Middleton, honorary research fellow o Jacek Mrukowicz, executive director p Dianne O’Connell, senior epidemiologist q Andrew D Oxman, director f Bob Phillips, associate fellow r Holger J Schünemann, associate professor g,s Tessa Tan-Torres Edejer, medical officer t David Tovey, Editor y Jane Thomas, Lecturer, UK Helena Varonen, associate editor u Gunn E Vist, researcher f John W Williams Jr, professor v Stephanie Zaza, project director w a) Agency for Healthcare Research and Quality, USA b) Children's National Medical Center, USA c) Centers for Disease Control and Prevention, USA d) University of Newcastle upon Tyne, UK e) German Cochrane Centre, Germany f) Norwegian Centre for Health Services, Norway g) McMaster University, Canada h) Scottish Intercollegiate Guidelines Network, UK i) Fédération Nationale des Centres de Lutte Contre le Cancer, France j) University of Newcastle, Australia k) McMaster University, Canada l) National Institute for Clinical Excellence, UK m) Università di Modena e Reggio Emilia, Italy n) Centro per la Valutazione della Efficacia della Assistenza Sanitaria, Italy o) Australasian Cochrane Centre, Australia p) Polish Institute for Evidence Based Medicine, Poland q) The Cancer Council, Australia r) Centre for Evidence-based Medicine, UK s) National Cancer Institute, Italy t) World Health Organisation, Switzerland u) Finnish Medical Society Duodecim, Finland v) Duke University Medical Center, USA w) Centers for Disease Control and Prevention, USA x) University of London, UK Y) BMJ Clinical Evidence, UK

a) Agency for Healthcare Research and Quality, USA b) Children's National Medical Center, USA c) Centers for Disease Control and Prevention, USA d) University of Newcastle upon Tyne, UK e) German Cochrane Centre, Germany f) Norwegian Centre for Health Services, Norway g) McMaster University, Canada h) Scottish Intercollegiate Guidelines Network, UK i) Fédération Nationale des Centres de Lutte Contre le Cancer, France j) University of Newcastle, Australia k) McMaster University, Canada l) National Institute for Clinical Excellence, UK m) Università di Modena e Reggio Emilia, Italy n) Centro per la Valutazione della Efficacia della Assistenza Sanitaria, Italy o) Australasian Cochrane Centre, Australia p) Polish Institute for Evidence Based Medicine, Poland q) The Cancer Council, Australia r) Centre for Evidence-based Medicine, UK s) National Cancer Institute, Italy t) World Health Organisation, Switzerland u) Finnish Medical Society Duodecim, Finland v) Duke University Medical Center, USA w) Centers for Disease Control and Prevention, USA x) University of London, UK Y) BMJ Clinical Evidence, UK

Grading System

• trade-off benefits and risks • do it (or don’t do it) • probably do it (or probably don’t do it) • quality of underlying evidence • high quality (well done RCT) • moderate (quasi-RCT) • low (well done observational) • very low (anything else)

Moving quality down

• poor (RCT) design, implementation • randomization, blinding, concealment, follow-up, intention to treat principle • inconsistency • indirect • patients, interventions, outcomes • A vs B, but have A to C, B to C • reporting bias

Reporting bias

 high likelihood of reporting bias could lower quality  reporting of outcomes  reporting of studies – publication bias    number of small studies industry sponsored funnel plots may help

Moving quality up

• • magnitude of effect • Strong association: RR > 2 or RR < 0.5

• Very strong association: RR > 5 or RR < 0.2

dose-response • biases favor control

Judgements about the overall quality of evidence

 Most systems not explicit    Options: – strongest outcome – primary outcome – benefits – weighted – separate grades for benefits and harms – no overall grade – weakest outcome Based on lowest of all the critical outcomes Beyond the scope of a systematic review

Quality assessment criteria

Quality of evidence Study design High Randomised trial Moderate Low Very low Observational study Any other evidence Lower if Study quality: -1 Serious limitations -2 Very serious limitations -1 Important inconsistency Directness: -1 Some uncertainty -2 Major uncertainty -1 Sparse or imprecise data -1 High probability of reporting bias Higher if Strong association: +1 Strong, no plausible confounders +2 Very strong, no major threats to validity +1 Evidence of a Dose response gradient +1 All plausible confounders would have reduced the effect

Categories of quality

    High: Further research is very unlikely to change our confidence in the estimate of effect. Moderate: Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate.

Low: Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate.

Very low: Any estimate of effect is very uncertain.

                          

Strength of recommendation

“The extent to which one can be confident that adherence to a recommendation will do more good than harm.”  quality of the evidence    translation of the evidence into practice in a specific setting uncertainty about baseline risk trade-offs (the relative value attached to the expected benefits, harms and costs)

Judgements about the balance between benefits and harms

 Before considering cost and making a recommendation

Judgment: Benefits vs Risks/Costs

        quality of evidence seriousness of outcome magnitude of effect precision of treatment effect risk of target event risk of adverse events cost of therapy values

Value and preference statements

 underlying values and preferences always present  sometimes crucial  important to make explicit

Values and preferences

Stroke guideline: patients with TIA clopidogrel over aspirin (weak, moderate quality) Underlying values and preferences on minimizing drug expenditures.

: This recommendation to use clopidogrel over aspirin places a relatively high value on a small absolute risk reduction in stroke rates, and a relatively low value

Values and preferences

Peripheral vascular disease: aspirin be used instead of clopidogrel (weak, high quality).

Underlying values and preferences : This recommendation places a relatively high value on avoiding large expenditures to achieve small reductions in vascular events.

Judgements about recommendations

1. Benefit and downside evaluation

Benefits<< Downsides  Benefits ?

≤ Downsides Benefits ?

≥ Downsides ?

  ?

2. Recommendation (wording)

STRONG Recommend don’t do it WEAK suggest WEAK suggest probably don’t do it probably do it should not do it might not do it might do it Benefits >> Downsides  STRONG recommend do it should do it

Observational studies – high or moderate quality?

 Strong association  Dose response relationship  Plausible confounders would have reduced the effect

Strong recommendation when evidence is weak?

  Balance of benefits and downsides clearly on one side Not frequent if quality is low or very low

Comparison of GRADE and other systems

• • • • • • • • • • • • • Explicit definitions Explicit, sequential judgements Components of quality Overall quality Relative importance of outcomes Balance between health benefits and harms Balance between incremental health benefits and costs Consideration of equity Evidence profiles International collaboration Software Consistent judgements?

Communication?

Who is interested in GRADE

• • • • • • • • • • WHO American Endocrine Society American College of Chest Physicians (ACCP) Norwegian Centre for Health Services UpToDate Close relationship with Cochrane Collaboration American Society of Clinical Oncology (ASCO) American Thoracic Society (ATS) Urologists worldwide (EBUro) NICE

Conclusion

 Challenges in grading – judgment always required  Must consider study design, execution, consistency, directness, reporting bias – magnitude, precision, only when extreme  Balance of benefits and risks/cost – magnitude of effects; precision of effects; values and preferences  Separation between recommendation and quality of the evidence  GRADE working group active in obtaining feedback and dissemination

Further GRADE developments

     Diagnostic tests Costs (Equity) Empirical evaluations Adoption of some elements by Cochrane software application

Population: Intervention: (comparison) Outcome:

The clinical question

In patients with chronic atrial fibrillation and no prior history of stroke does oral anticoagulation compared with no therapy reduce the risk for embolic stroke, hemorrhage and death?

     

The evidence

Systematic Review* 5 RCTs 2,313 Patients randomised Warfarin in all studies Studien 1.5 years mean follow-up Outcomes: Ischemic Stroke, hemorrhage (major, including intracranial), death (vascular and all cause) and dependency *Systematischer Review: Aguilar & Hart. Cochrane Database of Systematic Reviews 2005, Issue 3.

Population: Intervention: (comparison) Outcome: Different risk groups: Low, moderate, high In patients with chronic atrial fibrillation and no prior history of stroke does oral anticoagulation compared with no therapy reduce the risk for embolic stroke, hemorrhage and death?

Other outcomes: Inconvenience, quality of life

Outcomes/endpoints



All disabling or fatal stroke (isch. and hemorrh.)



Major hemorrhage (non IC)



All cause mortality



Minor bleeding (hematoma, prolonged bleeding of minor wounds)

*Systematic Review: Aguilar & Hart. Cochrane Database of Systematic Reviews 2005,

Outcomes/endpoints

How important is the endpoint for decision making?

Judgment about the relative importance for each endpoint on a scale from 9 (most important) to 1 (least important): • 7 – 9: the endpoint is critical for decision making.

• 4 – 6: the endpoint is important but not critical.

• 1 – 3: the endpoint is not important.

Outcomes/endpoints



All disabling or fatal stroke (isch. and hemorrh.) 9

  

Major hemorrhage (non IC) 7 All cause mortality 9 Minor bleeding (hematoma, prolonged bleeding of minor wounds) 5

*Systematic Review: Aguilar & Hart. Cochrane Database of Systematic Reviews 2005,

Quality assessment criteria

Disabling or fatal stroke Study design:



5 RCTs Quality of evidence for this endpoint:



High

Disabling or fatal stroke

Detailed design and execution   Concealment Follow-up  In two studies (CAFA; SPINAF) both patients and outcome assessors were blinded; in the other studies only outcomes assessors.

Quality of evidence for this endpoint now: High (or -1  Moderate)

Disabling or fatal stroke Consistency:



No inconsistency Quality of evidence for this endpoint now:



High

Directness of evidence

 indirect treatment comparisons – interested in A versus B – have A versus C and B versus C  alendronate vs risedronate – both versus placebo, no head-to-head

Directness - patients

 patients meet trials’ eligibility criteria  not included, but no reason to question – slight age difference, comorbidity, race  some question, bottom line applicable – valvular atrial fibrillation  serious question about biology – heart failure trials applicability to aortic stenosis

Directness - interventions

 same drugs and doses – captopril 100 mg. tid in heart failure  similar drugs and doses – captopril in lower doses  same class and biology – other ACEI in heart failure  questionable class and biology – ARB in heart failure – sigmoidoscopy for colon ca prevention

Directness - outcomes

 same outcomes – alendronate over 3 years on fracture  similar but questionable – alendronate over long-term  serious question – surrogate outcomes – bone density; arrhythmia suppression; laboratory exerice capacity; cardiac function

Disabling or fatal stroke Directness of the evidence:



Population, Intervention, Outcomes



Direct Quality of evidence for this endpoint now:



High

Disabling or fatal stroke Imprecise or sparse data:



Would few additional events or larger studies likely alter the results?

Disabling or fatal stroke Imprecise or sparse data:



No imprecise or sparse data Quality of evidence for this endpoint now:



High

Disabling or fatal stroke



Reporting bias:



Not present Quality of evidence for this endpoint now:



High

Disabling or fatal stroke



Strong association?



present (RR = 0.46)

strong, no plausible confounder, consistent and

Quality of evidence for this endpoint now:



High [or +1



High (from moderate)]

Major extracranial hemorhage

• • • •

Study design: 4 RCTs → Quality: High Study details and execution: No serious limitations No inconsistency and direct Imprecise or sparse data?

Imprecise or sparse data

There is not an empirical basis for defining imprecise or sparse data. Two possible definitions are:   Data are sparse if the results include just a few events or observations and they are uninformative Data are imprecise if the confidence intervals are sufficiently wide that an estimate is consistent with either important harms or important benefits.

These different definitions can result in different judgments.

Major extracranial hemorhage

• • • •

Study design: 4 RCTs → Quality: High Study details and execution: no serious limitations No inconsistency and direct Imprecise or sparse data?

• Imprecise data (wide confidence intervals/few events)

GRADE Evidence Profile Author(s):

Schunemann

Date:

8/28/2005

Question:

Should warfarin vs placebo or no treatment be used for patients with non-valvular atrial fibrillation?

Patient or population:

Patients with non-valvular atrial fibrillation

Settings:

Long term outpatient management

Systematic review:

Quality across all endpoints

Summary of findings Quality assessment No of patients Effect Quality Importance No of studies Design Limitations Consistency Directness Other considerations warfarin placebo or no treatment Disabling or Fatal Stroke (ischemic and hemorrhagic)

(Neuroimaging or autopsy 6 Follow up: Mean: 1.5 years)

Relative (95% CI) Absolute (95% CI)

5 Randomised trials No limitation 8,12 No important inconsistency No uncertainty Strong association (+1) 13 18/1154 (1.6%)

Extracranial hemorrhage

(Transfusion or invasive procedure requirement 1 Follow up: 1.5 years) 39/1159 (3.4%) RR 0.46 (0.27 to 0.81) 20/1 000 (30 fewer/1000 to 10 fewer/1000)  High 9 14 4 Randomised trials No limitations 8 No important inconsistency No uncertainty Imprecise or sparse data ( 1) 11

All cause mortality

3,5 (Direct patient follow-up Follow up: 1.5 years) 17/1154 (1.5%) 16/1159 (1.4%) RR 1.06 (0.54 to 2.09) 0/1 000 (10 fewer/1000 to 10 more/1000)   Moderate 7 5 Randomised trials No limitations 15 No important inconsistency No uncertainty None 69/1225 (5.6%) 99/1236 (8%) RR 0.70 (0.52 to 0.95) 20/1 000 (40 fewer/1000 to 1 fewer/1000)  High 9

Footnotes:

1. required transfusion of two or more units of red blood cells, hospitalization, or invasive procedures to control bleeding and those that resulted in death or permanent functional impairment (e.g. blindness) were included. 2. These consisted of death due to stroke, heart disease, hemorrhage and sudden deaths of unknown cause. 3. All cause mortality: death from any cause (vascular and non- vascular) within 30 days from onset of stroke symptoms. For this outcome, results of published data, which included about 6% of patients with prior stroke or TIA, were used. 4. The diagnosis of MI was usually based upon electrocardiographic changes, elevation of enzymes or post-mortem examination. 5. From Fig 10 6. Follow-up for this outcome was less than 100%. 7. Ischemic stroke was an identified outcome in all trials, with the ischemic nature conrmed by neuroimaging or autopsy in the majority of cases. 8. In two studies (CAFA; SPINAF) patients and outcome assessors were blind to OAC administration, while in the remaining trials treatment was given open label with outcomes verified by those unaware of treatment assignment. 9. Methodological quality was not downgraded because the lack of blinding in some studies did not have important impact on the results 10. Only 5 events in the OAC group and 2 events in the control group, confidence intervals wide 11. Only 17 events in the OAC and 16 events in the control group 12. Loss to follow-up not reported in AFASAK I and CAFA, ranged from 0 to 3% in the other studies 13. Strong association present: RR 0.46 14. Importance is rated on a scale from 1 to 9. 1 represents least important (not important for decision making) and 9 most important (for decision making). 15. Lack of blinding in two trials of lesser concern 16. Strong association present: RR 0.36

Risk groups

Risk for cardio-embolic stroke:  High (prior TIA or stroke*, > 75 yrs,  LVEF/CHF, HTN or DM): 10%/year – Benefits greater downsides: do it, high   Moderate risk (65 to 75 years) or one risk factor: 3 to 4%/year – Benefits greater downsides: do it, high Low risk (< 65 years): 0.5%/year – Benefits smaller than downsides: values: probably do not do it, high

GRADE Evidence Profile Author(s):

Schunemann

Date:

8/28/2005

Question:

Should warfarin vs placebo or no treatment be used for patients with non-valvular atrial fibrillation?

Patient or population:

Patients with non-valvular atrial fibrillation

Settings:

Long term outpatient management

Systematic review:

Aguilar MI, Hart R. Oral anticoagulants for preventing stroke in patients with non-valvular atrial fibrillation and no previous history of stroke or transient ischemic attacks. The Cochrane Database of Systematic Reviews 2005, Issue 3. Art. No.: CD001927.pub2. DOI: 10.1002/14651858.CD001927.pub2.

Quality assessment No of patients Summary of findings Effect No of studies Design Limitations Consistency Directness Other considerations warfarin placebo or no treatment Disabling or Fatal Stroke (ischemic and hemorrhagic)

(Neuroimaging or autopsy 6 Follow up: Mean: 1.5 years) 5 Randomised trials No limitation 8,12 No important inconsistency No uncertainty Strong association (+1) 13 18/1154 (1.6%) 39/1159 (3.4%) RR 0.46 (0.27 to 0.81) Low: 2 fewer/1000 (1 fewer/1000 to 3 fewer/1000) Mean (moderate): 20 fewer/1 000 (30 fewer/1000 to 10 fewer/1000) High: 50 fewer (20 fewer to 75 fewer)  High

Extracranial hemorrhage

(Transfusion or invasive procedure requirement 1 Follow up: 1.5 years)

Relative (95% CI) Absolute (95% CI) Low risk (0.5%/year) Mean/moderate risk (3.5%/year) High risk (10%/year) Quality Importance

9 14 4 Randomised trials No limitations 8 No important inconsistency No uncertainty Imprecise or sparse data ( 1) 11

All cause mortality

Judgment: Benefits vs Risks/Costs

        quality of evidence seriousness of outcome magnitude of effect precision of treatment effect risk of target event risk of adverse events cost of therapy values

Value and preference statements

 underlying values and preferences always present  sometimes crucial  important to make explicit

Values and preferences

Stroke guideline: patients with TIA clopidogrel over aspirin (weak, moderate quality) Underlying values and preferences on minimizing drug expenditures.

: This recommendation to use clopidogrel over aspirin places a relatively high value on a small absolute risk reduction in stroke rates, and a relatively low value

Values and preferences

Peripheral vascular disease: aspirin be used instead of clopidogrel (weak, high quality).

Underlying values and preferences : This recommendation places a relatively high value on avoiding large expenditures to achieve small reductions in vascular events.

Observational studies – high or moderate quality?

 Strong association  Dose response relationship  Plausible confounders would have reduced the effect

Strong recommendation when evidence is weak?

  Balance of benefits and downsides clearly on one side Not frequent if quality is low or very low

Conclusion

 challenges in grading – judgment always required  must consider study design, execution, consistency, directness, reporting bias – magnitude, precision, only when extreme  balance of benefits and risks/cost – magnitude of effects; precision of effects; values and preferences

GRADE Evidence Profile

GRADE Evidence Profile Author(s):

Schunemann, Flottorp & Vist

Date:

18.06.2005

Question:

Should intravenous recombinant tissue plasminogen activator vs no thrombolysis be used for acute ischemic stroke within 3 h?

Patient or population:

acute stroke and excluded other diagnosis by CT scan

Settings:

inpatient hospital

Systematic review:

Wardlaw JM, del Zoppo G, Yamaguchi t, Berge E. Thrombolysis for acute ischemic stroke. The Cochrane Database of systematic reviews. 2003 issue 3

No of studies Design Quality assessment No of patients Limitations Consistency Directness Other considerations intravenous recombinant tissue plasminogen activator no thrombolysis Summary of findings Relative 3 (95% CI) Effect Absolute (95% CI) Death or dependency at the end of follow up

(Barthel Index and modified Rankin score 2-6, where 6 = dead and 0-1 no event Follow up: 3 months)

5 Randomised trials Serious limitations (-1) 1 No important inconsistency No uncertainty None 232/465 (49.9%) 280/465 (60.2%)

Quality

RR 0.83 (0.74 to 0.93) 111/1 000 (5/1000 to 17/1000) NIHSS 0 to 9 100/1000 9 to 14, 136/1000 < 9, 146/1000 Moderate

Importance 5

Death from all causes during follow-up

( Follow up: 3 months) 6 Randomised trials Serious limitations (-1) 1 No important inconsistency No uncertainty None

Symptomatic intracraneal haemorrhage

(repeat CT scan Follow up: 3 months) 4 Randomised trials Serious limitations (-1) 1 No important inconsistency No uncertainty Strong association (+1) 2 83/479 (17.3%) 20/153 (13.1%) 83/478 (17.4%) 5/153 (3.3%) RR 0.98 (0.74 to 1.29) RR 3.27 (1.36 to 7.89) 90/1 000 (30/1000 to 150/1000) Moderate High 9 8

Footnotes:

1. Randomisation uncussessful in the largest study + no intention to treat, lack of information about losses to follow up 2. Strong association with RR 3.27 3. All data are based on fixed effect models – RR calculated from data provided by Wardlaw et al. 4. The first absolute effect is based on the mean baseline risk in the control group across all studies and is filled in whenever there is a convincing impact of the intervention. The three risk groups for death and dependency are based on the NINDS trial baseline risk categories for the outcome of NIHSS ultimate score 0 or 1. 5. On 1 to 9 scale where 9 is most important

Clinical Evidence questions       concerns about what happens when there are 4 high quality RCTs for a given comparison and outcome and one poor one - can the latter drag the GRADE rating down? variability between observer rating of the various parameters, and how this might be dealt with concerns that our harms data, which uses both RCT and observational studies, might mean that every recommendation (categorisation in CE speak) is down graded how surgical treatments are covered extra work involved over and above what we presently do whether there are potential savings in respect of people time e.g if evidence falls into the very low quality rating, can we reasonably not extract the data or report the findings?

Technology:    extensibiltiy and flexibility of the software, can it be configured to suit our needs / processes? IPR of the software whether open access to the software source codes are allowed

GRADEpro©

GRADE Profiler

GRADEpro©

      

Visual studio.net

Windows based (Mac version coming) Easy installation Help file Will be integrated with Revman (trial) Free availability Beta version

Development of GRADE profiles

GRADE Evidence Profile Author(s):

Schunemann

Date:

8/28/2005

Question:

Should warfarin vs placebo or no treatment be used for patients with non-valvular atrial fibrillation?

Patient or population:

Patients with non-valvular atrial fibrillation

Settings:

Long term outpatient management

Systematic review:

Summary of findings Quality assessment No of patients Effect No of studies Design Limitations Consistency Directness Other considerations warfarin placebo or no treatment Disabling or Fatal Stroke (ischemic and hemorrhagic)

(Neuroimaging or autopsy 6 Follow up: Mean: 1.5 years)

Relative (95% CI)

5 Randomised trials Serious limitations (-1) 8,12 No important inconsistency No uncertainty Strong association (+1) 13 18/1154 (1.6%) 39/1159 (3.4%)

Absolute (95% CI) Quality

RR 0.46 (0.27 to 0.81) 20/1 000 (30 fewer/1000 to 10 fewer/1000)  High

Importance

9 14

Intracranial Hemorrhage

(Clinical diagnosis confirmed by CT or post-mortem Follow up: Mean follow-up: 1.5 years) 5 Randomised trials No limitations No important inconsistency No uncertainty Imprecise or sparse data ( 1) 10

Extracranial hemorrhage

(Transfusion or invasive procedure requirement 1 Follow up: ) 5/1244 (0.4%) 2/1159 (0.2%) RR 1.75 (0.47 to 6.41) 3/1 000 (0 more/1000 to 10 more/1000)   Moderate 8 4 Randomised trials Serious limitations (-1) 8 No important inconsistency No uncertainty

All cause mortality

3,5 (Direct patient follow-up Follow up: ) Imprecise or sparse data ( 1) 11 17/1244 (1.4%) 16/1159 (1.4%) RR 0.93 (0.48 to 1.79) 0/1 000 (10 fewer/1000 to 10 more/1000)   Low 7 5 Randomised trials No limitations 15 No important inconsistency No uncertainty None 69/1225 (5.6%) 99/1236 (8%) RR 0.70 (0.52 to 0.95) 20/1 000 (40 fewer/1000 to 1 fewer/1000)  High 9

Vascular death

2 (Death due to stroke, heart disease, hemorrhage and sudden death Follow up: ) 5 Randomised trials No limitations 15 No important inconsistency No uncertainty None 43/1244 (3.5%) 51/1159 (4.4%) RR 0.79 (0.53 to 1.17) 1/1 000 (3 fewer/1000 to 1 more/1000)  High 9

All ischemic stroke

(Neuroimaging or autopsy 6,7 Follow up: 1.5 years) 5 Randomised trials No limitations 8,9 No important inconsistency No uncertainty None 22/1154 (1.9%) 69/1159 (6%) RR 0.32 (0.20 to 0.51) 40/1 000 (60 fewer/1000 to 20 fewer/1000)  High 7

8. In two studies (CAFA; SPINAF) patients and outcome assessors were blind to OAC administration, while in the remaining trials treatment was given open label with outcomes verified by those unaware of treatment assignment.

Process of developing recommendations

Prioritise Problems, establish panel  Systematic Review  Evidence Profile  Relative importance of outcomes  Overall quality of evidence  Benefit – downside evaluation  Strength of recommendation  Implementation and evaluation of guidelines

Process of developing recommendations

GRADE Evidence Profile Author(s):

Schunemann

Date:

8/28/2005

Question:

Should warfarin vs placebo or no treatment be used for patients with non-valvular atrial fibrillation?

Patient or population:

Patients with non-valvular atrial fibrillation

Settings:

Long term outpatient management

Systematic review:

(Neuroimaging or autopsy 6 Follow up: Mean: 1.5 years)

Relative (95% CI)

5 Randomised trials Serious limitations (-1) 8,12 No important inconsistency No uncertainty Strong association (+1) 13 18/1154 (1.6%) 39/1159 (3.4%)

Absolute (95% CI) Quality

RR 0.46 (0.27 to 0.81) 20/1 000 (30 fewer/1000 to 10 fewer/1000)  High

Importance

9 14

Intracranial Hemorrhage

(Clinical diagnosis confirmed by CT or post-mortem Follow up: Mean follow-up: 1.5 years) 5 Randomised trials No limitations No important inconsistency No uncertainty Imprecise or sparse data ( 1) 10 5/1154 (0.4%)

Extracranial hemorrhage

(Transfusion or invasive procedure requirement 1 Follow up: 1.5 years) 2/1159 (0.2%) RR 1.87 (0.51 to 6.82) 3/1 000 (0 more/1000 to 10 more/1000)   Moderate 8 4 Randomised trials Serious limitations (-1) 8 No important inconsistency No uncertainty

All cause mortality

3,5 (Direct patient follow-up Follow up: 1.5 years) Imprecise or sparse data ( 1) 11 17/1154 (1.5%) 16/1159 (1.4%) RR 1.06 (0.54 to 2.09) 0/1 000 (10 fewer/1000 to 10 more/1000)   Low 7 5 Randomised trials No limitations 15 No important inconsistency No uncertainty None 69/1225 (5.6%) 99/1236 (8%) RR 0.70 (0.52 to 0.95) 20/1 000 (40 fewer/1000 to 1 fewer/1000)  High 9

Vascular death

2 (Death due to stroke, heart disease, hemorrhage and sudden death Follow up: 1.5 years) 5 Randomised trials No limitations 15 No important inconsistency No uncertainty None 43/1154 (3.7%) 51/1159 (4.4%) RR 0.85 (0.57 to 1.26) 1/1 000 (3 fewer/1000 to 1 more/1000)  High 9

All ischemic stroke

(Neuroimaging or autopsy 6,7 Follow up: 1.5 years) 5 Randomised trials No limitations 8,9 No important inconsistency No uncertainty Strong association (+1) 16 22/1154 (1.9%) 69/1159 (6%) RR 0.32 (0.20 to 0.51) 40/1 000 (60 fewer/1000 to 20 fewer/1000)  High 7

Footnotes:

GRADEpro

    

Reproducible Transparent

– Footnotes – Judgments

GRADE profiles

– Summary

Integration with Revman Real time

Judgements about recommendations

• “We recommend”…”should” …“Do it” • “We suggest”…”may” … “Probably do it” • “We suggest not”… “may not” …“Probably don’t do it” • “We recommend not”…”should not”… “Don’t do it” No recommendation This could include considerations of costs; i.e. “Is the net gain (benefits-downsides) worth the costs?”