Transcript A Framework For Evaluating Health Care Program

Systematic Reviews of the Literature
and Meta-analyses:
….problems or panacea?
Daren K. Heyland, MD, FRCPC, MSc
Queen’s University, Kingston, Ontario

Updated Jan 2009

Summarizes >200 trials studying 21283 patients

34 topics
17 recommendations
www.criticalcarenutrition.com
Clinical Practice Guidelines
evidence
+
practice
guidelines
integration of values
Validity
Homogeneity
Safety
Feasibility
Cost
In Search of Truth...
…Does it work?

Begins with a hypothesis or question



Does Drug X reduce the incidence of problem Y in patients
with condition Z
Application of experimental or observational methods to
determine the answer
Results of our observations leads to conclusions that are
correct (truth) or incorrect (due to bias or chance)
Levels of Evidence
Systematic reviews
 RCT’s
 Cohort Studies
 Case Control
 Case Series

less bias/strong inferences
more bias/weaker inferences
RCT Average Patient Population Size
per Year
250
200
150
100
50
0
1976 1983 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008
198 RCT’s Reviewed in Critical Care Nutrition Guidelines
PLOS 2008;5: e4
Learning Objectives
 Will
be able to appraise and incorporate results of
systematic reviews into clinical decision making.
 understand
the role of systematic reviews in
research and policy settings.
 List
the strengths and weakness of meta-analyses
Overview
 Definition
and Classification
 Usefulness
 Methodological Quality
 Making Inferences
 Conclusions
Systematic Review…
 Form
of scientific investigation to assess the
effectiveness of healthcare interventions
 Integrative research
 Subjects= original or primary studies
 Employs methods that limit bias and reduce random
error
Feature
Narrative
Review
Systematic
Review
Question
No specific question,
usually broad in scope
Not usually specified,
potentially biased
Not usually specified,
potentially biased
Variable
Focused clinical
question
Comprehensive, explicit
strategy
Criterion-based
selection
Rigorous critical
appraisal
Qualitative +
Quantitative
Evidence-based
Search
Selection
Appraisal
Synthesis
Qualitative
Inferences
Sometimes evidencebased
Systematic Reviews and
Meta-analysis
Narrative
Reviews
Systematic
Reviews
Meta-analysis
Number of Systematic Reviews Published
The Frailties of Narrative Reviews
 If
the original studies of thrombolytics therapies
had been subject to a systematic review, the
treatment effect would have been apparent in the
1970s instead of 1980s.
 Narrative reviews omitted effective therapies and
endorsed ineffective therapies.
Antman JAMA1992;268;240 and Lau NEJM 1992;327:248
Clinical Decision Making and
Systematic Reviews

Case Scenario
 77
y.o. male with presumptive Dx of Urosepsis
 PMHX: MI, Prostate
 BMI 21
 After initial resuscitation





FiO2 = 100%, PO2 = 55
MAP = 65, CVP 13, levophed 20 mcg/kgk/min
rising Cr, 20 ml of urine, acidemic
High NG drainage
Going to start on EN but not likely to tolerate
Role for early supplemental PN?
Clinical Decision Making and
Systematic Reviews
 Problem
 100s
of citations across scores of journals published over
the last 20 years In diverse patient populations or diverse
settings with variable or inconsistent results!
How do you make sense
of this all?
kcal
Impact of Caloric Debt
Adequacy of
EN
Prescribed Engergy
2000
1800
1600
1400
1200
1000
800
600
400
200
0
Energy Received From Enteral Feed
Caloric Debt
1
3
5
7
9
11
13
15
17
19
21
Days
 Caloric debt associated with:


Longer ICU stay
Days on mechanical ventilation

Complications

 Mortality
Rubinson CCM 2004; Villet Clin Nutr 2005; Dvir Clin Nutr 2006; Petros Clin Nutr 2006
2007 International Nutrition
Practice Survey
 Point
prevalence survey of nutrition practices in
ICU’s around the world conducted Jan. 27, 2007
 Enrolled 2772 patients from 158 ICU’s over 5
continents
 Included ventilated adult patients who remained
in ICU >72 hours
Hypothesis
 There
is a relationship between amount of energy
and protein received and clinical outcomes (mortality
and # of days on ventilator)
 The relationship is influenced by nutritional risk
 BMI is used to define chronic nutritional risk
What Study Patients Actually Rec’d
 Average
 1034
Calories in all groups:
kcals and 47 gm of protein
Result:
 Average caloric deficit in Lean Pts:
 7500kcal/10days
 Average
caloric deficit in Severely Obese:
 12000kcal/10days
Relationship Between Increased Calories
and 60 day Mortality
BMI Group
Odds
Ratio
Overall
0.76
0.61
0.95
0.014
<20
0.52
0.29
0.95
0.033
20-<25
0.62
0.44
0.88
0.007
25-<30
1.05
0.75
1.49
0.768
30-<35
1.04
0.64
1.68
0.889
35-<40
0.36
0.16
0.80
0.012
>=40
0.63
0.32
1.24
0.180
95%
Confidence
Limits
P-value
Legend: Odds of 60-day Mortality per 1000 kcals received per day adjusting for nutrition days, BMI, age,
admission category, admission diagnosis and APACHE II score.
RESULTS:
WHO IS AT RISK?
RCT Level of Evidence that
More EN= Improved Outcomes

RCTs of aggressive feeding protocols


Results in better protein-energy intake
Associated with reduced complications and improved survival
Taylor

et al Crit Care Med 1999; Martin CMAJ 2004
Meta-analysis of Early vs Delayed EN


Reduced infections: RR 0.76 (.59,0.98),p=0.04
Reduced Mortality: RR 0.68 (0.46, 1.01) p=0.06
www.criticalcarenutrition.com
More is Better!
Our Field of Dream
If you feed them (better!)
They will leave (sooner!)
ICU patients are not all created equal…should we
expect the impact of nutrition therapy to be the same
across all patients?b
What if you can’t provide
adequate nutrition enterally?
… to TPN or not to TPN,
that is the question!
Current practice in nutritional support in
septic patients: Results of national,
prospective multicenter German Study
 Point
prevalence study
 454 ICUs from 310 hospitals
in Germany
 399 patients septic patients
included
EN only
PN only
EN +PN
none
 Median
APACHE II 26
 68% had no GI pathology
 46% in shock
 Overall mortality 55.2%
Elke
CCM 2008;36:1762
Current practice in nutritional support in
septic patients: Results of national,
prospective multicenter German Study
70
 Point
prevalence study
 454 ICUs from 310 hospitals
in Germany
 399 patients septic patients
included
 Median
APACHE II 26
 68% had no GI pathology
 46% in shock
 Overall mortality 55.2%
P=0.005
60
50
40
%
mortality
30
20
10
0
EN only PN only EN +
PN
none
Multivariate analysis:
PN independent predictor for mortality
(OR 2.09, 95% CI 1.29-3.37)
Early Supplemental PN is Associated with
Increased Infection in
Critically Ill Trauma Patients
 Retrospective,
multicenter, cohort study of 597 severely
injured patients
 Compared those that rec’d PN within 7 to those who did
not.
 Also compared early PN group to subgroup of ‘EN
tolerant’ (tolerated 1000 kcal any day during first week)
 Adjusted for differences in key baseline demographics
Sena J Am Coll Surg 2008;207:459
Early Supplemental PN is Associated with Increased
Infection in Critically Ill Trauma Patients
No Early PN
Early PN
Odds Ratio
P value
Nosocomial Infections
27%
56%
2.1 (1.3-3.5)
P=0.003
Late ARDS
1%
8%
3.4 (1.0-11.0)
P=0.04
Death
8%
23%
1.5 (0.8-3.0)
P=0.24
Nosocomial Infections
42%
69%
2.5 (1.1-5.9)
P=0.03
Late ARDS
2%
9%
5.4 (1.1-27.4)
P=0.04
Death
8%
19%
2.7 (0.8-9.3)
P=0.10
Overall Adjusted
EN tolerant analysis
Differences not due to differences in glycemic control
Prospective Studies of Supplemental PN
Effect on Mortality
www.criticalcarenutrition.com
What if you can’t provide
adequate nutrition enterally?
… to TPN or not to TPN,
that is the question!
Maximize EN delivery prior to
initiating PN
Use of Supplemental PN
in Sepsis?
 Results
of meta-analysis
 Results of single RCTs of Septic Patients
 Results of observational studies
 Consideration of Individual Patient
Characteristics
Using Systematic Reviews in
Clinical Practice
 Summarizes
large body of knowledge
 Answers specific clinical question
 Less likely to be biased than narrative reviews
 More accurate and precise estimate of treatment
effect
Using Systematic Reviews in
Research Setting
 Research
Question:
 What
is the effect of Glutamine and Antioxidant
supplementation on survival in critically ill patients?
 Methods:
A
meta-analysis
Effect of Glutamine in Critically Ill:
A Systematic Review of the Literature
 Comprehensive
 Selection
search
criteria
 Randomized
 Surgical
or critically ill adults
 Glutamine
 Clinically
(EN or PN) vs. placebo
important outcomes
20 RCT’s
Effect of Glutamine:
A Systematic Review of the Literature
Mortality
Updated Jan 2009, see www.criticalcarenutrition.com
Effect of Glutamine:
A Systematic Review of the Literature
Infectious Complications
Updated Jan 2009, see www.criticalcarenutrition.com
Effect of Glutamine:
A Systematic Review of the Literature
Hospital Length of Stay
Updated Jan 2009, see www.criticalcarenutrition.com
Results of Subgroup Analysis
EN
(n=9)
PN
(n=17)
Mortality
Infection
0.81 (0.48-1.34)
P=0.41
0.71 (0.55-0.92)
P=0.008
0.83 (0.64-1.08)
P=0.16
0.76(0.62-0.93)
P=0.008
PN>>>EN?
REducing Deaths from OXidative Stress:
The REDOXS study
Factorial 2x2 design
antioxidants
glutamine
R
1200 ICU patients
Evidence of
organ failure
Fed enterally
Concealed
R
placebo
Stratified by
site
antioxidants
placebo
R
placebo
Using Systematic Reviews in
Research Setting
 Summarizes
what is known; identifies gaps
 Background of grant proposals
 Generates hypotheses
 Estimate of treatment effect
N
 Subgroup analysis
Using Systematic Reviews in
Policy Making
As an ICU, should you make an argininesupplemented diet available for general use in
your institution?
Meta-analyses of
Arginine-supplemented Diets
o
22 RCTs of IEDs




All arginine-containing IED, not just IMPACT/IMMUNAID
Non english, more recently published studies
Excluded duplicates
Excluded single agents
Heyland JAMA 2001;286:944
Overall Effect on Mortality
RR
1.10 (0.93-1.31)
Overall Effect on Complications
RR
0.66 (0.54-0.80)
1.18 (0.88,1.58)
Effect of Arginine-supplemented Diets
Mortality
in the Critically Ill Patient
Updated Jan 2009, see www.criticalcarenutrition.com
Effect of Arginine-supplemented Diets
in the Critically Ill Patient
Infectious Complications
Updated Jan 2009, see www.criticalcarenutrition.com
Effect of Arginine-supplemented Diets
in the Critically Ill Patient
Hospital Length of Stay
Updated Jan 2009, see www.criticalcarenutrition.com
Using Systematic Reviews in
Policy Making
 Greatest
generalizability
 Consistent with perspective of policy makers
 Related to other forms of integrative research
Assessing the Validity of
Systematic Reviews
Validity= fxn { inputs, process, results }
Assessing the Validity of
Systematic Reviews
 Inputs
 selection
of studies
 clinical
homogeneity
 explicit, reproducible criteria
 methodological
quality of studies
 outdated/unmeasured co-interventions
Assessing the Validity of
Systematic Reviews
 Process
 comprehensive
search strategy
 publication/timing
bias
 data
excess
 language bias
 judgements
about inclusion explicit/reproducible
 data abstraction reproducible
Assessing the Validity of
Systematic Reviews
 Results
 few
studies
 few clinical endpoints
 statistical heterogeneity
Methdological Quality of
Meta-analyses
lots of bias
weak
inferences
little bias
strong
inferences
Strong clinical
recommendations
Making Inferences from a
Meta-Analysis of RCT’s
Weaker Inferences






Small number of trials
Weak trial methodology
Outdated/unmeasured
co-interventions
Surrogate endpoints
Statistical heterogeneity
Fixed effects model
Stronger Inferences






Large number of trials
Strong trial methodology
Current/documented cointerventions
Clinically important
endpoints
Statistical homogeneity
Random effects model
Meta-analysis vs. Large RCT’s
“…if no subsequent randomized, clinical trial,
the meta-analysis would have led to the
adoption of an ineffective treatment in 32%
cases and rejection of useful treatment in
33% cases.”
LeLorier NEJM 1997;337:536
“I still prefer conventional narrative reviews …
Editorial, NEJM
Meta-analysis vs. Large RCT’s
RCT #2
RCT #1
RCT #3
RCT #5
RCT #4
Meta-analysis vs. Large RCT’s
 Argument
is with Meta-analysis, not the concept of
systematic reviews
 Assumes the latest single large trial is the GOLD
standard
 Assumes RCT and Meta-analysis are measuring
the same thing
 Differences in Generalizability
 Bias exists in both TOOLS.
Resolving Discrepancies Between a Metaanalysis and a Subsequent Large RCT




Recent meta-analysis found calcuim
supplementation to be effective in preventing
preeclampsia
Large RCT found no risk reduction in health
nulliparous women
Exploration of heterogeneity across studies
Stratify for high and low baseline risk
JAMA 1999;282:664
Resolving Discrepancies Between a Metaanalysis and a Subsequent Large RCT
JAMA 1999;282:664
JAMA
2008;300:933
Role of Systematic Reviews
in Medical Education
 Good
source of medical knowledge
 Promotes EBM practices
 Helps locate original articles
 Facilitates critical appraisal of original research
 Considered a scholarly research activity
Conclusions
 Important
tool to determine the effectiveness of
therapeutic interventions
 Need to understand the strengths, weaknesses and
limitations
 Useful in clinical and policy decision making and
research setting
 Encourage use of and generation of systematic
reviews amongst learners.