Uses and Impact of systematic reviews and meta

Download Report

Transcript Uses and Impact of systematic reviews and meta

DrPH Seminar Session 1
Use of Systematic Review in Public Health Policy
&
Getting Started Defining SR Questions
Mei Chung, PhD, MPH
Research Assistant Professor
Nutrition/Infection Unit, Department of Public Health and
Community Medicine, Tufts School of Medicine
1
Going Over the Syllabus
• Session 1 (9/11 5:30-8:30pm)
– Homework due by 9/22 (Monday): posting
your homework on the class website
(http://sites.tufts.edu/systematicreviews/ho
me-page/) and comment on others’ posting
• Session 2 (9/22 4-7pm at Sackler 510Computer Lab)
– Bring your “Building a search strategy”
worksheet to the class
– Homework due by 10/8 (Monday)
2
Going Over the Syllabus
• Session 3 (10/9 5:30-8:30pm)
– Homework due by 10/20 (Monday)
• Session 4 (10/23 5:30-8:30pm)
– Post your final presentation slides on the
class website by noon on 10/23
3
Outline of Session 1
• Why systematic reviews are needed in public
health policy and practice
• Very quick overview of SR methods
• SR versus traditional narrative review
– Current debates on the scientific value of SR
• How to formulate a SR research question
• How to evaluate a SR (focusing only on SR
protocol/methods)
4
Evidence-based X
• Evidence-based public health is defined as the
development, implementation, and evaluation
of effective programs and policies in public
health through application of principles of
scientific reasoning, including systematic uses
of data and information systems, and
appropriate use of behavioral science theory
and program planning models.
Source: Brownson, Ross C., Elizabeth A. Baker, Terry L. Leet, and Kathleen N.
Gillespie, Editors. Evidence-Based Public Health. New York: Oxford University
Press, 2003.
5
Evidence-based X
• Evidence-based dietetic practice is the use of
systematically reviewed scientific evidence in
making food and nutrition practice decisions by
integrating best available evidence with
professional expertise and client values to
improve outcomes.
Source: Academy Scope of Dietetics Practice Framework
https://www.andeal.org/evidence-analysis-process-overview
6
Organizations producing systematic reviews or using
systematic review to inform evidence-based policy and
practice guidelines
http://sites.tufts.edu/systematicreviews/mainpage/
7
Synthesizing Evidence
• Narrative Reviews
• Systematic Reviews
• Meta-Analysis
• Decision Analysis
• Cost-effectiveness analysis
• Clinical practice guidelines
• Algorithms
8
What is a Systematic Review?
(Sometimes called systematic evidence-based reviews
or evidence review)
•Systematic review
– a comprehensive summary of all available
evidence that meets predefined eligibility criteria to
address a specific clinical question or range of
questions
•Meta-analysis
– commonly included in systematic reviews, a
statistical method that quantitatively combines the
results from different studies
9
Basic Steps in a SR
Prepare
topic
Search for
studies
Screen studies Extract
data
Analyze and
synthesize
data
• Apply
qualitative
and/or
quantitative
methods
Report
findings
10
Steps of Performing a Systematic Review
Ask
FORMULATE STUDY QUESTION
ESTABLISH PROTOCOL
Identify
LITERATURE SEARCH / RETRIEVAL
Acquire
CRITICAL APPRAISAL
PAPER SELECTION per PROTOCOL
Appraise
Synthesize
DATA EXTRACTION and
QUALITY ASSESSMENT
ANALYSIS and INTERPRETATION
WEIGHTED AVERAGE
REGRESSION
SENSITIVITY ANALYSIS
11
Systematic review process flowchart
12
Systematic review and
meta-analysis is a
retrospective exercise,
suffering from all the
limitations of being an
observational design.
13
For all research, and for
systematic review in particular,
a clear research question is needed
An important clinical/public health
question
might not be a meaningful
research question
14
Steps of Performing a Systematic Review
Ask
FORMULATE STUDY QUESTION
ESTABLISH PROTOCOL
Identify
LITERATURE SEARCH / RETRIEVAL
Acquire
CRITICAL APPRAISAL
PAPER SELECTION per PROTOCOL
Appraise
Synthesize
DATA EXTRACTION and
QUALITY ASSESSMENT
ANALYSIS and INTERPRETATION
WEIGHTED AVERAGE
REGRESSION
SENSITIVITY ANALYSIS
15
Example - poorly formulated
question:
Should dietary supplements be
recommended to patients with
hypertension?
16
Example: The Well - Formulated Question
The Cochrane Collaboration "How to Conduct a Cochrane Systematic Review" 1996
Intervention
Outcomes
Population
Setting
Condition of
interest
Does omega-3 fatty acids decrease long-term CVD outcomes in older persons with mild to moderate hypertension?
Fish oil
EPA+DHA
ALA
Fish intake
Walnut
> 1 year
Fatal and non-fatal strokes
Fatal and non-fatal
Coronary Heart Disease
(MI, sudden death)
Cardiovascular events
(above plus aneurysm,
congestive heart failure,
transient ischemic
attacks)
Total Mortality
> 60 yrs old
outpatients
Systolic 140-179
Diastolic 90-109
17
PICO approach to formulating
answerable research question
Counsell, 1997
•
•
•
•
•
P
I
C
O
D
population
intervention (or exposure)
comparator
outcomes
study design
You’ll also see PICO, PICOS (study design),
PICODD (+duration), PICOT (time), and others
18
PI(E)COS approach to formulating answerable
research question
•What is the relevant population?
•What is the intervention/exposure of
interest?
•What is the appropriate comparison?
•What are the important outcomes of
interest?
•In what setting would the results be
applicable?
You’ll also see PICOT (timing),
PICOD (design/duration), and others
19
The PICO method to formulate research
question on interventions
Population
Primary
prevention
Secondary
prevention
Interventions /
Exposure
Fish, fish oil,
ALA
Dosage
Comparator
Outcomes
Placebo
Overall
mortality
No control
Sudden death
Background
intake
Active
comparator
Revascularization
Duration
Stroke
Blood
pressure
20
Populations
Problems with defining condition
• Varying definitions
– lack of an adequate reference standard
(e.g., patients with anemia; patients with
metabolic syndrome)
• Different levels of rigor
– Loose vs. strict definitions (e.g. elderly vs.
adults men and women who are 60 years
old or grater)
– Applicability/generalizability tradeoffs
21
Example – population of interest
• Primary Prevention - patients without prior
history of cardiovascular disease
– Country
– Background diet
• Secondary Prevention – patients with prior
history of cardiovascular disease
22
Example – Intervention / Exposure of
Interest
What is an omega-3 fatty acid?
• EPA, DHA, (fish oil, fish)
– Levels differ by type of fish
– Levels (and/or effect) may differ by
preparation (broiled, fried fish sandwiches)
• ALA (plant source: walnut, canola oil,
mustard seed, etc.)
23
Example – Outcomes of interest
• Hard outcomes (clinical events)
–
–
–
–
–
Overall mortality
Stroke
Myocardial infarction
Sudden death
Revascularization
• Soft [surrogate, intermediate] outcomes
(biomarkers, measurements)
– Coronary flow
– Blood pressure
– Lipid levels
• Intermediate
– Diagnosis of hypertension
24
Analytic Framework
• Series of specific questions can be
formulated into a model that analyzes all
effects and interactions between intervention
or exposure and outcomes
• Analytic framework can be used to clarify and
generate questions (topics)
• Can highlight what aspects are known and
unknown
• Can clarify what study designs may be best to
address specific questions
25
25
USPSTF, Nelson et al., Ann Intern Med. 2009;151(10):727-737
26
26
Omega-3 FAs and CVD
Target Populations
Healthy Adults
Adults with elevated risk for CVD
Adults with known CVD
Diabetes, Hypertension, Hyperlipidemia
Adverse
Events
Omega-3 Fatty Acid Consumption
Source, Dosage, Duration
Tissue / Plasma Levels
Plasma Phospholipid FAs
Platelet Phospholipid FAs
Modifiers
RBC Phospholipid FAs
WBC ghost Phosphlipid FAs
Others
Drug Interactions
Other
Intermediate Outcomes / Biological Effects
Cholesterol
Lipoprotein (a)
C Reactive Protein
Fibrinogen
Blood Pressure
Diabetes Markers
Heart Rate Variability
Carotid Intima Media Thickness
Coronary Arteriography Markers
Others
Clinical Outcomes
Death
Myocardial Infarction
Stroke
Unstable Angina
Ventricular Arrhythmia
Others
27
27
Steps of Performing a Systematic Review
Ask
Identify
Acquire
Appraise
Synthesize
FORMULATE STUDY QUESTION
ESTABLISH PROTOCOL
LITERATURE SEARCH / RETRIEVAL
CRITICAL APPRAISAL
PAPER SELECTION per PROTOCOL
Will be
covered in
session 2
DATA EXTRACTION and
QUALITY ASSESSMENT
ANALYSIS and INTERPRETATION
WEIGHTED AVERAGE
REGRESSION
SENSITIVITY ANALYSIS
28
Goals of Data Extraction & Quality
Assessment
• Data extraction
– To collect key study characteristics and
results from published articles pertaining
to the SR research question
– Important to use a standardized form,
customized for the SR research question
• Quality / risk of bias assessment
– To avoid “Garbage in, garbage out”
– To assess the confidence in the validity of
study findings
29
Rationale for Quality
Assessment
• Quality assessment of all studies
included in the SRs is important:
– estimate extent to which study’s design
and methods prevented systematic errors
(biases)
– variation in quality may explain differences
in results of SRs
– necessary even if there is little variability
among studies (consistent trash is still
trash)
30
Tools for Quality/Risk of Bias Assessment
• Many tools, but few “validated” tools:
– Cochrane risk of bias assessments
(http://bmg.cochrane.org/assessing-risk-bias-includedstudies): RCTs
– The Newcastle-Ottawa Scale
(http://www.ohri.ca/programs/clinical_epidemiology/oxford.as
p): Observational studies
• No well-accepted nutrition/public health
specific [content specific] quality
assessment tools
– Lichtenstein AH, Yetley EA, Lau J. Application of
systematic review methodology to the field of
nutrition. J Nutr 2008;138:2207-2306
31
Linking Quality Assessment to Analysis
• as a threshold for inclusion and
exclusion of studies in the review
(generally not recommended)
• as a possible explanation for differences
in results between studies
• as a variable in sensitivity analysis (test
of robustness)
• as weights in statistical analysis of the
results
32
Steps of Performing a Systematic Review
Ask
FORMULATE STUDY QUESTION
ESTABLISH PROTOCOL
Identify
LITERATURE SEARCH / RETRIEVAL
Acquire
CRITICAL APPRAISAL
PAPER SELECTION per PROTOCOL
Appraise
Synthesize
DATA EXTRACTION and
QUALITY ASSESSMENT
ANALYSIS and INTERPRETATION
WEIGHTED AVERAGE
REGRESSION
SENSITIVITY ANALYSIS
33
Qualitative & Quantitative Syntheses
• Qualitative synthesis – required
– Summary tables (many different forms)
• Key study characteristics
• Summary of study results
– Graphical presentation of study results (a plus)
• Quantitative synthesis (a.k.a meta-analysis) optional
– highly depending on types of results/data, and
reporting of the data
– may not be appropriate – to pool or not to pool can
be a tricky decision
34
Summary Tables (I)
• Combine data from multiple studies to illustrate
trends in the data
• May be focused on describing study
characteristics, results, or both
• Can be designed to include characteristics of all
included studies
– Examples: funding sources, assessment method, country of study
• Can be designed for subsets of included studies
– Examples: summary tables for randomized controlled trials,
prevalence studies, harms/side effects, outcomes for specific
treatments
35
Summary Tables (II)
• Simplified entry (one row) for each study
• Table columns may include, for example:
–
–
–
–
–
PICOTS (may be listed in table title or headers)
Methodological quality
Applicability
Study size (weight)
Magnitude of effect
• A single study may be represented in multiple
summary tables (e.g., different outcomes)
PICOTS = population, intervention, comparator, outcomes, timing, and setting
36
Example:
Summary Table of Study Characteristics
 A basic summary table is
the “study
characteristics” table.
 The overall summary
provides an overview of
the state of the available
studies in the literature.
Hartmann KE, et al. AHRQ Evidence Report/Technology Assessment
No. 187. Available at:
http://www.ahrq.gov/downloads/pub/evidence/pdf/ bladder/bladder.pdf.
37
Example:
Summary Table of Study Characteristics (More
descriptive, most common)
38
Example:
Summary Table for Cohort Studies
Wang C, et al. AHRQ Evidence Report/Technology Assessment No. 94.
Available at: http://www.ahrq.gov/downloads/pub/evidence/pdf/o3cardio/o3cardio.pdf.
39
Example:
Summary tables can be specialized for different types of outcomes
40
Summary Matrix
Wang C, et al. AHRQ Evidence Report/Technology Assessment No. 94.
Available at: http://www.ahrq.gov/downloads/pub/evidence/pdf/o3cardio/o3cardio.pdf.
41
Example:
Graphical
presentation
of the study
results
42
An Example of an Assessment of
Strength of Body of Evidence
High
• High level of assurance with the validity of the results (based on
quality, applicability, effect size, consistency) for the key question
• At least 2 high quality studies with long-term followup
• No important disagreement across studies
Moderate
• Good to moderate level of assurance with the validity of the
results
• Fewer than 2 high quality studies
• Little disagreement across studies in the results
Low
• Low level of assurance with the validity of results
• Based on studies of moderate to poor quality or limited
applicability
Insufficient
• Little data or disagreement across or within studies
43
43
Key Messages
• Summary tables provide key information on study
characteristics and study findings. Through table
and graphical formats, respectively
• Properly constructed summary tables:
– Effectively convey results
– Provide an overview of the literature in a given field
– Enable the reader to grasp results for subsets of the literature
44
Synonyms of Meta Analysis
• Quantitative overview/Synthesis
• Pooling
– Less precise
– Suggests that data from multiple sources are
simply lumped together
• Combining
– Preferred by some
– Suggests applying statistical procedures to data
45
Reading a Generic Forest Plot
Reference: Szajewska H. The role of meta-analysis in the evaluation of the effects of early nutrition on
mental and motor development in children. Am J Clin Nutr. 2011 Dec;94(6 Suppl):1889S-1895S. Epub
2011 Apr 27. Review
46
Heterogeneity of Data
~Diversity~
Are the characteristics and effects of studies sufficiently similar
to estimate an average effect?
Clinical
Methodological
• Are studies of
similar
treatments,
populations,
settings,
design etc.
such that an
average effect
would be
clinically
meaningful?
• Are studies of
similar design
and conduct
such that an
average effect
would be
clinically
meaningful?
Statistical
• Is the observed
variability of
effects greater
than that
expected by
chance alone?
47
Summary
48
RCT
Observational
Systematic review
of RCTs
49
PRISMA Checklist
50
PRISMA Checklist
51
PRISMA Checklist
52
References
•
•
•
IOM (Institute of Medicine). 2011. Finding What Works in Health Care:
Standards for Systematic Reviews, Washington, DC: Natl. Acad. Press
IOM (Institute of Medicine). 2011. Clinical Practice Guidelines We Can Trust.
Washington, DC: The National Academies Press.
Systematic reviews: synthesis of best evidence for clinical decisions. Ann Intern
Med. 1997 Mar 1;126(5):376-80.
53