Transcript An Introduction to Systematic Reviews

An Introduction to
Systematic Reviews
1
Information explosion
2
Daily
• 46 RCTs
• 1000 Medline New articles
• 6,000 biomedical articles
3
Annually
 3 million articles
 30,000 journals
4
Increasing Knowledge
Number of articles on Hypertension cited in
Medline by Year
8000
6000
4000
Articles
2000
0
1966
1976
1986
1996
5
Most research published in medical journals is
too poorly done
or
insufficiently relevant
to be clinically useful
6
Too much information, too little time
• There is simply too much information
around for people to keep up to date.
• On top of this, high quality information is
often not easy to find.
7
Review articles
8
• A ‘review’ is the generic term for any
attempt to synthesis the results and
conclusions of two or more publications on
a given topic.
9
Some reviews are usually based on narrative or
commentary and are produced by a
‘content expert’
10
What’s the problem with
“Expert Opinion”?
11
Professor Paul Knipschild has described how
Nobel prize winning biochemist Linus Pauling
used selective quotes from the medical literature to
"prove" his theory that
vitamin C helps you live longer and feel
better.
12
When Knipschild and his colleagues searched the
literature systematically for evidence
They found that
13
• One or two trials did strongly suggest that
vitamin C could prevent the onset of the
common cold.
• There were far more studies which did not
show any beneficial effect.
14
Unfortunately, expert reviewers often :
• Make conflicting recommendations
• Their advice frequently lags behind
• Inconsistent with the best available evidence.
15
The use of unsystematic approaches to
collecting and summarizing the evidence.
16
In one study, self-rated expertise was
inversely related to the methodologic rigor
of the review
Oxman AD, Guyatt GH. The science of reviewing research. Ann N Y Acad Sci.
1993;703:125-133; discussion 133-134.
17
The problems are that …
• Too much information, too little time
• Many too poorly done or insufficiently
relevant to be clinically useful
• Many have conflicting results
• On top of these, high quality information is
often not easy to find
18
Timely response
• A classic example: streptokinase in
myocardial infarction
– First trial published in 1959
– 15 trials published up to 1977
– Introduced as normal practice in late 80s
– Meta-analysis of treatments in 1992: fall in
mortality by 1977, after inclusion of 15 trials
19
Thrombolytic Therapy
Textbook/Review
Recommendations
2.0
2
65
3
149
4
316
7
1793
10
11
15
17
22
2544
2651
3311
3929
5452
23
5767
27
30
33
43
54
65
67
70
6125
6346
6571
21 059
22 051
47 185
47 531
48 154
1960
1965
1970
1980
1985
1990
Experimental
23
Rare/Never
1
Specific
Pts
Not Mentioned
1.0
Routine
Cumulative
Year
RCTs
0.5
21
5
P<.01
1
10
1
2
2
8
7
8
1
P<.001
P<.00001
M
M
1
M
M
5
1
2
15
8
6
1
M
M
Odds Ratio (Log Scale)
Favours Treatment
Favours Control
12
8
7
2
4
3
1
1
20
Bias
Bias means something that will cause a
consistent deviation from the truth.
This is different from the play of chance.
21
There are three possible sources of bias in reviews:
• bias arising from the studies included
in the review
• bias arising from the studies not
included in the review
• Bias arising from the way the review is
done.
22
Minimizing bias
We need to do as much as possible to
minimize the effects of anything that will
cause the results to deviate from the truth.
23
What is a Systematic Review?
24
Systematic review Or Overview
Comprehensively
• locates
• evaluates
• synthesizes
all the available literature on a given topic
using a strict scientific design which
must itself be reported in the review
25
A ‘systematic review’, therefore, aims to be:
• Systematic (e.g. in its identification of
literature)
• Explicit (e.g. in its statement of objectives,
materials and methods)
• Reproducible (e.g. in its methodology and
conclusions)
26
The ‘systematic’ part of systematic reviews is all about
minimizing bias in the way
the review is carried out
27
Systematic reviews are the same as
ordinary reviews, only bigger!
• not simply "comprehensive"
• but to answer a specific question
• to reduce bias in the selection and inclusion of studies
(language, database, publication, reporting, citation,
multiple publication)
• to appraise the quality of the included studies
– Internal validity: minimised systematic error (bias)
 External validity: generalisability of findings
• to summarise them objectively
28
They are different!!
29
The UK Medical Research Council now
requires a systematic review has been prepared
before it will commission a new trial.
• The question has not already been answered
• The results of previous research are used in designing
the new trial.
30
The Cochrane Collaboration
www.cochrane.org
31
Archie Cochrane
“It is surely a great criticism of our profession that we have not
organised a critical summary, by specialty or subspecialty, adapted
periodically, of all relevant randomised controlled trials.”
32
Section 2
33
Conducting a Systematic Review
34
Stages of a systematic review
• Planning the review – i.e. identifying the need
for a review, and documenting the methodology
• Conducting the review – i.e. finding, selecting,
appraising, extracting and synthesising
primary research studies
• Reporting and dissemination – i.e. writing up
and disseminating the results of the review
35
Development of a review protocol
36
Formulating review questions
Searching & selecting studies
Study quality assessment
Extracting data from studies
Data synthesis
37
‫‪Step 0‬‬
‫• انجام اين مطالعات به پول‪ ،‬زمان و نيروی متخصص نياز دارند‪.‬‬
‫• به منظور اولويتبندي مداخالت و پژوهشهاي مربوط به‬
‫‪ HIV‬الزم بود شيوع و قدرت اثرهريك ازعوامل تعيين كنندهي‬
‫اين بيماري محاسبه گردد‪.‬‬
‫‪38‬‬
Formulating review questions
• The first and most important decision in
preparing a review is to determine its focus
• This is best done by asking clearly framed
questions.
39
Formulating review questions
• define a four part clinical question
breaking the question down
into its component parts
40
Question components : PICO
• What types of
• What types of
• What types of
• What types of
Participants?
Interventions?
Comparison?
Outcomes?
41
A PICO question
Time-consuming question:
What is the best strategy to prevent smoking in
young people?
42
An answerable question
Q. Are mass media (or school-based or communitybased) interventions effective in preventing smoking in
young people?
43
How are these questions different?
• Does aspirin improve survival after acute
myocardial infarction?
• In patients with acute myocardial infarction,
does daily, low-dose, oral aspirin lead to
higher survival rates as compared to
placebo?
44
Formulation of a therapy question
Intervention
Outcome
Is Zinc effective in treating cold?
Patient/problem
Intervention
In children with common cold, is oral Zinc effective in
reducing the duration of symptoms, as compared to placebo?
Outcome
+ RCTs
Comparison
45
‫سواالت مرور سيستماتيك‬
‫• شيوع عدم آگاهي در مورد راههاي انتقال ‪ ،HIV‬رفتار پر خطر‬
‫جنس ي و اعتياد تزريقي در گروههاي مختلف مردم ايراني باالي‬
‫‪ 14‬سال در ‪ 10‬سال اخير چه ميزان ميباشد؟‬
‫• بزرگي اثر هر يك از عوامل فوق در آلودگي با ‪ HIV‬در گروههاي‬
‫مختلف مردم ايراني باالي ‪ 14‬سال چگونه است؟‬
‫‪46‬‬
Formulating review questions
Searching & selecting studies
Study quality assessment
Extracting data from studies
Data synthesis
47
Selecting studies
• performing a comprehensive, objective, and
reproducible search of the literature
• selecting studies which meet the original
inclusion and exclusion criteria
can be the most time-consuming and
challenging task in preparing a systematic
review
48
Data sources for a systematic review
• Electronic databases
– MEDLINE and EMBASE
– Specialized or local databases (CINAHL,
CENTRAL, AMED,…)
• Hand searching
• “Grey literature” ( thesis, Internal reports, pharmaceutical
industry files)
• Checking reference lists
• Unpublished sources known to experts in the specialty
(seek by personal communication)
• Raw data from published trials
49
Developing a search strategy
• It is always necessary to strike a balance
between comprehensiveness and precision
when developing a search strategy.
50
Selecting studies
• assessing titles and abstracts
• full text
51
Searching the literature
• Published and unpublished literature
• unbiased search (not just Medline which
contains < half of trials)
• conference papers, company reports, et.
• personal communication with experts
52
‫•‬
‫•‬
‫•‬
‫ايط انتخاب‬
‫ر‬
‫ش‬
‫نوع مطالعه‬
‫توصيفي‬
‫توصيفي‪-‬تحليلي‬
‫تحليلي‬
‫جمعيت تحت مطالعه‬
‫مردان يا زنان باالي ‪ 14‬سال ايراني‬
‫مكان مطالعه‬
‫ايران _ مديترانه شرقي _ جهان‬
‫• زمان مطالعه‬
‫‪ 10‬سال اخير‬
‫• محتوي مطالعه‬
‫آگاهي و نگرش نسبت به انتقال يا پيشگيري از ‪HIV‬‬
‫اثر آموزش بر آگاهي يا نگرش نسبت به راههاي انتقال ‪HIV‬‬
‫اعتياد تزريقي‬
‫رفتار پر خطر جنس ي‬
‫كيفيت قابل قبول مطالعات‬
‫•‬
‫‪53‬‬
‫جستجوي مقاالت و اسناد‬
‫منابع جستجو‪:‬‬
‫•‬
‫•‬
‫‪IRAN MEDEX‬‬
‫‪IRAN DOC‬‬
‫•‬
‫•‬
‫•‬
‫•‬
‫چكيده مقاالت كنگره ها‬
‫پايان نامه ها‬
‫مجالت داخلي و خارجي‬
‫مراكز دولتي (وزارت بهداشت‪ ،‬مطبوعات‪،‬‬
‫بهزيستي‪ ،‬وزارت ارشاد‪ ،‬مركزمديريت بيماريها‪،‬‬
‫زندان‪ ،‬مراكز تحقيقاتي)‬
‫‪Cochrane Library‬‬
‫•‬
‫•‬
‫•‬
‫‪MEDLINE‬‬
‫‪EMBASE‬‬
‫بررس ي مراجع مقاالت يافت شده‬
‫•‬
‫‪54‬‬
‫•‬
‫•‬
‫•‬
‫‪NGO‬ها‬
‫مصاحبه با افراد صاحبنظر وكليدي‬
‫سازمانهاي جهاني(‪)UN-WHO‬‬
‫لغات كليدي مورد استفاده‬
‫‪AIDS‬‬
‫‪HIV‬‬
‫‪Knowledge‬‬
‫‪Attitude‬‬
‫‪Knowledge/ Attitude/practice‬‬
‫‪Risk factors‬‬
‫)‪Intra venous drug use (IDU‬‬
‫‪High risk sex‬‬
‫‪Sex workers, heterosexual,‬‬
‫‪homosexual, MSM‬‬
‫‪STI‬‬
‫‪IRAN‬‬
‫‪Prevalence‬‬
‫‪55‬‬
‫‪relative risk,odds ratio, effect‬‬
‫•‬
‫•‬
‫•‬
‫•‬
‫•‬
‫•‬
‫•‬
‫•‬
‫•‬
‫•‬
‫•‬
‫•‬
‫•‬
‫•‬
‫•‬
‫•‬
‫•‬
‫•‬
‫•‬
‫•‬
‫•‬
‫•‬
‫•‬
‫•‬
‫آموزش‬
‫نگرش‬
‫آگاهي‬
‫ايدز‬
‫سندرم نقص ايمني اكتسابي‬
‫شيوع‬
‫عوامل خطر ‪ //‬عوامل تعيين كننده‬
‫ايران‬
‫اعتياد تزريقي‪ ،‬سوء مصرف تزريقي مواد‪،‬‬
‫اعتياد‬
‫رفتار پر خطر جنس ي‪ ،‬روسپي گري‪ ،‬بيماري‬
‫هاي منتقله از راه جنس ي‪ ،‬زنان ويژه‬
‫قدرت اثر‪ ،‬نسبت شانس‪ ،‬خطر نسبي‪ ،‬نسبت‬
‫خطر‪ ،‬بزرگي اثر‬
‫مرور خالصه مقاالت‬
‫• مرور خالصه مقاالت توسط دو نفر به صورت مستقل و انتخاب‬
‫با توجه به معيارهاي انتخاب صورت گرفت‪.‬‬
‫• در نهايت از بيش از ‪ 1000‬خالصه مقاله ‪ 93‬مطالعه انتخاب‬
‫شدند‪.‬‬
‫• سپس متن كامل اين ‪ 93‬مطالعه از نظر كيفيت و معيار هاي‬
‫انتخاب مورد بررس ي قرار گرفت ‪HIV assessment‬‬
‫‪form.doc‬‬
‫‪56‬‬
Experts in a particular area frequently have
pre-formed opinions that can bias their
assessments of both the relevance and
validity of articles
57
Formulating review questions
Searching & selecting studies
Study quality assessment
Extracting data from studies
Data synthesis
58
Assessment of study quality
• Assess each study for:
– eligibility for inclusion
– study quality
– reported findings
• Ideally will involve two independent
reviewers
59
Assessment of study quality
• Validity: the degree to which the trial design,
conduct, analysis, and presentation have
minimized or avoided systematic biases.
60
'Quality' scales and checklists
61
‫• در این مطالعه صرف نظر از گزارشهای شفاهی صاحبنظران‪،‬‬
‫از ‪ 93‬مطالعه موجود‪ 53 ،‬مطالعه دارای معیارهای انتخاب‬
‫بودند‬
‫‪62‬‬
Formulating review questions
Searching & selecting studies
Study quality assessment
Extracting data from studies
Data synthesis
63
Collecting data
• data collection forms
•
•
•
•
Methods
Participants
Interventions
Outcome measures and results
64
‫جدول مستندات‬
‫•‬
‫•‬
‫•‬
‫•‬
‫•‬
‫•‬
‫•‬
‫•‬
‫•‬
‫•‬
‫گروه هدف‬
‫جمعيت تحت مطالعه (سن‪ ،‬جنس‪ ،‬شغل)‬
‫عامل يا عوامل خطر مورد بررس ي درمقاله‬
‫مكان مطالعه(كشور ‪ /‬شهر)‬
‫زمان مطالعه‬
‫نوع مطالعه (توصيفي ‪ /‬تحليلي‪)population based ،‬‬
‫مخدوش كننده هاي مورد بررس ي و نحوه ي كنترل آن ها درمطالعه‬
‫ابزار اندازهگيري متغيرهاي مورد مطالعه (پرسشنامه ‪ /‬مدارك موجود)‬
‫روايي و پايايي ابزار مورد استفاده‬
‫ميزان شيوع و قدرت اثر هر يك از عوامل تعيين كننده مذكور‬
‫‪Form Final.xls‬‬
‫‪65‬‬
‫‪Extraction‬‬
Formulating review questions
Searching & selecting studies
Study quality assessment
Extracting data from studies
Data synthesis
66
Graphical Display
• The graphical display of results from individual
studies on a common scale is a “Forest plot”.
In the forest plot each study is represented by a
black square and a horizontal line (CI:95%).The
area of the black square reflects the weight of the
study in the meta-analysis.
Forest plot is an important step, which allows a
visual examination of heterogeneity between
studies.
67
Odds Ratio
Line of no significance
less than 1
1
more than 1
68
Odds Ratio – with pooled
effect size
Best/point estimate
Confidence Interval
less than 1
1
more than 1
69
Forest Plot
Combined
.071399
.012601
var1
70
Forest Plot
Combined
-.004408
.60356
var1
71
Meta-Analysis
• when an overview incorporates a specific
statistical strategy for assembling the results
of several studies into a single estimate
72
• Systematic reviews do not have to have a
meta-analysis
• There are times when it is not appropriate
or possible.
73
‫تجزيه و تحليل نهايي‬
‫• امكان انجام متاآناليز با توجه به وجود هتروژنيتي ( نمودار و‬
‫آماري) بررس ي گرديد‪.‬‬
‫• شيوع عدم آگاهي در زير گروههاي مختلف ما بين ‪ 4/24‬تا ‪6/32‬‬
‫محاسبه گرديد‪.‬‬
‫‪75‬‬
76
A Range of Function of
Systematic Review
• 1- Identify heterogeneity in effects among
multiple studies and, where appropriate,
provide summary measure.
• 2- Increase statistical power and precision
to detect an effect
• 3- Develop, refine and test hypothesis
• 4- Reduce the subjectivity of study
comparisons by using systematic and
explicit comparison procedure
77
Remain:
• 5- Identify data gap in the knowledge base
and suggest direction for future research
• 6- Calculate sample size for future studies
78
Section 3
79
Steps of a systematic review
• Step 1: Framing question for a review
• Step 2: Identifying relevant literature
• Step 3: Assessing the quality of the
literature
• Step 4: Summarizing the evidence
• Step 5: Interpreting the finding
80
• The principal findings should be related to
the main question formulated in step1.
• Other finding should be considered
secondary.
81
Validity of the main finding
• Are the searches adequate?
• Is there a risk of publication and related
biases?
• Is the quality of the included studies high
enough?
82
Limitations of Systematic Review
• Reporting bias and the inadequate quality of
primary research are potentially serious problems
for systematic reviews.
• The dissemination of research findings is not a
random process; rather it is strongly influenced by
the nature and direction of results.
83
Type of reporting bias
• Publication bias
• The publication or
non-publication of
research findings,
depending on the
nature and direction of
the results
84
Publication bias
• 1-Arising from the researchers deciding
whether or not to submit result
• 2- Arising from the tendency of journals to
reject negative studies
• 3-sponsorship
• ….
85
Methods of preventing publication bias
• 1-Registeries
• 2-Editorial policy
86
• Time lag bias
• The rapid or delayed
publication of research
finding, depending on
the nature and
direction of the results
87
• Multiple (duplicate)
publication bias
• The multiple or
singular publication of
research finding,
depending on the
nature and direction of
the results
88
• Citation bias
• The citation or noncitation of research
finding, depending on
the nature and
direction of the results
89
• Language bias
• The publication of
research finding in a
particular language,
depending on the
nature and direction of
the results
90
• Outcome reporting
bias
• The selective reporting
outcomes but not of
others , depending on
the nature and
direction of the results
91
Funnel Plot
• Plots of the trials’ effect estimates against sample
size, may be useful to assess the validity of metaanalyses
• A symmetrical shape is expected, since greater
scatter in estimate is expected for smaller study.
• The cardinal sign of publication bias is a hole in the
middle or one side of the plot, that is an area where
we would expect to see study result but where there
are apparently none.
92
93
Thanks
94