Transcript Understanding Systematic Reviews

http://bahrain.cochrane.org
http://www.rt.ae
The Bahrain Branch of the UK Cochrane Centre
In Collaboration with
Reyada Training & Management Consultancy, Dubai-UAE
Cochrane Collaboration and Systematic Review
Workshop, 20-21 February 2007, Dubai - UAE
W02
Dr. Zbys Fedorowicz, Dr. Dunia Al Hashimi, Dr. Ahmed Al Asfoor
Understanding Systematic
Reviews
Reviews and Meta-Analyses



Review: An article that summaries a
number of different primary studies and may
draw conclusions on the effectiveness of a
particular intervention.
A review may or may not be systematic.
Meta - analysis by definition is a process
of synthesizing research results by various
statistical methods to retrieve, select, and
combine results from previous separate but
related studies.
Meta - analyses are reviews that use
systematic methods to summarizes the
results of previous studies
Types of Reviews


Narrative (Traditional)
Systematic/Overview
A Systematic Review:
A review of the evidence on a clearly
formulated question that uses
systematic and explicit methods to
identify, select and critical appraise
relevant primary research, and extract
and analyze data from the studies that
are included in the review.
Statistical Methods (meta-analysis) may
or may not be used.
Differs from traditional reviews in
that they use a replicable,
scientific and transparent approach
to minimise bias.
Feature
Narrative Review
Systematic Review
Questions
Often broad in Scope
Often a focused
question
Sources and Search
Not usually specified,
potentially biased
Comprehensive sources
and explict search
strategy
Selection
Not usually specified,
potentially biased
Criterion-based
selection, uniformly
applied
Appraisal
Variable
Rigorous critical
appraisal
Synthesis
Often a qualitative
summary
Quantitative summary*
Inferences
Sometimes evidencebased
Usually evidence-based
* Quantitative summary that includes a statistical analysis is a meta-analysis
Literature review, systematic review, metaanalysis
Literature
review
Systematic
review
Metaanalysis
Anatomy of a Systematic Review
1.
2.
3.
4.
5.
6.
7.
8.
The Question
Survey the review literature
Further define question
Design the Protocol
Search and Study Selection
Data Extraction
Analysis
Reporting
1. The Question

Systematic reviews usually have a focused:




Population/problem, setting(s).
Intervention (treatment, diagnosis, technology,
etc.)
Outcome (patient, economic, usability, etc.)
(EBM uses PICO as a guide to question
formation)
2. Literature Review



Check to see if a review has already
been done on your question.
This process can help re-define and
focus your question.
You can gain an understanding of the
literature in the subject area.
3. Further Refine Your
Question
4. Develop the Protocol

The protocol is your research plan and
should include: 



The background
The problem
The methodology
Protocols should be developed
prospectively in limit bias.
Protocols should include:



Specific questions to be answered
Strategies and sources for finding information
Inclusion/Exclusion criteria including



Population, settings, problem
Exposure, intervention
Controls
Outcome specified : definitions, what is being measured,
how it’s being measured, where and by whom.
Cont..

– Type(s) of study design






Types of studies to include –experimental,
observational, qualitative, etc.
Defined outcomes measuring what, how, and by
whom
Study selection
Data Extraction strategy
Analysis strategy
Presentation of results
5. Finding the information




Start with the question
Decide on sources to search
Develop search strategies
Keep records and use a bibliographic
software program to manage the
information
Search Strategy




Is an iterative process developed through
input from the review team.
Generates the lists of published (and
unpublished) research studies
Affects the validity of the results of the review
The protocol guides the search by stating
sources to search, types of information to
include (published & unpublished),
languages, study design, key concepts.
Search strategy cont’d


The thoroughness of the searching is one of
the defining differences between systematic
and traditional reviews
Must decide whether to go for a
Sensitive search strategy – increases retrieval and
irrelevant results
OR
 Specific search strategy – increases precision but
may miss relevant material

Sources of Information









Databases of primary research articles
Hand searching core subject journals
Checking cited references
Conference proceedings
Contacting researchers and experts
Dissertations
Grey Literature
Web of Science – citation tracking
Internet
Systematic Searching
Why so thorough?


Missing studies can have significant
effect on the results of the review
Publication Bias

The tendency for studies with positive
results to get published and studies
showing no difference or negative results
not getting published.
Thorough searching




much research is never published
not all research is published in journals
not all research published in journals is
indexed on major databases
not all research indexed on databases
can be easily retrieved
Sources of bias in trials
Target population
Sources of bias
Allocation
Intervention
group

Exposed to
intervention

Follow-up

Outcomes
Control
group

Not exposed to
intervention

Follow-up

Outcomes
Selection bias
Performance bias
Attrition bias
Detection bias
‘Bias’

publication bias


language bias


Positive results
Positive results more likely to be published
in English language publications
geographical coverage bias of journals
and databases
Why do we search widely?

Savoie et al estimated that 29.2% of
items in their review were uncovered
by:





searching the web
handsearching
scanning reference lists
personal communication
searching specialised databases and web
sites.
Publication bias


Song et al. Publication and related
biases.
Methods to dealing with publication and
related biases in systematic reviews:




literature searching
locating unpublished studies
assessment of the risk of publication and
related biases
detecting publication bias
Why search widely….

Allen & Hanburys found that:


only 51% of the clinical trials relating to
their respiratory products were published in
journals indexed by MEDLINE, EMBASE or
CINAHL (46% are in grey lit)
Wallace et al.

11 of 65 trials in end stage renal disease
reviews were found by searching beyond
major databases.
Why search widely....


Long lead times before publication:
 publication gaps after conference
presentation
 indexing publication lag before recorded in
databases
Cheng et al, 1998
 8.1% of a set of conference papers
achieved publication within 12 months
 29% within 2 years and 40% within 5
years
Delay and non publication



Non-publication
Cheng et al, 1998
 only 32% of abstracts presented at chronic
fatigue conferences were subsequently
published in full
Petticrew et al, 1999
 50.6 % of oral paper presented at the
Society for Social Medicine 1996 achieved
publication
Key factors






thorough searches
searching beyond published articles
citation searching
Internet searching
hand searching
searching for ongoing research
Sources of Information

REVIEWS:




Cochrane Database of Systematic Reviews (CDSR)
DARE
Cochrane Central Database of Controlled Trials (CCDCT)
other sources such as Health TechnologyAssessment
sources



HTA ( http://www.york.ac.uk/inst/crd/htahp.htm) or
CCOHTA – Canadian Coordinating Office for Health
Technology Assessment http://www.ccohta.ca/ (has a
number of publications)
International Network of Agencies for Health Technology
Assessment
http://www.inahta.org/inahta_web/index.asp
INDEXES:






Medline
EMBASE
CINAHL
PsychInfo
ERIC
Computer Science (INSPEC, Compendex,
etc)
OTHER:







Reference Lists & Citations
Web of Science cited reference searches
Hand searching key journals
Researchers in the subject area
Conference Proceedings (PapersFirst, Proceedings
First)
Dissertations (Dissertation Abstracts International)
Internet
Grey Literature
Databases of grey literature
 Grey Literature Report
http://www.nyam.org/library/grey.shtml




SIGLE System for Information on Grey Literature in
Europe contains citations to reports and nonconventional (grey) literature issued informally
throughout EC member countries.
NetPrints -A repository of non-peer reviewed original
research http://clinmed.netprints.org/home.dtl
Organizations ….
Clinical Trials
6. Information from included studies
Bibliographic details
Study characteristics
- design/methods
- participants
- interventions
- outcome measures
Study results
- means, SD and/or SE
- no. of events and N
Make sense of a review
A. Are the results of the review valid?
B. What are the results?
C. Will the results help locally?
A. Are the results of a review valid?
1. Did the review address a clearly focused issue?
2. Did the authors look for the appropriate sort of papers?
3. Do you think the important, relevant studies were
included?
4. Did the review’s authors do enough to assess the
quality of the included studies?
5. If the results of the review have been combined, was it
reasonable to do so?
B. What are the results?
6. What is the overall result of the review?
‘Bottom line’ results; NNT, OR, RR, RD?
7. How precise are the results?
Confidence interval, p values
C. Will the results help locally?
8. Can the results be applied to the local
population?
9. Were all important outcomes considered?
10. Are the benefits worth the harms and costs?
No Questions?