Transcript Creating an evidence-based framework to support

Gavin Stewart
Centre for Evidence-Based Conservation
University of Birmingham, UK
What are systematic reviews and why
do we need them?
What is the problem?




Good policy needs an evidence-base
Quantity of information on a subject is large
Accessibility of information is variable (some
may be hard to find)
Information quality is variable and so is the
outcome
Quantity





1995 - 2 million articles were being published each year in
20,000 journals (medical example)
a pile of paper 500 metres high
if researchers tried to stay current by reading two articles per
day, in one year they would fall 55 centuries behind!
or, if you try to read everything of possible relevance, you would
have to read 5,500 articles per day.
Similar story for ecology? Have you read the latest issues of
Nature, Science, Biological Conservation, Journal of Applied
Ecology, Conservation Biology, TREE, Journal of
Environmental Management etc ?
Accessibility & Variability






You manage an upland NNR
There are 58 references in the management
handbook (and a further 77 you don’t know
about?)
Information from other side of the hill
quality is variable
And the outcomes are different
What do you do?
What Is Systematic Review?

Systematic review is a tool that provides
empirical answers to scientific research
questions using existing available evidence

Key features
–
–
–
Systematically locate data
Critically appraise methodology
synthesise evidence
They are not conventional Reviews

Follow a strict methodological and statistical
protocol
–
more comprehensive
–
minimising the chance of bias
–
improves transparency, repeatability and reliability
Differences Between Traditional and
Systematic Reviews
(Adapted from Cook, D. J. et. al. (1997). Ann. Intern. Med. 126: 376-380)
Feature
Traditional Review
Systematic Review
Question
Often broad in scope
Focused question
Sources &
search
Not usually specified,
potentially biased
Comprehensive sources &
explicit search strategy
Selection
Rarely specified,
potentially biased
Criterion-based selection,
uniformly applied
Appraisal
Variable
Rigorous critical appraisal,
uniformly applied
Synthesis
Often a qualitative summary
Quantitative summary* when
appropriate
Inferences
Sometimes evidence-based
Evidence-based
*A quantitative summary that includes a statistical synthesis is a metaanalysis
Stages of a Review








Identification of need for review
Formulating a question
Generating a search strategy
Study relevance
Study quality
Data extraction
Synthesis of data
Recommendations
Identification of need for review and
formulating a question and protocol
Define the
hypothesis
Set the
scope of the
question
Identify and
expand
concepts
Generating a search strategy




Multiple electronic databases and the
internet using a range of Boolean searchterms
Foreign language searches
Include grey literature to avoid publication
bias (see subsequent slides)
Search bibliographies and contact experts
Appraising study relevance
Use the question elements
 Subject e.g. arctic-alpine flora
 Intervention e.g. grazing (deer)
 Outcome e.g. Change in frequency at a site
scale (5% decline is deleterious)
If information is presented about arctic-alpines in
relation to grazing and frequency is
measured, the article is relevant irrespective
of what it says!
Appraising study quality



There is no such thing as a perfect study, all studies
have weaknesses, limitations, biases
Interpretation of the findings of a study depends on
design, conduct and analysis, as well as on the
population, interventions, and outcome measures
The researchers in a primary study did not
necessarily set out to answer your review question
What do we do with quality
assessment results?




Determine minimum quality threshold for inclusion
Explore differences in quality as an explanation for
heterogeneity in study results
To weight individual study results in relation to their
validity or the amount of information they contain
Guide interpretation and overall recommendations
Data extraction



Undertaken with synthesis in mind
Standardised methodology with a priori
rationale
Pilot data extraction and ensure repeatability
using two reviewers (good practice for data
hygiene in any case)
Synthesising evidence



Qualitative synthesis
Meta-analysis to synthesise results across
studies
Bayesian synthesis of disparate data types
particularly using experience as a prior
Needs-led research
Review &
Dissemination
Unit
Decision-makers
Needs-Led
Research
Research
Community
Funding
Bodies