Transcript BCURE Evidence-Informed Decision-Making Capacity Building Workshop 1st and 2nd June 2015 Pretoria, South Africa What Are Systematic Reviews, And Why Do We Need Them? Philip.

BCURE Evidence-Informed Decision-Making Capacity Building Workshop
1st and 2nd June 2015
Pretoria, South Africa
What Are Systematic Reviews,
And Why Do We Need Them?
Philip Davies
International Initiative for Impact Evaluation [3ie]
Philip Davies
www.3ieimpact.org
Group Exercise 1:
How do you know ahead of time if a policy
intervention works?
Philip Davies
www.3ieimpact.org
Why Do We Need Systematic Reviews?
Sheer amount of available evidence
“Beyond the capacity
of the human mind”
Philip Davies
www.3ieimpact.org
What Are Systematic Reviews?
• A way of establishing
the overall balance of
the global empirical
evidence on a topic or
policy
• And separating higher quality from lower quality
evidence
• A way of identifying what is generalisable and
what is context specific
Philip Davies
www.3ieimpact.org
Why Do We Need Systematic Reviews?
 Single studies can:
 Misrepresent the balance of research
evidence
 Illuminate only one part of a policy issue
 Be sample-specific, time-specific, contextspecific
 Often be of poor quality
Consequently, give a
biased view of the
overall evidence
Philip Davies
www.3ieimpact.org
Types of Systematic Review
• Statistical Meta-Analyses
• Narrative Systematic Reviews
• Qualitative Systematic Reviews
• Rapid Evidence Assessments
• Evidence Maps and Gap Maps
Philip Davies
www.3ieimpact.org
What Makes a Review Systematic?
• Systematic searching for studies
• Systematic critical appraisal of
identified studies – separating the
wheat from the chaff
• Systematic and transparent
inclusion/exclusion of studies for
final review
• Systematic and transparent
extraction of data
• Systematic statistical testing and
analysis
• Systematic reporting of findings
Philip Davies
www.3ieimpact.org
Searching for Evidence
Philip Davies
www.3ieimpact.org
Key Databases for Finding Evidence
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
ABI/INFORM Global
Africawide Information
Applied Social Sciences Index and Abstracts (ASSIA)
Business Source Complete
CAB Abstracts
EconLit
ERIC (Education Resources Information Center)
International Bibliography of the Social Sciences (IBSS)
Medline/PubMed
PAIS International
PolicyFile
PsycINFO
Sociofile (Sociological Abstracts)
Social Science Citation Index (SSCI)
World Wide Political Science Abstracts
Philip Davies
www.3ieimpact.org
Sifting of Studies to Determine Eligibility
Filter evidence based on PICOS
- Title and Abstract stage
- Inclusion decisions by 1 person
- Over-inclusive (‘sensitivity’)
- Full Text Screening Stage
- Decisions about inclusion and
exclusion by explicit and
transparent criteria
- Inclusion decisions by 2 people
– 3rd person acts as arbiter
where necessary
- Rigorous evidence remains limited
for many interventions (problem
of ‘empty reviews’ if PICOS set
too narrowly)
Philip Davies
www.3ieimpact.org
Sifting and Developing a Flow Chart
FFS Review
28,525 titles screened
Title and Abstract Sifting
1453 abstracts
screened
Full Text Sifting
369 full text obtained
134 papers included:
93 included FFS
evaluations studies
71 individual FFS
programmes
Philip Davies
www.3ieimpact.org
126 no access
235 excluded:
128 on relevance
58 on design (no
comparison group)
49 non-FFS impact
evaluations
Critical Appraisal – Key Themes
•Critical appraisal is an essential part of SR
•Separates out high quality from low quality studies
•Uses explicit and transparent criteria for including and
excluding primary studies for review
•These criteria vary according to the methods of
primary studies
•Some methodological areas have more developed
criteria than others (e.g. experimental studies vs. QEs
vs. qualitative)
Philip Davies
www.3ieimpact.org
Critical Appraisal of Studies
Scope: Is this study within the scope of the review?
• The Populations/subgroups/areas studied
• The Interventions studied
• The Comparators or contexts used
• The Outcomes used/measured
• The Study designs
Methodological Quality: Does this study meet
acceptable standards of design and execution?
• The reporting of the primary studies
•The internal, external & construct validity of the
studies
• The legitimacy of the conclusions drawn
Philip Davies
www.3ieimpact.org
Philip Davies
www.3ieimpact.org
Critical Appraisal
Sorts of Questions to Ask: Experimental Studies
Internal Validity of Primary Studies
• Was the sample size adequate/sufficient power?
• Are the experimental and control groups truly comparable at
baseline?
• Was randomisation allocation & concealment carried out
properly?
• Is there other evidence of risk of bias, e.g.:
• Selection bias?
• Performance bias?
• Attrition bias?
Philip Davies
www.3ieimpact.org
Critical Appraisal
Sorts of Questions to Ask: Experimental Studies
Adequacy of Reporting of Articles
• Have the necessary details of the study (design, methods,
participants, interventions, outcomes) been reported?
• Have appropriate statistics been used in the analysis of data?
• Have the appropriate statistics been fully reported?
• Has the attrition rate been presented and explained?
• Has an intention to treat analysis been undertaken?
• What missing data need to be gathered/followed-up?
• Do the findings and analysis support the author’s conclusions?
Philip Davies
www.3ieimpact.org
Sorts of Questions to Ask:
External Validity of studies
External Validity: Does the study have relevance to the ‘real world’
in which you are working?
•Is the sample in the study/report similar to the population you are
dealing with?
•Are the outcomes in the study/report the same outcomes that are
interest to your work?
•Are there any contextual factors mentioned in the study/report that
would limit its relevance to your work?
•Does the study examine efficacy (ideal conditions) or effectiveness
(real world) of intervention?
Philip Davies
www.3ieimpact.org
Philip Davies
www.3ieimpact.org
Critical Appraisal
Quality Criteria - Qualitative Studies
• Was there a clear statement of the aims of the research?
• Is a qualitative methodology appropriate?
• Was the research design appropriate to address the aims of the
research?
• Was the recruitment strategy appropriate here to the aims of the
research?
• Were the data collected in a way that addressed the research issue?
Source: Critical Appraisal Skills Programme (CASP). 10 questions to help you
make sense of qualitative research.
Philip Davies
www.3ieimpact.org
Critical Appraisal
Quality Criteria - Qualitative Studies (cont)
• Has the relationship between researcher and participants been
adequately considered?
• Have ethical issues been taken into consideration?
• Was the data analysis sufficiently rigorous?
• Is there a clear statement of findings?
• How valuable is the research?
Source: Critical Appraisal Skills Programme (CASP). 10 questions to help you
make sense of qualitative research.
Philip Davies
www.3ieimpact.org
Data Extraction Form (Quantitative)
Extract Information on:
• Each study’s eligibility for the review
• Each study’s methods and research design
• The likely risk of bias
• The study’s participants, interventions, outcomes,
results and conclusions
• Tests of significance used
• The average effect size (standard mean difference,
odds ratio)
• Data on variance
quartiles)
(standard
• Confidence intervals and p values
Philip Davies
www.3ieimpact.org
deviations,
range,
Data
Oxford Evidentia
Philip Davies
Extraction Form
www.3ieimpact.org
Extract from a Table of Included Studies (FFS Review)
Philip Davies
www.3ieimpact.org
Extract from a Table of Excluded Studies (FFS Review)
Philip Davies
www.3ieimpact.org
Statistical Meta-Analytical Reviews
• Involves data-pooling and statistical synthesis of
independent studies
• And aggregating/cumulating samples and findings
• Seeks to measure and control bias
• Requires included studies to
be as similar (homogeneous)
as possible
Photo © Amos Gumulira
Philip Davies
www.3ieimpact.org
Requirements for Homogeneity
Similarity of:
• Population (or sub-groups)
• Intervention
• Comparator
• Outcome
Overlapping confidence intervals
Philip Davies
www.3ieimpact.org
Statistical Meta-Analytical Reviews
Source: David B. Wilson, 2006, A systematic review of drug court effects on recidivism
Philip Davies
www.3ieimpact.org
Narrative Systematic Reviews
• Provide a narrative account of what the
evidence tells us
• Provide descriptive and inferential statistics
• But on each included study individually, not
aggregatively
• Provide a summary analysis of what the
evidence suggests
• Provide the ‘Signal’ and ‘Noise’ of evidence
Philip Davies
www.3ieimpact.org
Qualitative Systematic Reviews
• Synthesise qualitative and ethnographic evidence
• In-depth interviews, focus groups, observational
studies, documentary analysis, case studies
• Seek common themes, concepts and principles
across different studies
• Detailed attention to
context/contextual specificity
• And stakeholders’ views
• Do not seek generalisations
Photo © Albert Gonzalez Farran - UNAMID
Philip Davies
www.3ieimpact.org
Some Key Sources of Sythesised Evidence
• 3ie Impact Evaluations Database
(http://www.3ieimpact.org/en/evidence/impact-evaluations/)
• 3ie Systematic Reviews Database
• (http://www.3ieimpact.org/en/evidence/systematic-reviews/)
(http://www.3ieimpact.org/en/evidence/systematic-reviews/)
• Best Evidence Encyclopedia (http://www.bestevidence.org/)
• Cochrane Collaboration (www.cochrane.org)
• Campbell Collaboration (www.campbellcollaboration.org)
• Coalition for Evidence-Based Policy
(http://coexgov.securesites.net/index.php?keyword=a432fbc34d71c7)
• National Institute for Health and Clinical Excellence (http: www.nice.org.uk/)
• NHS Evidence (http://www.evidence.nhs.uk/)
• National Guidelines Clearinghouse (USA) (www.guidelines.gov)
• Prospero: International Prospective Register of Systematic Reviews
(http://www.crd.york.ac.uk/prospero/
• Social Care Institute for Excellence (http://www.scie.org.uk/)
• Social Programs That Work (http://evidencebasedprograms.org/)
Philip Davies
www.3ieimpact.org
Some Key Sources of Sythesised Evidence
Philip Davies
www.3ieimpact.org
Thank you
Philip Davies
Email: [email protected]
+44 (0)207 958 8350
Visit www.3ieimpact.org
Philip Davies
www.3ieimpact.org