Transcript Supporting Systematic Reviews for STEM researchers & educators
Overview
0
What are systematic reviews?
0
Role of librarians in systematic reviews
0
Center for Systematic Reviews at Texas A&M University Libraries
0
STEM Systematic Reviews
Dr. Ben Goldacre, award-winning Bad Science columnist and medical doctor, forward in Testing
Treatments
What are systematic reviews?
A systematic review “attempts
0 0
to
collate all
empirical evidence that fits
pre-specified
eligibility criteria in order to answer a question.
specific
research
0 0
It uses
explicit
, systematic methods that are selected with a view to
minimizing bias
, thus providing more and decisions made.” 1
reliable findings
from which conclusions can be drawn 1 Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from www.cochrane-handbook.org
., Section 1.2.2
Comparing review types
Narrative Reviews
o Depend on authors’ inclination (bias) o o Author selects any criteria Search any databases o o Methods not usually specified Can’t replicate review
Systematic Reviews
o Scientific approach to a review article o o Criteria determined at outset Comprehensive search Explicit methods of appraisal and synthesis
Narrative Review
Subjective
Systematic Review Meta analysis
Objective
citation
Steps of systematic review methods The process should be:
0
Transparent: record & report all methods
0
Follow standards and evidence based practices
0
Minimize bias 1. Define 2. Search 3. Select 4. Code 5. Synthesize
Specific types of reviews
Types vary by
0
Specificity of the question
0
Comprehensiveness of search and inclusion
0
Depth assessment and coding
0
Type of synthesis
0
Rapid review: narrow, quick search and assessment of very specific question
0
Scoping review: assessment of potential size/scope of available literature
0
Integrative review: includes qualitative/ quantitative/ theory
0
Meta-analysis: quantitative synthesis
0
Umbrella review: review of systematic reviews Grant, M. J., & Booth, A. (2009). A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26(2), 91-108. doi:10.1111/j.1471-1842.2009.00848.x
What makes this search different??
0
It will be evaluated by many
0
All articles the search retrieved must be labeled
0
Search one database at a time
0
Sensitivity is more important than specificity
0
Bias during the search is a one of the biggest threats to the review
Publication bias
When due to nature & direction of research findings a situation occurs making it more difficult to locate studies
Publication bias
• Publication or non publication of findings
Time-lag bias
• Rapid/delayed publication of findings
Language bias
• More likely to be published in English
Citation bias
• Citation/non-citation of research findings
Selective outcome reporting bias
• Selective reporting of some outcomes but not others
Location bias
• journals with different ease of access/levels of indexing in standard databases
Multiple (duplicate) publications
• Multiple/single publications
Database bias
• Some databases are more likely to index certain languages/journals Rothstein, D. H. R., Sutton, D. A. J., & Borenstein, D. M. (2006). Publication Bias in Meta-Analysis. Publication Bias in Meta-Analysis(pp. 1-7) doi:10.1002/0470870168.ch1
PRISMA
Preferred Review Items for Systematic reviews and Meta Analysis flowchart Records retrieved from database searching (n=3000 ) Total retrieved records (n=4500 ) Records screened by title (n=3500 ) Records retrieved from additional sources (n=1500 ) Duplicates removed (n= 2000 ) Records excluded (n=2500 ) Records screened by abstract (n=1000 ) Records screened by full text (n=300) Records included in qualitative synthesis (n=500 ) Records included in quantitative synthesis (n= 25) Records excluded (n= 700) 300 wrong population 400 wrong intervention Records excluded (n=250 ) 100 wrong population 100 wrong intervention 50 wrong education issue Records excluded (n= 25 ) 12 wrong outcome 13 lack of follow up See full PRIMSA statement at http://www.prisma-statement.org
Role of Librarian in SRs
Project leader Document supplier Project manager Critical appraiser Report writer Literature searcher Data extractor Disseminator Reference manager Data synthesizer
McGowan, J., & Sampson, M. (2005). Systematic reviews need systematic searchers. Journal of the Medical Library Association : JMLA, 93(1), 74-80.
Team- roles in steps
Step
1) Define 2) Search 3) Selection 4) Assess
Description
Develop in PICO form (if relevant); plan project management High recall search for published and unpublished material; minimize bias Primary, secondary
Team players
Subject experts; librarian Librarian; subject experts Subject experts; librarian Assess quality of studies; Code info from studies Subject experts; statistician; librarian to manage data 5) Synthesis Reported according to standard; give detail for replication Subject expert; librarian Adapted from: McGowan, J., & Sampson, M. (2005). Systematic reviews need systematic searchers. Journal of the Medical Library Association : JMLA, 93(1), 74-80.
Standards
Institute of Medicine of the National Academies
(2011) has two standards specifically referring to librarians:
0 0
3.1.1
Work with a librarian or other information specialist trained in performing systematic reviews to plan the search strategy
3.1.3
Use an independent librarian or other information specialist to peer review the search strategy 27 items in the PRISMA check list…
0
7. “information sources,” “describe all information sources…in the search and date last searched.”
0
8. “Search,” “present full electronic search strategy for at least one database, including any limits used, such that it could be repeated.” Standards for Systematic Reviews from Institute of Medicine of the National Academies http://www.iom.edu/~/media/Files/Report%20Files/2011/Finding-What-Works-in-Health-Care-Standards-for-Systematic Reviews/Standards%20for%20Systematic%20Review%202010%20Insert.pdf
PRIMSA statement at http://www.prisma-statement.org
Authorship (Vancouver statement)
1.
2.
3.
Authorship credit should be based on: substantial contributions to conception and design, or acquisition of data, or analysis and interpretation of data drafting the article or revising it critically for important intellectual content final approval of the version to be published.
Authors should meet conditions 1, 2, and 3 International Committee of Medical Journal Editors. Uniform requirements for manuscripts submitted to biomedical journals. JAMA. 1997. Mar 19; 277(11):927.–34.
Center for Systematic Reviews
Who can use the service?
all Texas A&M University students, faculty, and staff
What is offered?
0
Consultations
0
Class presentations
0
Webinars
0
Workshops Majority of requests from:
0
College of Medicine
0
School of Public Health
0
College of Veterinary Medicine
0 0
College of Education
0
Department of Health & Kinesiology
0
Educational Psychology College of Agriculture
Consultations
0 0
Initial consultations Establish that question is sound Set up criteria Research plan Locate relevant reviews Discuss type of consultation Follow up consultations Continue search Demonstrate steps/ provide examples
# of consultations
300 250 200 150 100 50 0 30 60 100 174 102 161 260 2007 2008 2009 2010 2011 2012 2013
Evaluating the service
Annual survey:
0
Researcher/review demographics
0
State of review/ level of understanding of SRs before & after consultations
0
Consultations
0
Tools offered by services
0
Future services
0
Comments
Levels of consultation
as consultant:
• guidance with systematic review steps • databases need to be searched • search techniques and methods • Few meetings
as contributor:
• Consultant plus: • Multiple meetings to walk through the steps • Training on information management
as author:
• contributor plus: • management of searches • write methods section • creating tables about the process of the review • Screening and/or coding as appropriate
Reviews in STEM education
0
Effectiveness of educational interventions
0
Test theories
0
Identify gaps in current research
0
Provide organized evidence for decision making
0
Help in planning new interventions
SR Organizations in education
Campbell Collaboration
• education, criminal justice, social work • International • Provides guidelines/ standards • Provides forum to publish reviews
What Works Clearinghouse
• Institute of Education Sciences • Procedures and Standards Handbook • Guidance for practical use of reviews
Evidence for Policy and Practice Information and Co-ordinating Centre
• International • Education and Health Interventions • Provides methods • Databases of reviews
Overview of SRs in Eng. Education
0 0 0
Introduction
0
Systematic reviews in other disciplines
0
Purposes and goals of reviews Steps in conducting the review 1.
Deciding to do a systematic review 2.
3.
4.
5.
6.
7.
Identifying scope and research questions Defining inclusion criteria Finding and cataloging sources Critique and appraisal Synthesis Limitations, validity, and reliability concerns Systematic identification of reviews Borrego, M., Froyd, J., and Foster, M. (2014) Systematic Literature Reviews in Engineering Education and Other Developing Interdisciplinary Fields. Journal of Engineering Education 103(1): 45-76. DOI: 10.1002/jee.20038
Number of Reviews published
Critical Analysis of SRs
0
We started with 473 articles, resulting in 49 included reviews: 17 MAs, 32 SRs
0
Used Campbell Collaboration checklist to evaluate the article
0
Found that authors’ of reviews did not…
0 0 0 0 0
adequately describe inclusion criteria conduct appropriate comprehensive search independently screen or assess articles include flowchart of reasons articles were excluded appropriately address reviews limitations Campbell Collaboration. (2012). Education coordinating group systematic review checklist. from http://www.campbellcollaboration.org/artman2/uploads/1/ECG_Protocol-Review_Checklist_1108.pdf
Conclusion
0
Systematic reviews are becoming more common in disciplines outside of medicine
0
To conduct SR searches appropriately requires some training
0
SRs provide opportunity for collaboration with researchers