Transcript Slide 1

5.0
The Eight Steps of a High Quality
Systematic Review
1. Formulate Review Questions
2. Define inclusion and exclusion criteria:
3. Locate Studies
4. Select Studies
5. Assess Study Quality
6. Extract Data
7. Analyze and present results
8. Select Studies
Information Retrieval for HighQuality Systematic Reviews:
The Basics
Finding Evidence:
A Conceptual Framework
Cochrane Collaboration
Hand Searches
Published Literature
Electronic Searches
RCTs
CRTs
Electronic
C2-SPECTR
Registers
C2-PROT
Systematic Reviews
EPPI Center
Invisible College
C2-RIPE
Campbell Collaboration
Grey Literature
What Works Clearinghouse
Information Retrieval:
A Continuous Process
– Preliminary Searches
• Supports beginning steps: Definition of key concepts &
research question
• Use of standard reference tools and broad searches for
review articles and key primary studies
– Main Searches
• Identification of primary studies through searches of
online databases, printed indices, Internet, branching,
hand-searches
• Most difficult given a number of challenges
– Final Searches
• Occurs towards the end of the Review Process
• Refine search terms and update original searches
Main Searches: Decisions
• Scope of search: Which fields should be searched (e.g.
beyond Education)?
• Availability of indexing tools: Which tools do we have
access to at our institution?
• Format of indexing tools: What format are they in (e.g.
online, print, web-based)?
• Date: How far back does the indexing go for each tool?
• Language: What is the language of the material that is
indexed? How can we locate non-English material?
• Unpublished work: How can we access dissertations,
reports, & other grey literature?
Example: Parent Involvement
– Fields searched: Education, Psychology, Sociology, Social
Sciences, Economics, Population, Family
– Geographic scope of indexing tools: Mostly North
American, some UK
– Format of indexing tools: Online/web-based databases
– Coverage Date of Indices: Not noted
– Language: Foreign language included (when possible)
– Unpublished work: Search of dissertations, web sites,
policy documents
Main Searches: Decisions
– Preparation of Search Strategies
•
•
•
•
What are the key concepts to be searched?
How are these represented in each field?
What are their related terms?
How are these key concepts represented in the controlled
vocabulary within each indexing tool?
– Construction of the Search Statements
• What terms should be searched as descriptors or as “free
text”?
• What boolean operators should be used?
• What limiting features are available to target primary studies
only (e.g. use of Document Type codes)?
• What time period should be searched?
– Note. These decisions need to be made for each indexing tool
used.
Example: Parent Involvement
– Descriptors: Parent Involvement See: Parent participation
– Related descriptors: Family involvement, Parent-school relationship,
Parent-student relationship, Parent role, Parents as teachers
– Related keywords: parent* involvement, parent* effectiveness, parent*
support, famil* support, famil* participation
– Descriptors: Academic Performance See: Academic achievement
– Related descriptors: Science achievement, Reading achievement,
Literacy Achievement, Writing achievement, Achievement gains
– Descriptors: Elementary School Children See: Elementary school
students
– Related descriptors: Elementary education, Primary education,
Kindergarten
Example: Parent Involvement
•
Combining Keywords/Descriptors using Boolean & Proximity
operators:
1. (Parent participation OR Family involvement OR Parent-school
relationship OR Parent-student relationship OR Parent role OR
Parents as teachers) in DE
2. Parent* NEAR2 involvement or Parent* NEAR2 effectiveness OR
Parent* NEAR2 support OR Famil* NEAR2 support or Family
NEAR2 participation
3. #1 OR #2
4. (Academic achievement OR Science achievement OR Reading
achievement OR Literacy Achievement OR Achievement gains) in
DE
5. (Elementary school students OR elementary education OR primary
education OR kindergarten) in DE
6. #3 AND #4 AND #5
Limiting commands: Research reports (DTC=143)
Main Searches: Challenges
Precision vs. Recall (White, 1994)
– Precision: ratio of docs retrieved and judged relevant to all those
actually retrieved
– Recall: ratio of relevant docs retrieved to all those in a collection
that should be retrieved
– Regular searches have an inverse relationship - High precision vs
low recall (a small set of highly relevant records)
– Reviewers want high precision and should want high recall (a
large, comprehensive set of relevant records with minimal bias)
– Achieving this balance is the challenge for reviews!
• Use of incorrect terminology within a field may miss relevant
records (low recall)
• Use of of incorrect or redundant terminology may yield
unwanted records (low precision)
Another Example:
Distance Education
Compare the retrieval of 1 record vs 510!
1. elearn (1 record)
2. e near1 learn* (459 records)
AU: Holstrom,-Lisa
TI: Eliminating Barriers for All E-Learners.
SO: Educational Technology. v43 n6 p61-62 Nov-Dec 2003
DE: *Access-to-Education; *Distance-Education
DE: Feedback-; Higher-Education; Literacy-; Low-IncomeGroups; Skill-Development
IDM: *Barriers-to-Participation; *Electronic-Learning
IDR: Educational-Media-Use; University-of-Cincinnati-OH
3. (electronic learning) in DEM,DER,IDM,IDR (104 records)
4. ((electronic learning) in DEM,DER,IDM,IDR) or (e near1
learn*) or (elearn) (510 records)
Main Searches: Decisions
• Other Retrieval Strategies
– Branching:
• Scanning the reference section of review articles
– Hand searches:
• Scanning the Table of Contents of key journals and
conference proceedings
– Personal contacts:
• Contacting key researchers in the field
– Consulting C2 resources
Summary
“Shoestring-budget information retrieval is likely to introduce bias,
and should be avoided.” (IR Policy Brief, 2004)
• Importance of information retrieval process
– Not a “one-shot”deal
– Requires expertise in the planning and implementation
of searches
– Library personnel are important members of the team
• Use of bibliographic management software
– Reference Manager, EndNotes, & Biblioscape
• Ability to replicate review
– Documentation of entire process, including search
strategies used for each database, decisions taken, etc.