Transcript Collaboration Web Development: The Next Phase

How to do Cochrane systematic
reviews: Introductory workshop
Dr Jeremy Grimshaw
Canadian Cochrane Network and Centre
University of Ottawa
1
Outline of the course
Introductions
 Introduction to systematic reviews
 Introduction to The Cochrane
Collaboration
 Steps in doing a Cochrane review
Coffee Break
 Advanced methods
 Appraising the risk of bias
- Locating and selecting studies for
your systematic review

© 2005 The Canadian Cochrane Network and Centre
2
Introductions




Name
Background
Previous experience of systematic
reviews
What you want to get out of the
training?
© 2005 The Canadian Cochrane Network and Centre
3
Session 1: Outline





Introduction to systematic reviews
Introduction to The Cochrane
Collaboration
Rationale for developing a protocol
Format of a protocol
Resources to support reviewers
© 2005 The Canadian Cochrane Network and Centre
4
Problems of information
management in healthcare




Over 20,000 health journals published
per year
Published research of variable quality
and relevance
Research users (consumers, health care
professionals and policy makers) often
poorly trained in critical appraisal skills
Average time professionals have
available to read = <1 hour/week
© 2005 The Canadian Cochrane Network and Centre
5
Systematic reviews

Systematic reviews use rigorous
scientific methods to identify, assess
and synthesise the worldwide available
evidence
© 2005 The Canadian Cochrane Network and Centre
6
Systematic reviews

A review of a clearly formulated
question that uses systematic and
explicit methods to identify, select, and
critically appraise relevant research,
and to collect and analyse data from the
studies that are included in the review.
Cochrane Collaboration (2005) Glossary of Terms in The Cochrane Collaboration
© 2005 The Canadian Cochrane Network and Centre
7
Meta-analysis

The use of statistical techniques in
a systematic review to integrate
the results of included studies.
Cochrane Collaboration (2005) Glossary of Terms in The Cochrane Collaboration
© 2005 The Canadian Cochrane Network and Centre
8
Meta-analysis
© 2005 The Canadian Cochrane Network and Centre
9
Systematic reviews
Systematic reviews involve:
 stating the objectives of the research
 defining eligibility criteria for studies to be
included
 identifying (all) potentially eligible studies
 applying eligibility criteria
 assembling the most complete dataset
feasible
 analysing this dataset, using statistical
synthesis and sensitivity analyses, if
appropriate and possible
 preparing a structured report of the research.
© 2005 The Canadian Cochrane Network and Centre
10
Systematic reviews





Reduce bias
Reduce random error
Explore variability
Provide reliable basis for making
decisions
Inform and influence future research
© 2005 The Canadian Cochrane Network and Centre
11
Traditional vs systematic
reviews
Traditional
 lack rigor
 methodology not
transparent
 different
reviewers reach
different
conclusions
 become out of
date
Systematic
 scientific rigor to
minimise bias
 explicit and
reproducible
methodology
 regularly updated
(Cochrane)
© 2005 The Canadian Cochrane Network and Centre
12
The Cochrane Collaboration
“It is surely a great
criticism of our
profession that we
have not organised a
critical summary, by
specialty or
subspecialty,
adapted periodically,
of all relevant
randomized
controlled trials”.
Archie Cochrane, 1979

© 2005 The Canadian Cochrane Network and Centre
13
The Prototype - Effective Care
in Pregnancy and Childbirth
© 2005 The Canadian Cochrane Network and Centre
14
The Cochrane Collaboration
The Cochrane Collaboration is an
international organization that
aims to help people make wellinformed decisions about
healthcare by preparing,
maintaining, and promoting the
accessibility of systematic reviews
of the effects of healthcare
interventions.
© 2005 The Canadian Cochrane Network and Centre
15
Structure of The Cochrane
Collaboration
Centres
Consumer
Network
Review
Groups
Steering
Group
Fields
Methods
Groups
© 2005 The Canadian Cochrane Network and Centre
16
Cochrane Centres

Promote awareness and support of The
Cochrane Collaboration on a geographical
basis
 support individual review authors
 support review groups in that area
 provide a unique function e.g. software
development, consumer support
 link to government and other agencies
 not a production house for reviews
© 2005 The Canadian Cochrane Network and Centre
17
Canadian Cochrane Centre staff
© 2005 The Canadian Cochrane Network and Centre
18
Cochrane Review Groups





International and multidisciplinary
Produce reviews
Focused around health problems
51, covering most of health care
Editorial base facilitates process
© 2005 The Canadian Cochrane Network and Centre
19
Cochrane Review Groups


Worldwide there are over 11,000 review
authors working with 51 registered
review groups
5 review groups have their editorial
base in Canada
 Back Group
 Effective Practice and Organisation of
Care Group (EPOC)
 Hypertension Group
 Inflammatory Bowel Disease and
Functional Bowel Disorders Group
 Musculoskeletal Group
© 2005 The Canadian Cochrane Network and Centre
20
Cochrane Fields


Represents a population, group, or type
of care that overlaps multiple review
group areas
 E.g., primary care, health
promotion and public health, health
care of older people
 Child Health Field and the Health
Equity Field are located in Canada
Ensures that their priorities are
reflected in the work of review groups
© 2005 The Canadian Cochrane Network and Centre
21
Cochrane Methods Groups



Develops methodology and advise the
Collaboration on how the validity and
precision of systematic reviews can be
improved
Examples: Statistical Methods, NonRandomized Studies, Information
Retrieval
Reporting Bias Methods Group is
located in Canada
© 2005 The Canadian Cochrane Network and Centre
22
Cochrane Consumer Network




Facilitates the dissemination of
information to patients, their families,
friends and advocates throughout the
world
Plays an integral part to all activities
within The Cochrane Collaboration
Supports the role of consumers within
The Cochrane Collaboration
Liz Whamond (NB) current chair of The
Cochrane Consumer Network
© 2005 The Canadian Cochrane Network and Centre
23
Cochrane reviews


Cochrane reviews are conducted using
standardised rigorous approaches
including detailed critical appraisal of all
studies
Cochrane reviews are reported using
standardised structure including
abstract and consumer summary
© 2005 The Canadian Cochrane Network and Centre
24
Steps of a Cochrane
systematic review
STEP 1: formulate the problem and register the
title with CRG
STEP 2: write protocol, submit for peer review
and publish
STEP 3:
locate and select studies
STEP 4:
critically appraise studies for risk of bias
STEP 5:
collect data
STEP 6:
analyse and present results
STEP 7:
interpret results and write review
STEP 8:
improve and update review
© 2005 The Canadian Cochrane Network and Centre
25
Title registration




Decide which Cochrane CRG would be
responsible
 relevant CRG may be obvious
 most groups are disease (or organ) focused
 if unsure which CRG, contact CCN/C for
advice
Information on CRGs
 The Cochrane Library
 www.cochrane.org
Contact CRG directly
Complete title registration form (from CRG)
© 2005 The Canadian Cochrane Network and Centre
26
Tips for titles




Titles for Cochrane reviews have a set
format
Format ‘intervention’ for ‘problem’ in
‘category’
‘Caffeine’ for ‘daytime sluggishness’ in
‘adults’ (hypothetical for today)
Keep it short
© 2005 The Canadian Cochrane Network and Centre
27
Why have a protocol?






Systematic reviews are scientific
research
Plan methods
Reduce bias
Access to peer review
Avoid duplication of effort
Rest of review follows
 focus for thinking about review
 planning and allocating tasks
© 2005 The Canadian Cochrane Network and Centre
28
Format of a Cochrane protocol

Standard format
 to help review authors be systematic
 to help people reading the reviews to
find information quickly
© 2005 The Canadian Cochrane Network and Centre
29
Format of a Cochrane protocol
1.
2.
3.
4.
5.
6.
Title/review authors
Background
Objectives
Selection criteria
Search strategy
Methods
© 2005 The Canadian Cochrane Network and Centre
30
Background, objectives and
selection criteria
31
Background (Handbook Section 3)




Description of the condition and its
significance
Description of the intervention and its
role in practice
How the intervention might work
Why it is important to do the review
© 2005 The Canadian Cochrane Network and Centre
32
Objectives (Handbook Section 3)



Follow naturally from the background
What are the questions?
 does it work?
 is the effect consistent or variable?
Questions should be clearly framed
© 2005 The Canadian Cochrane Network and Centre
33
Lumping and splitting

Systematic review question can be
framed broadly or narrowly:
 Effects of audit and feedback to
improve chronic diseases across all
health care settings and professionals
 Effects of audit and feedback to
improve chronic diseases within
primary care
 Effects of audit and feedback to
improve diabetes care within primary
care
© 2005 The Canadian Cochrane Network and Centre
34
Lumping and splitting



The ‘lumping’ rationale - systematic reviews
aim to identify the common generalisable
features within similar interventions, minor
differences in trial design are not important.
‘Lumped’ reviews allow generalisability and
consistency of findings to be assessed
across wide range of settings and
populations – this reduces risk of bias or
chance results
The ‘lumping principle’ is that the results of
two interventions should be combined
unless there are good grounds to believe
they will have opposing effects.
© 2005 The Canadian Cochrane Network and Centre
35
Lumping and splitting




The ‘splitting’ rationale is that it is only
appropriate to combine trials which are very
similar in design, patient selection,
intervention characteristics and outcome
recording.
Split reviews avoid combining ‘apples and
oranges’
Reviews can be split by participants,
interventions or outcome
Very narrowly focused reviews are de facto
subgroup analyses
© 2005 The Canadian Cochrane Network and Centre
36
Lumping and splitting
Practical considerations
 Lumped reviews
 Logistically challenging (large
number of included studies)
 Analytically challenging
 Heterogeneity expected
 Interpretation maybe challenging –
seeing the woods for the trees
 Split reviews
 Easier, quicker, cleaner
© 2005 The Canadian Cochrane Network and Centre
37
Lumping and splitting




Effects of audit and feedback (85
studies)
Effects of audit and feedback to improve
chronic diseases across all health care
settings and professionals (18 studies)
Effects of audit and feedback to improve
chronic diseases within primary care
(14 studies)
Effects of audit and feedback to improve
diabetes care within primary care (3
studies)
© 2005 The Canadian Cochrane Network and Centre
38
Selection criteria
(Handbook Section 4)

Follow naturally from the
objectives
 type of participants
 type of interventions
 type of outcomes
 type of studies
© 2005 The Canadian Cochrane Network and Centre
39
Purpose of selection criteria




Help you design the search
Selecting studies
Start thinking about the analysis
One way to minimise bias
© 2005 The Canadian Cochrane Network and Centre
40
Small group exercises
© 2005 The Canadian Cochrane Network and Centre
41
Writing the methods and
completing the protocol
(Handbook sections 6,7,8)
42
Purpose of methods
(Handbook sections 6,7,8)




Plan what you will do before you start
Divide work among review authors
and establish timeline
Minimise bias
Start thinking about the analysis
© 2005 The Canadian Cochrane Network and Centre
43
Methods
(Handbook sections 6,7,8)

Inclusion of trials
- how?
- by who?
© 2005 The Canadian Cochrane Network and Centre
44
Methods
(Handbook sections 6,7,8)

Assessment of risk of bias
- type of assessment
- by who?
- how to incorporate in the review?
© 2005 The Canadian Cochrane Network and Centre
45
Methods
(Handbook sections 6,7,8)

Data extraction
- what will be extracted?
- by who?
- how to deal with missing data?
© 2005 The Canadian Cochrane Network and Centre
46
Methods
(Handbook sections 6,7,8)

Data synthesis
- what type of synthesis to do?
- what summary measures to use?
- what comparisons to make?
- plans to investigate for differences
in study results (heterogeneity) or
potential for missing studies
(publication bias)
- plans for subgroup or sensitivity
analyses?
© 2005 The Canadian Cochrane Network and Centre
47
Writing your protocol


Make sure all sections are completed
- title/review authors, background,
objectives, selection criteria, search
strategy, methods
Don’t forget the other bits and pieces
- contact details of review authors,
acknowledgements, conflict of
interest, sources of support,
references, additional tables
© 2005 The Canadian Cochrane Network and Centre
48
Resources for Cochrane
reviewers





The Cochrane Handbook
Open learning support
Revman
Support from CRGs:
 Trial search co-ordinators
 Review group co-ordinators
 Contact editor
Support from CCN/C
 Training
 Whatever else we can do!
© 2005 The Canadian Cochrane Network and Centre
49
After protocol is complete






Submit to CRG for editorial process
Internal and external peer review 1-2
months
May receive critical comments to respond
to
Protocol published on The Cochrane
Library
Reward yourself then start the review
Note - if you need to change your plan –
report it and justify it (be transparent)
© 2005 The Canadian Cochrane Network and Centre
50
COFFEE!
© 2005 The Canadian Cochrane Network and Centre
51
Session 2: Outline


Locating and selecting studies for your
systematic review
Appraising the risk of bias
© 2005 The Canadian Cochrane Network and Centre
52
Locating and selecting
studies for your
systematic review
53
Why is a rigorous approach
to searching important


To achieve up-to-date, relevant,
unbiased systematic reviews of the
effects of interventions
To overcome problems with limited
searching
 bias
 lack of precision and generalisability
© 2005 The Canadian Cochrane Network and Centre
54
Finding as much as possible
and being strategic
Use multiple sources and start with the
highest yield




Electronic databases
 Cochrane Review Group register
 Cochrane Central Register of Controlled
Trials (CENTRAL)
 MEDLINE
 EMBASE
 Other databases – depending on your topic
Reference lists of trials and reviews
Handsearching journals and conferences
Personal communication
© 2005 The Canadian Cochrane Network and Centre
55
Designing a search strategy

Work with TSC of review group
Formulate your question (PICO)

Identify your search terms

Put it all together

© 2005 The Canadian Cochrane Network and Centre
56
Formulate the question


Translate the clinical problem into a
structured question and identify the key
concepts
 Participants
 Interventions
 Outcomes
 Studies
Develop search terms (commonly based
on participants, interventions, study
design)
© 2005 The Canadian Cochrane Network and Centre
57
Searching tip


Searching is often a matter of trying
terms and seeing what result you get
Find additional terms from the citations
you find that are helpful: how are they
indexed; what are the keywords used?
© 2005 The Canadian Cochrane Network and Centre
58
Documenting your search



Important for you, so you can
remember what you’ve done
Important for users of your review
Document
 what you searched
 when you searched
 how you searched
© 2005 The Canadian Cochrane Network and Centre
59
Additional points



No right time to stop searching; be
guided by returns and resources
Ideas for search terms from citations
you know of; try them and check
returns
Use a referencing database to manage
citations
© 2005 The Canadian Cochrane Network and Centre
60
Minimising bias when searching

Don’t limit search on
 language
 source (which journal, database etc)
 publication status (full article,
conference abstract, unpublished etc)
 publication date
© 2005 The Canadian Cochrane Network and Centre
61
Inclusion and exclusion of
studies






Develop and pilot a form based on your
selection criteria
Scan and retrieve possibly relevant
studies
Decide if studies meet inclusion criteria
and record reasons for exclusion
How reproducible are the decisions?
Done by two independent review
authors
Beware of duplicates
© 2005 The Canadian Cochrane Network and Centre
62
Appraising the risk of bias
63
What is bias?


Bias is systematic (non-random) error
arising from the design or conduct of a
trial
Previously termed ‘methodological
quality’; updated to ‘susceptibility to
bias’ or ‘risk of bias’
© 2005 The Canadian Cochrane Network and Centre
64
Bias vs Quality
Bias
Systematic error influencing
trial estimate of the ‘true’
effect of an intervention
Quality
Based on the conduct of the
trial; did trialists go to sufficient
lengths to minimise potential bias?
© 2005 The Canadian Cochrane Network and Centre
65
Why assess susceptibility to
bias?



To determine whether biases could have
affected the results
Studies with less susceptibility to bias
better estimate the ‘truth’ about the
effect of an intervention
Biased primary studies that provide
misleading results could generate
misleading systematic reviews
© 2005 The Canadian Cochrane Network and Centre
66
© 2005 The Canadian Cochrane Network and Centre
67

Which biases make a difference to trial
results?
© 2005 The Canadian Cochrane Network and Centre
68
Biases
Selection
Performance
Detection
Attrition
© 2005 The Canadian Cochrane Network and Centre
69
Biases
Target Population
Allocation
Selection
Intervention group
Control group
Performance
Exposed to
intervention
Not exposed
to intervention
Detection
Outcome
Outcome
Follow up
Follow up
Attrition
© 2005 The Canadian Cochrane Network and Centre
70
Biases
Critical appraisal
Target Population
Allocation
Selection
Random sequence
Allocation
concealment
Intervention group
Control group
Performance
Blinding of
investigator/
subject
Exposed to
intervention
Not exposed
to intervention
Detection
Blinding of
outcome
assessment
Outcome
Outcome
Attrition
Intention-to-treat
Follow up
Follow up
Follow up
© 2005 The Canadian Cochrane Network and Centre
71
Biases
Target Population
Allocation
Selection
Intervention group
Control group
Performance
Exposed to
intervention
Not exposed
to intervention
Detection
Outcome
Outcome
Follow up
Follow up
Attrition
© 2005 The Canadian Cochrane Network and Centre
72
Selection bias

systematic differences in participant
characteristics at the start of a trial
Intervention group
Control group
© 2005 The Canadian Cochrane Network and Centre
73
Selection bias


Occurs during allocation into
comparison groups
Avoiding selection bias
 random ‘unpredictable’ sequence
 allocation concealment
(protection of the sequence –
most important)
© 2005 The Canadian Cochrane Network and Centre
74
Selection bias
Adequate allocation concealment
 central computer randomisation
 on-site computer from which
assignment can only be determined
after entering patient data
 serially numbered, sealed, opaque
envelopes
Inadequate allocation
concealment
 non opaque envelopes
 odd or even date of birth or
medical record number
© 2005 The Canadian Cochrane Network and Centre
75
Biases
Target Population
Allocation
Selection
Intervention group
Control group
Performance
Exposed to
intervention
Not exposed
to intervention
Detection
Outcome
Outcome
Follow up
Follow up
Attrition
© 2005 The Canadian Cochrane Network and Centre
76
Performance bias

Occurs during the intervention phase

Systematic differences, other than the
intervention being investigated, in the
treatment of the two groups

Participant expectations influencing their
response to treatment

Avoiding performance
bias
- blinding the provider
- blinding the participant
Schulz KF & Grimes DA 2002 Lancet
© 2005 The Canadian Cochrane Network and Centre
77
Allocation concealment vs
blinding
Randomisation
Time
Concealment
of allocation
Blinding
Selection bias
Performance bias
© 2005 The Canadian Cochrane Network and Centre
78
Biases
Target Population
Allocation
Selection
Intervention group
Control group
Performance
Exposed to
intervention
Not exposed
to intervention
Detection
Outcome
Outcome
Follow up
Follow up
Attrition
© 2005 The Canadian Cochrane Network and Centre
79
Detection bias



Occurs during the follow up phase
Systematic differences in the outcome
assessment between groups
Avoiding detection bias
- blinding the outcome assessor
- blinding data analyst/investigator?
© 2005 The Canadian Cochrane Network and Centre
80
Biases
Target Population
Allocation
Selection
Intervention group
Control group
Performance
Exposed to
intervention
Not exposed
to intervention
Detection
Outcome
Outcome
Follow up
Follow up
Attrition
© 2005 The Canadian Cochrane Network and Centre
81
Attrition bias




Occurs over the duration of follow up
Systematic differences in the loss of
participants to follow up between groups
Completeness of follow up
- participants lost to follow up/ not
included in outcome assessment, could
be different from those who remained in
the trial
avoiding attrition bias
- describe proportion of participants lost
to follow-up
- use intention-to-treat analyses
© 2005 The Canadian Cochrane Network and Centre
82
Attrition bias

Intention-to-treat (ITT) analysis
- “once randomised, analysed”
- regardless of whether participants:
 received the intervention
 deviated from the protocol
 withdrew from the study
- maintains randomised allocation of
participants
© 2005 The Canadian Cochrane Network and Centre
83

What is the evidence for these biases
influencing trial results?
© 2005 The Canadian Cochrane Network and Centre
84
Empirical evidence that bias affects
results
Selection bias
Inadequate/unclear allocation concealment
associated with exaggerated treatment effect
(average 29% increased benefit)1,2,3,4
Performance bias
‘Open trials’ (no blinding) exaggerate benefits
of treatment (average 14% increased benefit)
1,2,3,4
1.
2.
3.
4.
5.
Schulz et al, JAMA 1995
Moher et al, Lancet, 1998
Kjaergard et al, Ann Intern Med, 2001
Egger et al, HTA, 2003
Juni et al, JAMA, 1999
© 2005 The Canadian Cochrane Network and Centre
85
Empirical evidence that bias affects
results
Detection bias
Open outcome assessment exaggerated treatment
effects (17%1 and 35%5)
Attrition bias
Handling of dropouts did not affect results1,5
however, compared whole group differences (not
between group differences). Not an exclusion but
important to report.
1.
2.
3.
4.
5.
Schulz et al, JAMA 1995
Moher et al, Lancet, 1998
Kjaergard et al, Ann Intern Med, 2001
Egger et al, HTA, 2003
Juni et al, JAMA, 1999
© 2005 The Canadian Cochrane Network and Centre
86
Assessing susceptibility to
bias

How do we assess susceptibility to
bias?
- Scales
- Checklists
© 2005 The Canadian Cochrane Network and Centre
87
Beware of quality scales….



Different scales are available (>50)
- vary in dimensions covered, size,
complexity
- some provide composite numerical
score
Reliability and validity of many scales not
established
May lead to different conclusions
regarding the susceptibility to bias of the
same trial (Juni 1999)
© 2005 The Canadian Cochrane Network and Centre
88
Treatment estimates from
high-quality trials
Treatment estimates from
low-quality trials
Same group of
trials
divided into high
and
low quality using
25 different
Quality
Assessment
Scales
Juni P, Witschi A,
Bloch R, Egger M.
The hazards of
scoring the quality
of clinical trials for
meta-analysis.
JAMA
1999;282(11):105460
© 2005 The Canadian Cochrane Network and Centre
89
How to assess susceptibility to
bias

Component approach: assessment
against individual criteria
© 2005 The Canadian Cochrane Network and Centre
90
Assessing susceptibility to bias
in reviews






Was the allocation sequence adequately
generated (randomised)?
Was the allocation adequately concealed?
Was there blinding of participants,
personnel
Was there blinding of outcome assessors?
Were dropouts and exclusions adequately
addressed?
Other (including selective outcome
reporting and other criteria for addressing
topic-specific or design-specific biases)
© 2005 The Canadian Cochrane Network and Centre
91

How do we incorporate susceptibility to
bias in meta-analysis?
© 2005 The Canadian Cochrane Network and Centre
92
Incorporating susceptibility to
bias in meta-analysis


Exclude trials with high susceptibility to bias
- critical a priori cutoff for inclusion (e.g. only
trials with adequate allocation concealment
or blinding)
- may be problematic if few trials
Sensitivity analysis
- analyse separately above and below a certain
a priori level and compare results (does it
change conclusions?)
- for instance, analyse the impact of allocation
concealment, blinding, use of ITT analysis, on
results
© 2005 The Canadian Cochrane Network and Centre
93
Example of sensitivity analysis
© 2005 The Canadian Cochrane Network and Centre
94
But beware…

Scant empirical evidence about which biases are
important

Reporting of susceptibility to bias is not always the
same as the susceptibility to bias in the conduct of
the trial
- allocation concealment unclear in 87% of trials as per
published report (Pildal et al, 2005)
- allocation concealment unclear in about 80% of trials
(Middleton 2001)

CONSORT statement (1994, 2001) www.consort-
statement.org

Possible discrimination against older trials
© 2005 The Canadian Cochrane Network and Centre
95
Important points for your
review



Plan criteria and how you will
incorporate susceptibility to bias at the
protocol stage
Assessment undertaken once trials are
included and prior to data extraction
2 independent review authors
© 2005 The Canadian Cochrane Network and Centre
96