Transcript DESIGN OF RANDOMISED

Systematic Reviews and
Meta-Analysis
Methodologies for a new era summer school
School of Applied Social Studies, University
College Cork
20 June 2011
Dr Paul Montgomery
Aims
1) Discuss the advantages and main
features of systematic reviews
2) Introduce basic principles of metaanalysis
Course feedback
The Problem
Millions of articles published in
thousands of journals each year
Practitioners and researchers are busy
Subjective summaries may misrepresent
research
Reviews
Systematic Reviews


Aim to answer specific questions, reduce
uncertainty, identify outstanding questions
Common methods include narrative
synthesis, meta-analysis (meta-regression)
Traditional ‘journalistic’ reviews


Aim to persuade, draw attention to a topic,
synthesise information, etc.
Narrative synthesis most common
Systematic Review:
“the application of scientific strategies
that limit bias to the systematic
assembly, critical appraisal, and
synthesis of all relevant studies on a
specific topic."
Cook DJ, Sakett DL, Spitzer WO. Methodological guidelines for systematic
reviews of randomized contro trials in health care from the Potsdam Consultation
on Meta-Analysis. J. Clin. Epidemiol. 1995;48:167-71
Systematic Reviews
Clear Question

Define the population, problem, intervention,
alternative interventions, and outcomes
Replicable Method



Search strategy
Inclusion criteria
Analytical strategy
Transparent Process
Advantages
 Explicit methods limit bias in identifying and
rejecting studies
 Information can be understood quickly
 Reduced delay between discoveries and
implementation
 Results can be formally compared
 Heterogeneity can be identified and new
hypotheses generated
 Quantitative reviews increase precision
Producers
Cochrane
Campbell
EPPI
DARE
NICE
Interested practitioners/ academics
Cochrane Review Process
 Register titles and check for overlap
 Protocols developed and peer reviewed
 Searches performed widely on all main
databases, grey literature searches, personal
contacts
 Abstracts reviewed by two authors
 Data collected and trial quality assessed
 Data synthesis and analysis
 Write-up
 Reviewed by Cochrane/ Campbell editors, then
peer reviewed
Systematic reviews
Key components:
1. Ask a good question
2. Identify studies
3. Extract data
4. Synthesise data
5. Interpreting the results
Who Should Review?
“Experts, who have been steeped in a subject for
years and know what the answer ‘ought’ to be,
are less able to produce an objective review of
the literature in their subject than non-experts.
This would be of little consequence if experts'
opinions could be relied on to be congruent
with the results of independent systematic
reviews, but they cannot.”
(Trisha Greenhalgh)
PICO Mad-libs
P
I
C
O
For
____________ does
____________ compared to
____________ improve/reduce
____________ ?
Highly Sensitive Search
Electronic Searches
Databases/Indexes
Additional Electronic Searches
Hand Searches
Personal Contacts
Electronic databases:
PsycInfo
covers 73%
and has a
psychological
focus
Searching PsycLit and
Embase will cover 92%
of the core 505
‘psychiatric’ journals.
PsycInfo
Embase
Medline
Medline covers 23%
of the core 505
‘psychiatric’
journals, plus most of
the major biomedical
journals.
BA
Embase covers
67% plus lots of
European journals
that Medline
misses.
Biological Abstracts covers
48%, plus lots of life
sciences stuff.
Electronic Searches
 Sensitivity vs. Specificity
Even if 2 terms and 3 databases return almost all
literature on a subject, the goal of a systematic
review is to find everything.
Electronic Searches
Specific Authors
Reverse Citation
Agencies / Non-Profits
Funding Bodies
Academic Groups / Research Centers
Google
Additional Searches
Previous Reviews
Bibliographies of Related Articles
Hand Search Journals (that aren’t
indexed)
Conference Reports
(many are electronically published)
Personal Communication
 Call or Email Authors
 Attend Conferences
 Write to:
Agencies / Non-Profits
Providers / Manufacturers / Distributors
Funding Bodies
Academic Groups / Research Centers
Questions to Ask
 Which programs will be studied?
 Compared to what?
 What study designs are acceptable?
 What must a study measure?
 How must it be measured?
 Must researchers be blind at allocation,
during the trial, etc?
 How will dropouts be handled?
 What about missing data?
Inclusion and Exclusion
Types of studies
 Types of participants
 Types of comparisons

Specify:
Types of outcomes
Multiplicity (time, comparisons, measures,
statistics)

Transparency
Be clear about all definitions, searches,
inclusion and exclusion criteria, etc.
Report ongoing trials
List excluded studies, particularly if:



The trials contain valuable information
Exclusion was a close call
You discovered something about a trial
Evaluating a Review
Even if a review is
‘systematic’ it may not
be well-conducted.
How do we tell the
difference?

Validity
1. Did the review address a clearly
focussed question?
2. Were the right sort of studies
selected?
3. Was the search strategy explicit and
comprehensive?
4. Did the reviewers assess the quality
of the identified studies?
Importance:
1. Were the results similar from study to
study?
2. What is the overall result of the
review?
3. How precise are the results?
Potential Sources of Bias
Describe aspects of study design that
might have influenced the magnitude or
direction of results
Use of rating scales with fixed cut-offs
potentially misleading
Consider external validity
Juni P, Witschi A, Bloch R,
Egger M. The hazards of
scoring the quality of
clinical trials for metaanalysis. JAMA 1999; 282:
1054-1060
Tower of Babel
Studies that find a treatment effect are
more likely to be published in Englishlanguage journals.
Opposing studies may be published in
non-English-language journals.
Gregoire G, Derderan F, Le Lorier J. Selecting the language of the publications
included in a meta-analysis: is there a Tower of Babel Bias? J.Clin.Epidemiol.
1995;48:159-163
Publication Bias
“the tendency of investigators, reviewers
and editors to differentially submit or
accept manuscripts for publication on
the direction or strength of the study
findings.”
Cook DJ, Guyatt GH, Ryan G, Clifton J, Buckingham L, Willan A et al. Should
unpublished data be included in meta-analyses? Current convictions and controversies.
JAMA 1993; 269: 2749-2753
Unpublished data
Controversial
Unpublished data may not be a full or
representative sample (Cook 1993)
Publication is no guarantee of scientific quality
(Oxman 1991)
Cook DJ, Guyatt GH, Ryan G, Clifton J, Buckingham L, Willan A et al. Should
unpublished data be included in meta-analyses? Current convictions and controversies.
JAMA 1993; 269: 2749-2753
Oxman AD, Guyatt GH, Singer J, Goldsmith CH, Hutchison BG, Milner RA et al.
Agreement among reviewers of review articles. J.Clin.Epidemiol. 1991;44:91-98.
Meta-analyses:
“A systematic review that employs
statistical methods to combine and
summarise the results of several
studies.”
Cook DJ, Sakett DL, Spitzer WO. Methodological guidelines for systematic
reviews of randomized contro trials in health care from the Potsdam Consultation
on Meta-Analysis. J. Clin. Epidemiol. 1995;48:167-71
Summarising trials
Systematic reviews
Meta-analyses
Reviews
Meta-analyses
 Mathematically combine the results of
different studies
 For dichotomous or continuous outcomes
 From analytical (treatment) or observational
(aetiology, diagnosis, prognosis) studies
 ‘Weighted’ by study size (usually 1/se2)
and/or quality
Benefits of meta-analysis:
1. To increase statistical power for primary end
points and for subgroups.
2. To improve estimates of effect size.
3. To resolve uncertainty when reports disagree
4. To answer questions not posed at the start of
individual trials.
Sacks HS, Berrier J, Reitman D, Ancona-Berk VA, Chalmers TC. Meta-analyses of
randomized controlled trials. N.Engl.J.Med. 1987;316:450-455
Outcome Measures
Continuous / Dichotomous (/ Ordinal)
Objective / Subjective
Meta-analysis
 Some outcomes are measured on scales –
e.g. depression or continuously e.g. sleep
minutes
 Continuous outcomes can be calculated using
the scale on which they were measured
(WMD)
 If changes in depression are measured on
different scales it is still possible to combined
them but on a standardised scale
Meta-analysis
 Alternatively we might be interested in
binary data - two mutually exclusive states
 Dead/alive; hospitalised/not hospitalised
 These data will be measured in a different
way to continuous (scale) data
 Reported as ‘event rates’
Meta-analysis
Central Tendency:


Mean (Cohen’s d, Hedges’s g)
Odds Ratio / Relative Risk / Rate Ratio
Variance (Confidence Interval)
Clinical Significance (NNT/NNH)
Heterogeneity (I2, Q, Chi2)
Dichotomous Outcomes
Odds are calculated by dividing the
number of events by non-events (ie
clients experiencing the event divided
by clients not experiencing an event)
Risk/Rate is more widely reported in
reviews as it tends to be easier to
communicate
Weighting
Some studies contribute more weight to
the ‘average’ result than do others
The more precise the effect estimate,
the more weight is given
Wide variation is sometimes associated
with small studies
Weighting
Clinical trials are rarely conducted according to
identical protocols
Severity of the problem, intensity of the
intervention, duration, setting of trial, age may
account for differences in response
Apples and oranges?
Sources of Heterogeneity:







Study participants
Comparisons
Intervention design
Delivery
Duration of follow-up
Outcome measures
Methods
Heterogeneity


Estimates from individual trials vary more than can
be explained by the play of chance alone
N.B. Meta-analysis should NOT overlook important
material differences in subgroup response
Heterogeneity – approaches




Qualitative v. quantitative
Qualitative – reconsider pooling
Does it makes sense to average effects
from the studies?
Fixed v. random effects
Subgroup Analysis
If together there is excessive variation,
when analysed separately there is a
uniform response to treatment in each
subgroup
Hypothesis generating
Sensitivity analysis:
Sensitivity analyses investigate how the
conclusions of a review change when one
or more of the decisions or assumptions
are altered.
Testing for heterogeneity
Look at plots of results
Formal tests of homogeneity

I2

Q

Chi2
Assess qualitative differences in study
design or implementation
.
AnxietyAnxiety
(self-rated) (Self-Rated
at Post-Treatment
compared to No-Treatment
Symptoms)
at Post-Treatment
Weeks
Study name
Comparison
Outcome
Statistics for each study
Hedges's
g
Sample size
Hedges's g and 95% CI
Lower
limit
Upper
limit
Media
Comp
Relative
weight
4
Hassan 1992
Wait List
Combined
4.30
2.55
6.05
10
8
0.68
5
Bickel 2007
Wait List
Combined
0.81
-0.33
1.96
8
5
1.40
4
Milne 1998
Wait List (TaU)
Combined
-0.33
-1.35
0.70
7
6
1.67
8
Rosen 1976
Wait List
Combined
-0.73
-1.72
0.27
16
6
1.74
1
Klein 2001
Wait List
Combined
0.66
-0.18
1.51
10
12
2.20
8
Abramowitz 2009
Wait List
Combined
0.39
-0.45
1.22
11
10
2.23
8
Lidren 1994
Wait List
Combined
0.91
0.09
1.74
12
12
2.29
8
Richards 2006
Wait List
Combined
0.63
-0.14
1.41
23
9
2.46
12
Fletcher 2005
Wait List
HADS - Anxiety
0.10
-0.65
0.86
11
15
2.56
1
Heading 2001
Wait List
Combined
0.17
-0.58
0.92
13
13
2.57
13
Kiely 2002
Wait List (TaU)
Combined
0.85
0.12
1.58
16
14
2.66
8
Lewis 1978
Monitoring
Combined
0.58
-0.13
1.29
38
10
2.77
4
Jones 2002
Wait List (TaU)
Combined
0.58
-0.06
1.21
19
20
3.15
8
Grime 2004
Wait List
HADS - Anxiety
0.41
-0.22
1.04
16
23
3.16
10
Carlbring 2001
Wait List
Combined
0.81
0.18
1.44
21
20
3.18
8
Sorby 1991
No Int (Plus TaU)
Combined
0.62
-0.00
1.24
25
17
3.22
6
Smith 1997
Attention
Combined
0.33
-0.29
0.94
30
15
3.25
10
Titov 2009
Wait List
Combined
0.98
0.37
1.59
24
21
3.27
11
Arpin-Cribbie 2007
No Int
Combined
0.81
0.24
1.38
29
22
3.50
10
Berger 2009
Wait List
Combined
0.75
0.18
1.31
31
21
3.55
10
Carlbring 2006
Wait List
Combined
1.19
0.64
1.74
30
30
3.63
9
Carlbring 2007
Wait List
Combined
1.01
0.47
1.55
29
29
3.70
14
Hazen 1996
Wait List
Combined
0.43
-0.10
0.97
27
27
3.75
13
Zetterqvist 2003
Wait List
Combined
0.39
-0.07
0.84
37
45
4.28
12
Van Boeijen 2005
Treatment as Usual Combined
-0.13
-0.59
0.32
53
28
4.29
10
Titov 2008b
Wait List
Combined
0.79
0.34
1.24
41
40
4.32
10
Titov 2008c
Wait List
Combined
0.45
0.02
0.87
61
34
4.54
10
Titov 2008a
Wait List
Combined
0.83
0.42
1.23
50
49
4.64
13
Mead 2005
Wait List
HADS
0.18
-0.20
0.56
50
53
4.83
12
Rapee 2007
Wait List
Combined
0.38
0.00
0.76
56
52
4.87
9
Proudfoot 2004
Treatment as Usual BAI
0.38
0.10
0.66
99
98
5.65
0.55
0.40
0.70
903
764
-2.00
-1.00
0.00
1.00
2.00
Institutionalisation (RR<1 favours home visits)
Group by
Time
Study name
Time (m)
Risk ratio and 95% CI
Statistics for each study
Relative
weight
Risk
ratio
Lower
limit
Upper
limit
12-23
Dalby (2000)
14
1.20
0.35
0.01
8.41
12-23
Hogan (2001)
12
2.14
2.05
0.19
22.17
12-23
Newbury (2001)
12
3.21
0.94
0.13
6.55
12-23
Hall (1992)
12
7.41
0.38
0.11
1.37
12-23
Hebert (2001)
12
8.06
1.04
0.30
3.53
12-23
Kono (2004)
18
10.83
0.64
0.22
1.83
12-23
Yamada (2003)
18
17.26
1.30
0.56
3.01
12-23
Bernabei (1998)
12
21.52
0.67
0.32
1.43
12-23
Gill (2002)
12
28.37
0.72
0.38
1.39
0.78
0.55
1.10
12-23
24-35
Hall (1992)
24
42.30
0.16
0.04
0.67
24-35
Sorenson (1988)
30
57.70
1.02
0.81
1.28
0.46
0.08
2.80
24-35
36+
Hall (1992)
36
9.53
0.16
0.04
0.67
36+
Van Rossum (1993)
36
12.00
1.38
0.44
4.30
36+
Pathy (1992a)
36
12.86
1.29
0.46
3.66
36+
Byles (2004)
36
14.97
2.84
1.25
6.43
36+
Stuck (1995)
36
15.52
0.42
0.20
0.90
36+
Pathy (1992b)
36
16.48
0.56
0.29
1.09
36+
Stuck (2000)
36
18.64
1.51
0.99
2.30
0.90
0.49
1.67
36+
0.1
0.2
0.5
1
2
5
10
Funnel Plot of Standard Error by Log risk ratio
Mortality: Trim and Fill (missing studies shown, 1 trimmed)
0.0
Standard Error
0.5
1.0
1.5
2.0
-2.0
-1.5
-1.0
-0.5
0.0
Log risk ratio
0.5
1.0
1.5
2.0
Anxiety (Self-Rated
Symptoms) at Post-Treatment
Funnel Plot of Standard Error by Hedges's g
0.0
0.2
Standard Error
0.4
0.6
0.8
1.0
-4
-3
-2
-1
0
1
2
3
4
Regression of Ave Age on Log risk ratio
Mortality by age: Meta-regression
2.00
1.60
1.20
Log risk ratio
0.80
0.40
0.00
-0.40
-0.80
-1.20
-1.60
-2.00
67.16
69.01
70.86
72.70
74.55
76.40
Ave Age
78.25
80.10
81.94
83.79
85.64
Anxiety (Self-Rated Symptoms) at Post-Treatment:
Number of contacts with researchers and clinicians
Point
Estimate
SE
P
Slope
0.03
<0.01
<0.001
Intercept
0.27
0.08
<0.01
Q
df
P
Model
12.93
1
<0.001
Residual
30.34
27
43.27
28
0.30
0.03
Regression of Number of Contacts on Hedges's
Totalg
2.00
1.72
1.44
Hedges's g
1.16
0.88
0.60
0.32
0.04
-0.24
-0.52
-0.80
-1.35
1.47
4.29
7.11
9.93
12.75
15.57
18.39
21.21
24.03
26.85
Limitations
Junk-In, Junk-Out
The results of large trials sometimes
differ
Chance Events: Aggregation and
Disaggregation
Conclusion
Systematic reviews seek to reduce
bias and improve the reliability and
accuracy of the conclusions.
Meta-analysis is a powerful research
tool, but it should be conducted only
in the context of a systematic review,
and it has important limitations.