Practical Meta-Analysis David B. Wilson Evaluators’ Institute July 16-17, 2010 Practical Meta-Analysis -- D.
Download
Report
Transcript Practical Meta-Analysis David B. Wilson Evaluators’ Institute July 16-17, 2010 Practical Meta-Analysis -- D.
Practical Meta-Analysis
David B. Wilson
Evaluators’ Institute
July 16-17, 2010
Practical Meta-Analysis -- D. B. Wilson
1
Overview of the Workshop
Topics covered will include
Review of the basic methods
Problem definition
Document Retrieval
Coding
Effect sizes and computation
Analysis of effect sizes
Publication Bias
Cutting edge issues
Interpretation of results
Evaluating the quality of a meta-analysis
Practical MetaAnalysis -- D. B.
Wilson
2
Forest Plot from a Meta-Analysis of
Correctional Boot-Camps
Fa
vo
rs C
om
pa
rison
Fa
vo
rs B
oo
tca
m
p
H
a
rer &K
lein-S
a
ffran
,1
99
6
Jon
es &R
o
ss, 1
997
Fl. D
e
pt. of JJ (S
tu
art C
o.), 199
7
Fl. D
e
pt. of JJ (P
olk C
o., B
oys), 19
97
Jon
es (FY
9
7), 1
99
8
Jon
es (FY
9
4-95
), 1
99
8
M
acke
nzie&S
o
uryal (Illin
ois), 1
99
4
M
acke
nzie&S
o
uryal (Lo
uisia
na
), 1
994
Jon
es (FY
9
1-93
), 1
99
8
M
acke
nzie&S
o
uryal (Flo
rid
a), 1
99
4
Jon
es (FY
9
6), 1
99
8
M
arcus-M
end
oza (M
e
n), 1
99
5
M
acke
nzie
,e
t al. 1
99
7
P
e
nn. D
ep
t. o
fC
o
rrectio
ns, 2
001
Flow
e
rs, C
arr, &R
ub
ack 199
1
B
u
reauo
fD
ata a
ndR
esea
rch
, 19
96
M
acke
nzie&S
o
uryal (O
klaho
m
a
), 1
99
4
T3A
sso
ciate
s, 20
00
M
acke
nzie&S
o
uryal (N
ewY
o
rk), 19
94
P
e
ters, 1
99
6b
C
a
m
p &S
a
ndh
u, 199
5
M
acke
nzie&S
o
uryal (S
.C
., N
ew
), 1
99
4
Jon
es, 1
996
N
YD
C
S(88
-96R
ele
ases), 2
00
0
M
arcus-M
end
oza (W
om
en
), 1
995
A
u
stin, Jo
ne
s, &B
olya
rd, 199
3
B
u
rns &V
ito, 199
5
P
e
ters, 1
99
6a
Fl. D
e
pt. of JJ (B
ay C
o
.), 1
99
7
N
YD
C
S(96
-97R
ele
ases), 2
00
0
N
YD
C
S(97
-98R
ele
ases), 2
00
0
Fl. D
e
pt. of JJ (P
in
ella
sC
o
.), 1
99
6
Fl. D
e
pt. of JJ (M
ana
te
eC
o
.), 1
996
C
AD
e
pt. of theY
ou
thA
uth
ority, 19
97
B
o
yles, B
o
ke
nkam
p, &M
a
du
ra, 1
99
6
M
acke
nzie&S
o
uryal (S
.C
., O
ld), 19
94
Fl. D
e
pt. of JJ (P
olk C
o., G
irls), 1
99
7
Jon
es, 1
997
Th
om
as &P
e
ters, 1
99
6
W
righ
t &M
a
ys, 1
998
M
acke
nzie&S
o
uryal (G
eorg
ia
), 19
94
Practical MetaAnalysis -- D. B.
Wilson
O
verall M
e
anO
d
ds-R
atio
3
The Great Debate
1952: Hans J. Eysenck concluded that there were no
favorable effects of psychotherapy, starting a raging
debate
20 years of evaluation research and hundreds of studies
failed to resolve the debate
1978: To proved Eysenck wrong, Gene V. Glass
statistically aggregate the findings of 375 psychotherapy
outcome studies
Glass (and colleague Smith) concluded that
psychotherapy did indeed work
Practical“meta-analysis”
MetaGlass called his method
Analysis -- D. B.
4
Wilson
The Emergence of Meta-analysis
Ideas behind meta-analysis predate Glass’ work by
several decades
Karl Pearson (1904)
averaged correlations for studies of the effectiveness of
inoculation for typhoid fever
R. A. Fisher (1944)
“When a number of quite independent tests of significance
have been made, it sometimes happens that although few or
none can be claimed individually as significant, yet the
aggregate gives an impression that the probabilities are on the
whole lower than would often have been obtained by chance”
(p. 99).
Practical Meta Source of the idea of cumulating probability values
Analysis -- D. B.
Wilson
5
The Emergence of Meta-analysis
Ideas behind meta-analysis predate Glass’ work by
several decades
W. G. Cochran (1953)
Discusses a method of averaging means across independent
studies
Laid-out much of the statistical foundation that modern metaanalysis is built upon (e.g., Inverse variance weighting and
homogeneity testing)
Practical MetaAnalysis -- D. B.
Wilson
6
The Logic of Meta-analysis
Traditional methods of review focus on statistical
significance testing
Significance testing is not well suited to this task
Highly dependent on sample size
Null finding does not carry the same “weight” as a significant
finding
significant effect is a strong conclusion
nonsignificant effect is a weak conclusion
Meta-analysis focuses on the direction and magnitude
of the effects across studies, not statistical significance
Isn’t this what we are interested in anyway?
Direction and magnitude
areMetarepresented by the effect size
Practical
Analysis -- D. B.
Wilson
7
Illustration
Table 1
21 Validity Studies, N = 68 for Each
Observed
validity
Study
coefficient
1
0.04
2
0.14
3
0.31 *
4
0.12
5
0.38 *
6
0.27 *
7
0.15
8
0.36 *
9
0.20
10
0.02
11
0.23
12
0.11
13
0.21
14
0.37 *
15
0.14
16
0.29 *
17
0.26 *
18
0.17
19
0.39 *
20
0.22
21
0.21
* p < .05 (two tailed).
Simulated data from 21 validity studies. Taken from: Schimdt, F. L.
(1996). Statistical significance testing and cumulative knowledge in
psychology: implications
for training
of researchers. Psychological
Practical
MetaMethods, 1, 115-129.
Analysis -- D. B.
Wilson
8
Illustration (Continued)
Table 2
95% Confidence Intervals for Correlations From Table
1, N = 68 for Each
Observed
95% confidence
validity
interval
Study
coefficient
Lower
Upper
1
0.39
0.19
0.59
2
0.38
0.18
0.58
3
0.37
0.16
0.58
4
0.36
0.15
0.57
5
0.31
0.09
0.53
6
0.29
0.07
0.51
7
0.27
0.05
0.49
8
0.26
0.04
0.48
9
0.23
0.00
0.46
10
0.22
-0.01
0.45
11
0.21
-0.02
0.44
12
0.21
-0.02
0.44
13
0.20
-0.03
0.43
14
0.17
-0.06
0.40
15
0.15
-0.08
0.38
16
0.14
-0.09
0.37
17
0.14
-0.09
0.37
18
0.12
-0.12
0.36
19
0.11
-0.13
0.35
20
0.04
-0.20
0.28
21
0.02
-0.22
0.26
Practical MetaAnalysis -- D. B.
Wilson
9
When Can You Do Meta-analysis?
Meta-analysis is applicable to collections of research that
Are empirical, rather than theoretical
Produce quantitative results, rather than qualitative findings
Examine the same constructs and relationships
Have findings that can be configured in a comparable statistical
form (e.g., as effect sizes, correlation coefficients, odds-ratios,
proportions)
Are “comparable” given the question at hand
Practical MetaAnalysis -- D. B.
Wilson
10
Forms of Research Findings Suitable to Metaanalysis
Central tendency research
Pre-post contrasts
Prevalence rates
Growth rates
Group contrasts
Experimentally created groups
Comparison of outcomes between treatment and comparison
groups
Naturally occurring groups
Comparison of spatial abilities between boys and girls
Rates of morbidity among high and low risk groups
Practical MetaAnalysis -- D. B.
Wilson
11
Forms of Research Findings Suitable to Metaanalysis
Association between variables
Measurement research
Validity generalization
Individual differences research
Correlation between personality constructs
Practical MetaAnalysis -- D. B.
Wilson
12
Effect Size: The Key to Meta-analysis
The effect size makes meta-analysis possible
It is the “dependent variable”
It standardizes findings across studies such that they can be
directly compared
Practical MetaAnalysis -- D. B.
Wilson
13
Effect Size: The Key to Meta-analysis
Any standardized index can be an “effect size” (e.g.,
standardized mean difference, correlation coefficient,
odds-ratio) as long as it meets the following
Is comparable across studies (generally requires standardization)
Represents the magnitude and direction of the relationship of
interest
Is independent of sample size
Different meta-analyses may use different effect size
indices
Practical MetaAnalysis -- D. B.
Wilson
14
The Replication Continuum
Conceptual
Replications
Pure
Replications
You must be able to argue that the collection of studies you are
meta-analyzing examine the same relationship. This may be at
a broad level of abstraction, such as the relationship between
criminal justice interventions and recidivism or between schoolbased prevention programs and problem behavior. Alternatively
it may be at a narrow level of abstraction and represent pure
replications.
The closer to pure replications your collection of studies, the
easier it is to argue comparability.
Practical MetaAnalysis -- D. B.
Wilson
15
Which Studies to Include?
It is critical to have an explicit inclusion and exclusion
criteria (see pages 20-21)
The broader the research domain, the more detailed they tend to
become
Refine criteria as you interact with the literature
Components of a detailed criteria
distinguishing features
research respondents
key variables
research methods
cultural and linguistic range
time frame
publication types
Practical Meta-
Analysis -- D. B.
Wilson
16
Methodological Quality Dilemma
Include or exclude low quality studies?
The findings of all studies are potentially in error (methodological
quality is a continuum, not a dichotomy)
Being too restrictive may restrict ability to generalize
Being too inclusive may weaken the confidence that can be
placed in the findings
Methodological quality is often in the “eye-of-the-beholder”
You must strike a balance that is appropriate to your research
question
Practical MetaAnalysis -- D. B.
Wilson
17
Searching Far and Wide
The “we only included published studies because they
have been peer-reviewed” argument
Significant findings are more likely to be published than
nonsignificant findings
Critical to try to identify and retrieve all studies that meet
your eligibility criteria
Practical MetaAnalysis -- D. B.
Wilson
18
Searching Far and Wide (continued)
Potential sources for identification of documents
Computerized bibliographic databases
“Google” internet search engine
Authors working in the research domain (email a relevant
Listserv?)
Conference programs
Dissertations
Review articles
Hand searching relevant journal
Government reports, bibliographies, clearinghouses
Practical MetaAnalysis -- D. B.
Wilson
19
A Note About Computerized Bibliographies
Rapidly changing area
Get to know your local librarian!
Searching one or two databases is generally inadequate
Use “wild cards” (e.g., random? will find random,
randomization, and randomize)
Throw a wide net; filter down with a manual reading of
the abstracts
Practical MetaAnalysis -- D. B.
Wilson
20
Strengths of Meta-analysis
Imposes a discipline on the process of summing up
research findings
Represents findings in a more differentiated and
sophisticated manner than conventional reviews
Capable of finding relationships across studies that are
obscured in other approaches
Protects against over-interpreting differences across
studies
Can handle a large numbers of studies (this would
overwhelm traditional approaches to review)
Practical MetaAnalysis -- D. B.
21
Wilson
Weaknesses of Meta-analysis
Requires a good deal of effort
Mechanical aspects don’t lend themselves to capturing
more qualitative distinctions between studies
“Apples and oranges” criticism
Most meta-analyses include “blemished” studies to one
degree or another (e.g., a randomized design with
attrition)
Selection bias posses a continual threat
Negative and null finding studies that you were unable to find
Outcomes for which there were negative or null findings that were
not reported
Analysis of between
studyMetadifferences is fundamentally
Practical
correlational
Analysis -- D. B.
22
Wilson