Transcript META-ANALYSIS OF RESEARCH

LECTURE 4
EPSY 652 FALL 2009
Computing Effect Sizes- Mean
Difference Effects
 Glass: e = (MeanExperimental – MeanControl)/SD
o SD = Square Root (average of two variances) for randomized
designs
o SD = Control standard deviation when treatment might affect
variation (causes statistical problems in estimation)
 Hedges: Correct for sampling bias:
g = e[ 1 – 3/(4N – 9) ]
 where N=total # in experimental and control groups
 Sg = [ (Ne + Nc)/NgNc + g2/(2(Ne + Nc) ]½
Computing Effect Sizes- Mean Difference Effects Example from
Spencer ADHD Adult study
 Glass: e = (MeanExperimental – MeanControl)/SD
= (82 – 101)/21.55
= .8817
 Hedges: Correct for sampling bias:
g = e[ 1 – 3/(4N – 9) ]
= .8817 (1 – 3/(4*110 – 9)
= .8762
Note: SD computed from t-statistic of 4.2 given in article:
e = t*(1/NE + 1/NC )½
A
1
effect
2
1
3
2
4
3
5
4
6
5
7
6
8
7
9
8
10
11
mean
12 s(mean)
B
Mean E
1
0.3
0.8
0.5
0.2
0.4
1
0.46
C
Mean C
0.2
-0.4
0.28
-0.46
-0.8
-0.12
0.36
-0.5
D
SDE
1
1
1
1
1
1
1
1
E
SDC
1
1
1
1
1
1
1
1
F
G
H
Hedge
s g Ctrl N
d
0.60 0.58
10
-0.05 -0.05 21
0.54 0.52
9
0.02 0.02
18
-0.30 -0.30 73
0.14 0.14
52
0.68 0.68 117
-0.02 -0.02
8
0.2154 38.50
0.0793
I
J
Trmt N
13
20
9
21
94
71
115
8
N
23
41
18
39
167
123
232
16
43.88
82.38
K
w
L
wd
5.43
3.14
10.24 -0.53
4.35
2.25
9.69
0.20
40.65 -12.10
29.94
4.20
54.85 37.17
4.00 -0.06
159.15
34.28
Computing Mean Difference Effect Sizes from
Summary Statistics
½
 t-statistic: e = t*(1/NE + 1/NC )
 F(1,dferror):
e = F½ *(1/NE + 1/NC )½
 Point-biserial correlation:
e = r*(dfe/(1-r2 ))½ *(1/NE + 1/NC )½
 Chi Square (Pearson association):
 = 2/(2 + N)
e = ½*(N/(1-))½ *(1/NE + 1/NC )½
 ANOVA results: Compute R2 = SSTreatment/Sstotal
Treat R as a point biserial correlation
Excel workbook for Mean difference computation
STATISTIC
OUTCOME# TYPE
MEAN E MEAN C SD E SD C Ne Nc N
STUDY#
1
2
3
4
5
6
means,
1 SDs
1 t-statistic
101
101
SUMMARY COMPUTATI
STATISTIC ON
d
intermediate intermediate
computation computation hedges g
82 22 21 78 32 110
82
78 32 110
19 0.8817 0.8817
4.2 0.881705 0.881705
0.875563
0.875568
1 F-statistic
point1 biserial r
78 32 110
17.64 0.881705 0.881705
4.2
0.875568
78 32 110 0.374701 0.881705 0.881705
17.64
4.2 0.875568
1 chi square
p(tstatistic)
47 76 123
3.66 0.654634 0.654634 0.169989 0.169989 0.650568
47 76 123
0.05 0.654634 0.654634 1.979764
0.650568
Story Book Reading
References
1 Wasik & Bond: Beyond the Pages of a Book: Interactive Book Reading
and Language Development in Preschool Classrooms. J. Ed Psych 2001
2 Justice & Ezell. Use of Storybook Reading to Increase Print Awareness in
At-Risk Children. Am J Speech-Language Path 2002
3 Coyne, Simmons, Kame’enui, & Stoolmiller. Teaching Vocabulary During
Shared Storybook Readings: An Examination of Differential Effects.
Exceptionality 2004
4 Fielding-Barnsley & Purdie. Early Intervention in the Home for Children
at Risk of Reading Failure. Support for Learning 2003
Coding the Outcome
1 open Wasik & Bond pdf
2 open excel file “computing mean effects example”
3 in Wasik find Ne and Nc
4 decide on effect(s) to be used- three outcomes are
reported: PPVT, receptive, and expressive vocabulary
at classroom and student level: what is the unit to be
focused on? Multilevel issue of student in classroom,
too few classrooms for reasonable MLM estimation,
classroom level is too small for good power- use
student data
Coding the Outcome
5 Determine which reported
data is usable: here the AM and
PM data are not usable because we don’t have the
breakdowns by teacher-classroom- only summary tests can
be used
6 Data for PPVT were analyzed
as a pre-post treatment
design, approximating a covariance analysis; thus, the
interaction is the only usable summary statistic, since it is
the differential effect of treatment vs. control adjusting for
pretest differences with a regression weight of 1 (ANCOVA
with a restricted covariance weight):
Interactionij = Grand Mean – Treat effect –pretest effect
= Y… - ai.. – b.j.
Graphically, the Difference of Gain inTreat(post-pre) and Gain in Control (post –
pre)
• F for the interaction was F(l,120) = 13.69, p < .001.
• Convert this to an effect size using excel file Outcomes Computation
• What do you get? (.6527)
Coding the Outcome
Y
Gain not
“predicted” from
control
post
gains
pre
Control
Treatment
Coding the Outcome
7 For Expressive and Receptive Vocabulary, only the Ftests for Treatment-Control posttest results are given:
Receptive: F(l, 120) = 76.61, p < .001
Expressive: F(l, 120) =128.43, p< .001
What are the effect sizes? Use Outcomes Computation
1.544
1.999
Getting a Study Effect
• Should we average the outcomes to get a single study
effect or
• Keep the effects separate as different constructs to
evaluate later (Expressive, Receptive) or
• Average the PPVT and receptive outcome as a total
receptive vocabulary effect?
Comment- since each effect is based on the same sample
size, the effects here can simply be averaged. If missing
data had been involved, then we would need to use the
weighted effect size equation, weighting the effects by
their respective sample size within the study
Getting a Study Effect
 For this example, let’s average the three effects to put
into the Computing mean effects example excel filenote that since we do not have means and SDs, we can
put MeanC=0, and MeanE as the effect size we
calculated, put in the SDs as 1, and put in the correct
sample sizes to get the Hedges g, etc.
 (.6567 + 1.553 + 2.01)/3 = 1.4036
2 Justice & Ezell
 Receptive: 0.403
 Expressive: 0.8606
 Average = 0.6303
3 Coyne et al
• Taught Vocab: 0.9385
• Untaught Vocab: 0.3262
• Average = 0.6323
4 Fielding
• PPVT: -0.0764
Computing mean effect size
 Use e:\\Computing mean effects1.xls
A
1
Study
2
1
3
2
4
3
5
4
6
7
mean
8 s(mean)
B
C
Mean E
1.4036
0.6303
0.6323
0.5
Mean C
0
0
0
-0.46
D
SDE
1
1
1
1
E
SDC
1
1
1
1
F
G
d
0.65
0.63
0.63
0.02
Hedge
sg
1.40
0.61
0.62
-0.08
H
Ctrl N Trmt N
61
63
15
15
30
34
23
26
0.8054 32.25
0.1297
Mean
I
34.50
J
N
124
30
64
49
66.75
K
w
L
wd
24.87 34.91
7.16 4.39
15.20 9.49
12.20 -0.93
59.43 47.86
Computing Correlation Effect Sizes
 Reported Pearson correlation- use that
 Regression b-weight: use t-statistic reported,
e = t*(1/NE + 1/NC )½
 t-statistics: r = [ t2 / (t2 + dferror) ] ½
Sums of Squares from ANOVA or ANCOVA:
r = (R2partial) ½
R2partial = SSTreatment/Sstotal
Note: Partial ANOVA or ANCOVA results should be noted as such and compared
with unadjusted effects
Computing Correlation Effect Sizes
 To compute correlation-based effects, you can use the




excel program
“Outcomes
Computation correlations”
The next slide gives an example.
Emphasis is on disaggregating effects of unreliability
and sample-based attenuation, and correcting samplespecific bias in correlation estimation
For more information, see Hunter and Schmidt (2004):
Methods of Meta-Analysis. Sage.
Correlational meta-analyses have focused more on
validity issues for particular tests vs. treatment or
status effects using means
Computing Correlation Effects Example
STUDY#
OUTCOME#
1
2
3
4
5
6
x alpha
y alpha Ne Nc N
reliabiltiy reliabiltiy
1
0.80 0.77 47
1
0.70 0.80 33
1
0.75 0.90 22
1
0.88 0.70 111
1
0.77 0.85 34
1
0.90 0.78 47
N(r-rmean) N(rdis-rdismean)
0.229994041 0.356773
0.466932722 0.442974
2.827858564 4.92834
16.73286084 27.27103
3.469040187 5.839757
4.389591025 7.831295
76
55
45
111
34
45
r
rcorrected s(r )
123
88
67
222
68
92
0.352646
0.323444
0.190571
0.67
0.169989
0.177133
r(mean)=
rdis(mean)=
Var(rmean )=
s(rmean)=
0.394623
0.50317
0.001138
0.033739
s(emean)= 0.080498
s(edismean)=
0.351381
0.32178
0.18918
0.669165
0.168757
0.17619
0.079277
0.095995
0.118621
0.037071
0.118639
0.101539
Nr
43.21983
28.31665
12.67504
148.5545
11.47548
16.20946
disattenuate
N*(r-rmean) d r
Ndisr
-5.1631
-6.26369
-13.6715
61.13375
-15.2751
-20.0091
0.449313
0.432221
0.231956
0.853659
0.210119
0.211412
55.26549
38.03544
15.54103
189.5123
14.28811
19.44991
s(edis)
0.101008
0.128279
0.144381
0.047233
0.146647
0.12119
EFFECT SIZE DISTRIBUTION
 Hypothesis: All effects come from the same
distribution
 What does this look like for studies with different
sample sizes?
 Funnel plot- originally used to detect bias, can show
what the confidence interval around a given mean
effect size looks like
 Note: it is NOT smooth, since CI depends on both
sample sizes AND the effect size magnitude
EFFECT SIZE DISTRIBUTION
 Each mean effect SE can be computed from
SE = 1/ (w)
For our 4 effects: 1: 0.200525
2: 0.373633
3: 0.256502
4: 0.286355
These are used to construct a 95% confidence interval
around each effect
EFFECT SIZE DISTRIBUTION- SE of
Overall Mean
 Overall mean effect SE can be computed from
SE = 1/ (w)
For our effect mean of 0.8054, SE = 0.1297
Thus, a 95% CI is approximately (.54, 1.07)
The funnel plot can be constructed by constructing a SE
for each sample size pair around the overall mean- this
is how the figure below was constructed in SPSS, along
with each article effect mean and its CI
EFFECT SIZE DISTRIBUTIONStatistical test
 Hypothesis: All effects come from the same
distribution: Q-test
 Q is a chi-square statistic based on the variation of the
effects around the mean effect
Q =  wi ( g – gmean)2
k
Q 2 (k-1)
Example Computing Q Excel file
effect
d
1
0.58
5.43
0.7151598
0.397736175 no
2
-0.05
10.24
0.7326248
0.392033721 no
3
0.52
4.35
0.3957949
0.52926895 no
4
0.02
9.69
0.366319
0.545017585 no
5
-0.30
40.65
10.697349
0.001072891 yes
6
0.14
29.94
0.1686616
0.681304025 no
7
0.68
54.85
11.727452
0.000615849 yes
8
-0.02
4.00
0.2125622
0.644766516 no
0.2154
w
Qi
Q=
df
prob(Q)=
prob(Qi)
25.015924
7
0.0007539
sig?
Computational Excel file
 Open excel file: Computing Q
 Enter the effects for the 4 studies, w for each study
(you can delete the extra lines or add new ones by
inserting as needed)
 from the Computing mean effect excel file
 What Q do you get?
Q = 39.57
df=3
p<.001
Interpreting Q
 Nonsignificant Q means all effects could have come
from the same distribution with a common mean
 Significant Q means one or more effects or a linear
combination of effects came from two different (or
more) distributions
 Effect component Q-statistic gives evidence for
variation from the mean hypothesized effect
Interpreting Q- nonsignificant
 Some theorists state you should stop- incorrect.
 Homogeneity of overall distribution does not imply
homogeneity with respect to hypotheses regarding
mediators or moderators
 Example- homogeneous means correlate perfectly
with year of publication (ie. r= 1.0, p< .001)
Interpreting Q- significant
 Significance means there may be relationships with
hypothesized mediators or moderators
 Funnel plot and effect Q-statistics can give evidence
for nonconforming effects that may or may not have
characteristics you selected and coded for