Transcript META-ANALYSIS OF RESEARCH

Lecture 5
EPSY 642
Victor Willson
Fall 2009
EFFECT SIZE DISTRIBUTION
 Hypothesis: All effects come from the same
distribution
 What does this look like for studies with different
sample sizes?
 Funnel plot- originally used to detect bias, can show
what the confidence interval around a given mean
effect size looks like
 Note: it is NOT smooth, since CI depends on both
sample sizes AND the effect size magnitude
EFFECT SIZE DISTRIBUTION
 Each mean effect SE can be computed from
SE = 1/ (w)
For our 4 effects: 1: 0.200525
2: 0.373633
3: 0.256502
4: 0.286355
These are used to construct a 95% confidence interval
around each effect
EFFECT SIZE DISTRIBUTION- SE of
Overall Mean
 Overall mean effect SE can be computed from
SE = 1/ (w)
For our effect mean of 0.8054, SE = 0.1297
Thus, a 95% CI is approximately (.54, 1.07)
The funnel plot can be constructed by constructing a SE
for each sample size pair around the overall mean- this
is how the figure below was constructed in SPSS, along
with each article effect mean and its CI
EFFECT SIZE DISTRIBUTIONStatistical test
 Hypothesis: All effects come from the same
distribution: Q-test
 Q is a chi-square statistic based on the variation of the
effects around the mean effect
Q =  wi ( g – gmean)2
k
Q 2 (k-1)
Example Computing Q Excel file
effect
d
1
0.58
5.43
0.7151598
0.397736175 no
2
-0.05
10.24
0.7326248
0.392033721 no
3
0.52
4.35
0.3957949
0.52926895 no
4
0.02
9.69
0.366319
0.545017585 no
5
-0.30
40.65
10.697349
0.001072891 yes
6
0.14
29.94
0.1686616
0.681304025 no
7
0.68
54.85
11.727452
0.000615849 yes
8
-0.02
4.00
0.2125622
0.644766516 no
0.2154
w
Qi
Q=
df
prob(Q)=
prob(Qi)
25.015924
7
0.0007539
sig?
Computational Excel file
 Open excel file: Computing Q
 Enter the effects for the 4 studies, w for each study
(you can delete the extra lines or add new ones by
inserting as needed)
 from the Computing mean effect excel file
 What Q do you get?
Q = 39.57
df=3
p<.001
Interpreting Q
 Nonsignificant Q means all effects could have come
from the same distribution with a common mean
 Significant Q means one or more effects or a linear
combination of effects came from two different (or
more) distributions
 Effect component Q-statistic gives evidence for
variation from the mean hypothesized effect
Interpreting Q- nonsignificant
 Some theorists state you should stop- incorrect.
 Homogeneity of overall distribution does not imply
homogeneity with respect to hypotheses regarding
mediators or moderators
 Example- homogeneous means correlate perfectly
with year of publication (ie. r= 1.0, p< .001)
Interpreting Q- significant
 Significance means there may be relationships with
hypothesized mediators or moderators
 Funnel plot and effect Q-statistics can give evidence
for nonconforming effects that may or may not have
characteristics you selected and coded for
MEDIATORS
 Mediation: effect of an intervening variable that
changes the relationship between an independent and
dependent variable, either removing it or (typically)
reducing it.
 Path model conceptualization:
Treatment
Outcome
Mediator
MEDIATORS
 Statistical treatment typically requires both paths ‘a’
and ‘b’ to be significant to qualify as a mediator. Metaanalysis seems not to have investigated path ‘a’ but
referred to continuous predictors as regressors
 Lipsey and Wilson(2001) refer to this as “Weighted
Regression Analysis”
Treatment
a
Outcome
Mediator
b
Weighted Regression Analysis
 Model: e = b X + residual
 Regression analog: Q = Qregression + Qresidual
 Analyze as “weighted least squares” in programs such
as SPSS or SAS
 In SPSS the weight function w is a variable used as the
weighting
Weighted Regression Analysis
 Emphasis on predictor and its standard error: the
usual regression standard error is incorrect, needs to
be corrected (Hedges & Olkin, 1985):
SE’b = SEb / (MSe)½
where SEb is the standard error reported in SPSS,
and MSe is the reported regression mean square error
Weighted Regression Q-statistics
 Qregression = Sum of Squaresregression
df = 1 for single predictor
 Qresidual = Sum of Squaresresidual
df = # studies - 2
Significance tests: Each is a chi square test with
appropriate degrees of freedom
9
7
8
8
9
6
7
7
8
9
5
5
7
6
7
8
9
7
7
8
8
9
9
6
8.99
12.8
9.09
10.86
7.73
10.11
8.57
9.59
7.98
12.69
8.61
9.34
10.39
8.66
9.16
8.18
10.04
12.33
8.83
10.88
9.5
10.42
11.82
11.69
9.05
10.03
11.56
10.52
7.86
10.77
6.91
8.53
10.92
8.16
10.57
7.43
10.1
9.4
9.04
6.43
9.84
11.4
10.67
8.81
8.09
10.12
7.13
8.11
31
24
30
25
22
24
34
22
30
28
28
24
26
21
25
30
25
22
23
26
28
18
28
23
19
26
20
25
28
26
16
28
20
22
22
26
24
29
25
20
25
28
27
24
22
32
22
27
1.7026
0.267
0.7561
0.6532
1.5414
0.3507
0.4438
1.1245
0.542
0.6337
-0.5976
0.3771
0.7234
0.2413
0.6637
0.9038
0.4603
0.3948
-0.1726
0.4633
0.8481
0.7114
0.5407
0.4926
2
2
2
1
2
1
1
2
1
1
2
1
2
1
1
2
1
1
2
1
2
1
1
1
8.781319
12.369946
11.22962
11.867084
9.530347
12.291338
10.651743
10.659409
11.591384
11.739213
11.80079
12.262378
11.714913
12.094229
11.847643
10.928737
12.177485
12.087879
12.374215
12.154409
11.317137
10.885366
11.891682
12.05664
SPSS ANALYSIS OUTPUT
ANOVAb,c
Model Sum of Squares df
Mean Square
F
Regression
19.166 1
19.166
12.096 .002a
Residual
Total
34.858 22
54.024 23
1.584
Sig.
a. Predictors: (Constant), AGE
b. Dependent Variable: HEDGE d*
c. Weighted Least Squares Regression - Weighted by w
Coefficientsa,b
Model
Unstandardized Coefficients Standardized Coefficients t
B
Std. Error
Beta
(Constant)
-1.037 .465
-2.230
AGE
.215
.062
.596
3.478
a. Dependent Variable: HEDGE d*
b. Weighted Least Squares Regression - Weighted by w
Sig.
.036
.002
Example
 See SPSS “sample meta data set.sav” or the excel
version “sample meta data set regression”
 The d effect is regressed on Age
 b = 0.215, SEb = 0.062, MSe = 1.584
 Thus, SE’b = 0.062 / (1.584)½
= 0.0493
A 95% CI around b gives (0.117, 0.313) for the regression
weight of age on outcome, p<.001
Q-statistic tests
 Qregression = 19.166 with df=1, p < .001
 Qresidual = 34.858 with df=22, p = .040
 So- are the residuals homogeneous or not? Given a
large number of significance tests, one might require
the Type I error rate for such tests to be .001 or
something small