Transcript Effect Size Calculation for Meta

Effect Size Calculation for
Meta-Analysis
Robert M. Bernard
Centre for the Study of Learning and Performance
Concordia University
February 24, 2010
Main Purposes of a
Meta-Analysis
A meta-analysis attempts to …
7/21/2015
2
What is an Effect size?
An effect size is a …
7/21/2015
3
Types of Effect Sizes
Most reviews use …
• d-family of effect sizes, including the standardized
mean difference, or
• r-family of effect sizes, including the correlation
coefficient, or
• the odds ratio (OR) family of effect sizes,
including proportions and other measures for
categorical data.
7/21/2015
4
Effect Size Extraction
Effect size (ES) extraction involves …
• Locating descriptive or other statistical information
contained in studies.
• Converting statistical information into a standard
metric (effect size) by which studies can be compared
and/or combined.
7/21/2015
5
Choice of an Effect Size
When we have…
• continuous univariate data for two groups, we typically
compute a raw mean difference or a standardized
difference – an effect size from the d-family,
• continuous bivariate data, we typically compute a
correlation (from the r-family), or
• binary data (the patient lived or died, the student passed
or failed), we typically compute an odds ratio, a risk
ratio, or a risk difference.
7/21/2015
6
d-Family:
Zero Effect Size
ES = 0.00
Control
Condition
Treatment
Condition
Overlapping
Distributions
7/21/2015
7
d-Family:
Moderate Effect Size
ES = 0.40
Control
Condition
7/21/2015
Treatment
Condition
8
d-Family:
Large Effect Size
ES = 0.85
Control
Condition
7/21/2015
Treatment
Condition
9
Cohen’s (1988) Qualitative
Descriptors
Effect Size Interpretation
7/21/2015
10
Research designs for
d-Family Statistics
Independent Groups (posttest-only)
EXP
YPost
(Randomized or Non-randomized)
CT
YPost
YPost
One-group (pretest-posttest)
YPre
EXP
Independent Groups (pre-post)
YPre
EXP YPost
(Randomized or Non-randomized)
YPre
CT
EXP = Experimental Condition
7/21/2015
YPost
CT = Control Condition
11
Statistics for d-Family Effect
Size Extraction
Effect sizes can be extracted using the
following reported statistics:
• Descriptive statistics (means, SDs, sample sizes)
Preferred (by far).
• Exact test statistics (t-values, F-values, etc.)
• Exact probability values (p = .013, etc.)
• Approximate comparisons of p to α (p < .05, etc.)
By far, the least exact.
7/21/2015
12
d-Family with Independent
Groups (Basic Equation)
Glass 
dCohen 
SD pooled 
YExperimental  YControl
SDControl
YExperimental  YControl
SDPooled
(SD 2 E (nE  1))  (SD 2 C (nC  1))
NTotal  2
Note: this equation is the same as adding two SSs and dividing by dfTotal
7/21/2015
13
d Family Statistics:
Means and Standard Deviations
Study reports:
Treatment (n = 31)
Mean = 42.8
SD = 16.31
Control (n = 31)
Mean = 32.5
SD = 14.17
Procedure:
7/21/2015
1) Calculate Pooled SD
2) Calculate d
Alternative Methods of ES
Extraction: t-values and F-ratios
Study Reports: t(60) = +2.66
Study Reports: F(1, 61) = 7.076
Important Note: Report must indicate direction of the effect (+/–)
7/21/2015
15
Alternative Methods of ES
Extraction: Exact p-value
Study Reports: t(60) is sig. p = .01
Look up t-value for p = .01 (df = 60)
t = 2.66
Important Note: Report must indicate direction of the effect (+/–)
7/21/2015
Alternative Methods of ES
Extraction: p < α
Study Reports: p < .05, nT = 31, nC = 31
Important Note: Report must indicate direction of the effect (+/–)
Estimate +t(60) = +2.00
Compared with 0.676, this ES is only 75% accurate.
7/21/2015
d Family: Adjustment for
Small Samples
Cohen’s d tends to overestimate ES in small samples.
N = 60, g is 99% of d
N = 40, g is 98% of d
N = 20, g is 96% of d
N = 10, g is 90% of d
Recommendation: If there are small samples and large samples,
convert all d-family statistics to g.
7/21/2015
18
d-Family Statistics with
dependent Groups (pre-post)
Study Reports:
Pretest
Mean = 7.5
SD = 4.81
Posttest
Mean = 8.5
SD = 4.67
Study Reports:
Change
7/21/2015
r (pre/post) = 0.80
SD = 2.98
19
Relationship Between Effect
Size and Pre-Post Correlation
1.00
SD Change: d = 0.21, using r =
0.80
0.90
0.80
Correlation
0.70
0.60
0.50
0.40
0.30
Means and SDs: d = 0.21
0.20
0.10
0.00
0.00
7/21/2015
0.05
0.10
0.15
0.20
0.25
Effect Size
0.30
0.35
0.40
0.45
0.50
20
d-Family Statistics with
Independent Groups (pre-post)
Study Reports:
DT (post – pre)
Mean = 7.5
SD = 4.80
DC (post – pre)
Mean = 8.5
SD = 4.70
Calculate the pooled SD.
7/21/2015
21
Calculating Standard Error
The standard error of g is an estimate of the “standard deviation” of
the population, based on the sampling distribution of an infinite number
of samples all with a given sample size. Smaller samples tend to have
larger standard errors and larger samples have smaller standard errors.
Standard Error: ˆ g 
ˆ g 


1 1
g2
3
 
1

ne nc 2(ne  nc ) 
4(ne  nc )  9 
1
1
0.675 2 
3



1

31 31 2(31  31) 
4(31  31)  9 
ˆ g  0.076  1  0.0126 
ˆ g  0.266)  (0.987 
7/21/2015
ˆ g  0.263
22
95th% Confidence Interval
The 95th Confidence Interval is the range within which it can be stated
with reasonable confidence that the true population mean exists. As the
standard error decreases (the sample size increases), the confidence
interval decreases in width.
95th Confidence CIUL  g  (1.96  ˆ i )
Interval
Upper:
CIU  0.687  (1.96  0.26)
CIU  1.97
Lower:
CI L  0.687  (1.96  0.26)
CI L  0.177
7/21/2015
Conclusion: Confidence interval does not cross 0
(g falls within the 95th confidence interval).
23
Forest Plot
7/21/2015
24
Other Important Statistics
Variance:
ˆ 2g  (ˆ g )2
ˆ 2g  (0.262)2
The variance is the standard
error squared.
ˆ 2g  0.069
Inverse Variance (w): wi  1 ˆ 2
wi  1 0.069
wi  14.54
Weighted g (g*w):
The inverse variance (w) provides
a weight that is proportional to
the sample size. Larger samples
are more heavily weighted than
small samples.
Weightedg  (wi )(gi )  14.54  0.687  9.99
Weighted g is the weight (w) times the value of g. It can be + or –, depending on the sign of g.
7/21/2015
25
HedgesÕ
g
2.44
2.31
1.38
1.17
0.88
0.81
0.80
0.68
0.63
0.60
0.58
0.32
0.25
0.24
0.24
0.19
0.11
0.09
0.02
0.02
0.02
-0.11
-0.11
-0.18
-0.30
0.330
Standard
Variance
Error
( ˆ 2g )
( ˆ g )
95th
Lower
Limit
95th Upper
z-Value
Limit
p-Value
Weights
( wi )
Weighted
g
(wi )(gi )
19.94
34.60
11.11
27.70
34.60
69.44
156.25
30.86
3.84
59.17
11.89
82.64
156.25
25.00
44.44
69.44
69.44
156.25
17.36
34.60
14.79
17.36
12.76
20.66
277.78
48.65
79.93
15.33
32.41
30.45
56.25
125.00
20.99
2.42
35.50
6.90
26.45
39.06
6.00
10.67
13.19
7.64
14.06
0.35
0.69
0.30
-1.91
-1.40
-3.72
-83.33
0.22
0.17
0.30
0.19
0.17
0.12
0.08
0.18
0.51
0.13
0.29
0.11
0.08
0.20
0.15
0.12
0.12
0.08
0.24
0.17
0.26
0.24
0.28
0.22
0.06
0.05
0.03
0.09
0.04
0.03
0.01
0.01
0.03
0.26
0.02
0.08
0.01
0.01
0.04
0.02
0.01
0.01
0.01
0.06
0.03
0.07
0.06
0.08
0.05
0.00
2.00
1.98
0.79
0.80
0.55
0.57
0.64
0.33
-0.37
0.35
0.01
0.10
0.09
-0.15
-0.05
-0.05
-0.13
-0.07
-0.45
-0.31
-0.49
-0.58
-0.66
-0.61
-0.42
2.88
2.64
1.97
1.54
1.21
1.05
0.96
1.03
1.63
0.85
1.15
0.54
0.41
0.63
0.53
0.43
0.35
0.25
0.49
0.35
0.53
0.36
0.44
0.25
-0.18
10.89
13.59
4.60
6.16
5.18
6.75
10.00
3.78
1.24
4.62
2.00
2.91
3.13
1.20
1.60
1.58
0.92
1.13
0.08
0.12
0.08
-0.46
-0.39
-0.82
-5.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.22
0.00
0.05
0.00
0.00
0.23
0.11
0.11
0.36
0.26
0.93
0.91
0.94
0.65
0.69
0.41
0.00
0.03
0.00
0.28
0.38
12.62
0.00
7/21/2015
1458.21* 481.87*
Average g (g+) is the
sum of the weights
divided by the sum
of the weighted gs.
w

g 
 (w )(g )
i
i
481.87
g 
1458.21
g  0.333
26
i
Selected References
Borenstein, M. Hedges, L.V., Higgins, J.P..,&
Rothstein, H.R. (2009). Introduction to metaanalysis. Chichester, UK: Wiley.
Glass, G. V., McGaw, B., & Smith, M. L. (1981). Metaanalysis in social research. Beverly Hills, CA: Sage.
Hedges, L. V., & Olkin, I. (1985). Statistical methods for
meta-analysis. Orlando, FL: Academic Press.
Hedges, L. V., Shymansky, J. A., & Woodworth, G.
(1989). A practical guide to modern methods of
meta-analysis. [ERIC Document Reproduction
Service No. ED 309 952].
7/21/2015
27