Transcript Meta-Analysis

Conducting Meta-Analyses
Marsha Sargeant, M.S.
Design And Statistical Analysis Laboratory
University of Maryland, College Park
Department of Psychology
Overview of Presentation
1.
2.
3.
4.
2
What is a meta-analysis and why is it
important?
Overview of procedures involved in
conducting a quantitative meta-analysis
Database structure
Interpretation of effect sizes
Meta-analysis Definition


A statistical analysis of the summary findings
of many empirical studies
It’s quantitative!
–
3
Distinct from a meta-review
Background
Empirical findings grew exponentially in
the middle 50 years of the 20th century

–
–
4
Multiplied beyond our ability to comprehend
and integrate it
Hence a growing need to statistically and
technically review, rather than through narrative
Background


5
Review of practices and methods of
research reviewers and synthesizers in the
social sciences (Jackson, 1978)
Failure to report methods of reviewing
Benefits of Meta-analyses




6
Increased statistical power
Identification of sources of variability across
studies (e.g., inclusion of moderators)
Detection of biases (e.g., Tower of Babel
bias)
Detection of deficiencies in design, analysis,
or interpretation
Ioannidis & Lau, 1999
Limitations of Meta-analyses



Cannot improve the original studies
Method is frequently misapplied
Can never follow the rules of science
–
7
Sources of bias are not controlled
Ioannidis & Lau, 1999
Rules of the Game
It is quantitative
There is no arbitrary exclusion of data
File drawer effect



–
–
Meta-analysis seeks general conclusions

–
8
Dissertation research is research too!
Unpublished studies
It is contradictory to think that we can only compare
studies that are the same (if they were the same you
wouldn’t need to compare them!)
Glass, 2000
Methodological Adequacy of
Research Base



9
Findings must be interpreted within the
bounds of the methodological quality of the
research base synthesized.
Studies often cannot simply be grouped into
“good” and “bad” studies.
Some methodological weaknesses may bias
the overall findings, others may merely add
“noise” to the distribution.
From “Practical Meta-analysis” by D.B. Wilson
Confounding of Study Features


10
Important study features are often
confounding, obscuring the interpretive
meaning of observed differences
If the confounding is not severe and you
have a sufficient number of studies, you can
model “out” the influence of method features
to clarify substantive differences
From “Practical Meta-analysis” by D.B. Wilson
Meta-analysis Overview
Descriptives

–
–
Method section

–
–
–
–
–
11
Effect sizes (e.g., correlation coefficients)
Distribution and central tendency summarized
Databases searched
Journals
What attempts were made to not have a biased search?
Criteria for inclusion
No effect studies
Rosenthal, 2005
Meta-analysis Overview

Study quality
–
–
–
–
12
Use a weighting system
Use raters and non-dichotomous ratings to
avoid weighter bias
Optimally raters should be blind to the results of
the study
Ratings can be used as an adjustment on effect
size or as a moderator to determine whether
quality is related to obtained effect size
Rosenthal, 2005
Meta-analysis Overview
Consider independence of studies

–
Recorded variables

–
–
–
–
–
–
–
13
Treat non-independent studies as a single study with
different dependent variables
Number, Age, Sex, Education, etc
Volunteer status
Laboratory or field study?
Randomized?
Method of data collection (e.g., interview vs questionnaire)
How constructs are operationalized
etc.
Rosenthal, 2005
Meta-analysis Overview
Summarize recorded variables
Study characteristics could all be potential
moderators of outcome aside from those with
particular meaning for the specific area of research
Effect sizes (there are others)



–
–
–
R
Zr (Fisher’s r-Z transformation)
d family



14
Cohen’s d
Hedge’s g
Glass’s delta
Rosenthal, 2005
Examples of Different Types of Effect
Sizes

Standardized mean difference
–
Group contrast research


–

Inherently continuous construct
Odds-ratio
–
Group contrast research


–

Treatment groups
Naturally occurring groups
Inherently dichotomous construct
Correlation coefficient
–
15
Treatment groups
Naturally occurring groups
Association between variables research
From “Practical Meta-analysis - The Effect Size” by D.B. Wilson
Interpreting Effect Size Results

Cohen’s “Rules-of-Thumb”
–
standardized mean difference effect size



–
correlation coefficient



–
small = 0.10
medium = 0.25
large = 0.40
odds-ratio



16
small = 0.20
medium = 0.50
large = 0.80
small = 1.50
medium = 2.50
large = 4.30
From “Practical Meta-analysis” by D.B. Wilson
Interpreting Effect Size Results

Rules-of-Thumb do not take into account the
context of the intervention
–
–
–
17
a “small” effect may be highly meaningful for an
intervention that requires few resources and
imposes little on the participants
a small effect may be meaningful if the
intervention is delivered to an entire population
(prevention programs for school children)
small effects may be more meaningful for serious
and fairly intractable problems
From “Practical Meta-analysis” by D.B. Wilson
Meta-analysis Overview

Significance levels recorded
–
Recorded as the one-tailed standard normal
deviates associated with p’s

18
E.g., p’s of .10, .01., .001 would be recorded as Z’s of
1.28, 2.33, and 3.09
Meta-analysis Overview
Report central tendency

–
–
–
–
–
–
–
19
Unwieghted mean effect size
Weighted mean effect size (weighting by size of study –
can also use quality or other characteristic of interest)
Median
Proportion of studies showing effect sizes in the expected
direction
Report number of studies reported on
Optional: total number of participants on which the
weighted mean is based
Optional: median number of participants per obtained
effect size
Meta-analysis Overview

Report variability
–
–
–
20
Standard deviation
Max and min effect size found at the 75th and
25th percentile
If normally distributed, the standard deviation is
estimated at .75(Q3-Q1)
Database Structure

Database structures
–
–
–
–
–
21
The hierarchical nature of meta-analytic data
The familiar flat data file
The relational data file
Advantages and disadvantages of each
What about the meta-analysis bibliography?
From “Practical Meta-analysis – Database Structure” by D.B. Wilson
Database Structure



22
Meta-analytic data is inherently hierarchical
Any specific analysis can only include one
effect size per study (or one effect size per
sub-sample within a study)
Analyses almost always are of a subset of
coded effect sizes. Data structure needs to
allow for the selection and creation of those
subsets
From “Practical Meta-analysis – Database Structure” by D.B. Wilson
Example of a Flat Data File
Multiple ESs handled by having multiple
variables, one for each potential ES.
ID
22
23
31
36
40
82
185
186
204
229
246
274
295
626
1366
Paradigm
2
2
1
2
1
1
1
1
2
2
2
2
2
1
2
ES1
0.77
0.77
-0.1
0.94
0.96
0.29
0.65
DV1
3
3
5
3
11
11
5
0.97
3
0.86
7.03
0.87
3
3
3
ES2
DV2
-0.05
5
0.58
0.83
0.88
5
5
3
0.91
-0.31
6.46
-0.04
0.5
3
3
3.
3
3
ES3
DV3
0.48
5
0.79
3
3
3
0.1
Note that there is only one record (row) per study
23
From “Practical Meta-analysis – Database Structure” by D.B. Wilson
ES4
DV4
-0.2
11
0.068
5
1.17
0.57 .
0.9
3
3
Database Structure
Advantages and Disadvantages of a Single Flat File Structure
 Advantages
–
–
–

Disadvantages
–
–

Only a limited number of ESs can be calculated per study
Any adjustments applied to ESs must be done repeatedly
When to use
–
–
–
24
All data is stored in a single location
Familiar and easy to work with
No manipulation of data files prior to analysis
Interested in a small predetermined set of ESs
Number of coded variables is modest
Comfort level with a multiple data file structure is low
From “Practical Meta-analysis – Database Structure” by D.B. Wilson
Example of Relational Data Structure
(Multiple Related Flat Files)
Database Structure
Study Level Data File
ID
100
7049
PubYear
92
82
MeanAge
15.5
14.5
TxStyle
2
1
Effect Size Level Data File
Note that a single record
in the file above is
“related” to five records
in the file to the right
25
ID
100
100
100
100
100
7049
7049
7049
ESNum
1
2
3
4
5
1
2
3
Outcome
Type
1
1
1
1
1
2
4
1
TxN
24
24
24
24
24
30
30
30
From “Practical Meta-analysis – Database Structure” by D.B. Wilson
CgN
24
24
24
24
24
30
30
30
ES
-0.39
0
0.09
-1.05
-0.44
0.34
0.78
0
Example of a More Complex Multiple
File Data Structure
Database Structure
Study Level Data File
ID
100
7049
PubYear
92
82
MeanAge
15.5
14.5
Outcome Level Data File
ID
100
100
100
7049
7049
TxStyle
2
1
OutNum Constrct
1
2
2
6
3
4
1
2
2
6
Scale
1
1
2
4
3
Effect Size Level Data File
ID
100
100
100
100
100
100
7049
7049
7049
26
OutNum
1
1
2
2
3
3
1
1
2
ESNum
1
2
3
4
5
6
2
6
2
Months
0
6
0
6
0
6
0
12
0
TxN
24
22
24
22
24
22
30
29
30
CgN
ES
24 -0.39
22
0
24 0.09
22 -1.05
24 -0.44
21 0.34
30 0.78
28 0.78
30
0
Note that study 100 has 3 records
in the outcomes data file and 6
outcomes in the effect size data
file, 2 for each outcome measured
at different points in time (Months)
From “Practical Meta-analysis – Database Structure” by D.B. Wilson
Database Structure
Advantages & Disadvantages of Multiple Flat Files Data Structure
 Advantages
–
–
–
–

Disadvantages
–
–
–

Complex to implement
Data must be manipulated prior to analysis (creation of “working”
analysis files)
Must be able to select a single ES per study for any given analysis
When to use
–
27
Can “grow” to any number of ESs
Reduces coding task (faster coding)
Simplifies data cleanup
Smaller data files to manipulate
Large number of ESs per study are possible
From “Practical Meta-analysis – Database Structure” by D.B. Wilson
What about Sub-Samples?


So far I have assumed that the only ESs that have
been coded were based on the full study sample
What if you are interested in coding ESs separately
for different sub-samples, such as, by gender or SES
–
Just say “no”!


–
Well, if you must, plan your data structure carefully


28
Often not enough of such data for meaningful analysis
Complicates coding and data structure
Include a full sample effect size for each dependent measure
of interest
Place sub-sample in a separate data file
From “Practical Meta-analysis – Database Structure” by D.B. Wilson
Tips on Coding

Paper Coding
–
–

29
include data file variable names on coding form
all data along left or right margin eases data entry
Coding Directly into a Computer Database
From “Practical Meta-analysis – Database Structure” by D.B. Wilson
Example Screen from a Computerized
Database for Direct Coding
30
Figure 5.11: Example FileMaker Pro Screen for Data Entry from the Challenge
Coding Directly into a Computer
Database

Advantages
–
–
–
–

Avoids additional step of transferring data from paper to
computer
Easy access to data for data cleanup
Data base can perform calculations during coding process
(e.g., calculation of effect sizes)
Faster coding
Disadvantages
–
Can be time consuming to set up

–
31
the bigger the meta-analysis the bigger the payoff
Requires a higher level of computer skill
From “Practical Meta-analysis – Database Structure” by D.B. Wilson
Final Comments

Meta-analysis
–
–
–
–
32
is a replicable and defensible method of
synthesizing findings across studies
often points out gaps in the research literature,
providing a solid foundation for the next
generation of research on that topic
illustrates the importance of replication
facilitates generalization of the knowledge gain
through individual evaluations
From “Practical Meta-analysis” by D.B. Wilson
Thank You!
Email: [email protected]
Web: www.umd.academia.edu/MarshaSargeant
33