Transcript Meta-analysis

Meta-analysis
The EBM workshop
A.A.Haghdoost, MD; PhD of Epidemiology
[email protected]
Definition
Meta-analysis: a type of systemic review
that uses statistical techniques to
quantitatively combine and summarize
results of previous research
A review of literature is a meta-analytic
review only if it includes quantitative
estimation of the magnitude of the effect
and its uncertainty (confidence limits).
EBM workshop
Meta analysis
A.A.Haghdoost
Function of Meta-Analysis(1)
1-Identify heterogeneity in effects among
multiple studies and, where appropriate,
provide summary measure
2-Increase statistical power and precision
to detect an effect
3-Develop ,refine, and test hypothesis
continued
EBM workshop
Meta analysis
A.A.Haghdoost
Function of Meta-Analysis(2)
continuation
4-Reduce the subjectivity of study
comparisons by using systematic and
explicit comparison procedure
5-Identify data gap in the knowledge base
and suggest direction for future research
6-Calculate sample size for future studies
EBM workshop
Meta analysis
A.A.Haghdoost
Historical background
Ideas behind meta-analysis predate Glass’ work
by several decades
– R. A. Fisher (1944)
• “When a number of quite independent tests of significance have
been made, it sometimes happens that although few or none can
be claimed individually as significant, yet the aggregate gives an
impression that the probabilities are on the whole lower than
would often have been obtained by chance” (p. 99).
• Source of the idea of cumulating probability values
– W. G. Cochran (1953)
• Discusses a method of averaging means across independent
studies
• Laid-out much of the statistical foundation that modern metaanalysis is built upon (e.g., inverse variance weighting and
homogeneity testing)
EBM workshop
Meta analysis
A.A.Haghdoost
Basic concepts
The main outcome is the overall magnitude
of the effect.
It's not a simple average of the magnitude
in all the studies.
Meta-analysis gives more weight to studies
with more precise estimates.
– The weighting factor is 1/(standard error)2.
EBM workshop
Meta analysis
A.A.Haghdoost
Main magnitude of effects
Descriptive
Mean
Prevalence
Analytical
Additive
Mean difference
Standardized mean difference
Risk, rate or hazard difference
Correlation coefficient
Multiplicative
Odds ratio, Risk, Rate or Hazard Ratio
EBM workshop
Meta analysis
A.A.Haghdoost
Statistical concepts(1)
You can combine effects from different
studies only when they are expressed in
the same units.
Meta-analysis uses the magnitude of the
effect and its precision from each study to
produce a weighted mean.
EBM workshop
Meta analysis
A.A.Haghdoost
Statistical concepts(2)
The impact of fish oil consumption on Cardio-vascular diseases
EBM workshop
Meta analysis
A.A.Haghdoost
Forest plot
the graphical display of results from
individual studies on a common scale is a
“Forest plot”.
In the forest plot each study is represented
by a black square and a horizontal line
(CI:95%).The area of the black square
reflects the weight of the study in the
meta-analysis.
A logarithmic scale should be used for
plotting the Relative Risk.
EBM workshop
Meta analysis
A.A.Haghdoost
Forest plot
EBM workshop
Meta analysis
A.A.Haghdoost
Statistical concepts(3)
There are two basic approach to
Quantitative meta –analysis:
Weighted-sum
Fixed effect model
Random effect model
Meta-regression model
EBM workshop
Meta analysis
A.A.Haghdoost
Fixed effect model
General Fixed effect model- the inverse
variance – weighted method
Specific methods for combining odds ratio
Mantel- Haenszel method
Peto’s method
Maximum-Likelihood techniques
Exact methods of interval estimation
EBM workshop
Meta analysis
A.A.Haghdoost
Fixed effect model
In this model, all of the observed difference
between the studies is due to chance
Observed study effect=Fixed effect+ error
Xi= θ + ei
ei is N (0,δ2 )
Xi = Observed study effect
θ = Fixed effect common to all studies
EBM workshop
Meta analysis
A.A.Haghdoost
General Fixed effect model
Ť=∑ wiTi/ ∑ wi
The weights that minimize the variance of Ť
are inversely proportional to the
conditional variance in each study
Wi=1/vi
Var(Ť)=1/ ∑ wi
EBM workshop
Meta analysis
A.A.Haghdoost
Mantel- Haenszel method
Each study is considered a strata.
Ť=∑ai di / ni / ∑bi ci /ni
EBM workshop
Meta analysis
A.A.Haghdoost
Random effect model
The “random effect” model, assumes a
different underlying effect for each study.
This model leads to relatively more weight
being given to smaller studies and to
wider confidence intervals than the fixed
effects models.
The use of this model has been advocated
if there is heterogeneity between study
results.
EBM workshop
Meta analysis
A.A.Haghdoost
Source of heterogeneity
Results of studies of similar interventions usually
differ to some degree.
Differences may be due to:
- inadequate sample size
- different study design
- different treatment protocols
- different patient follow-up
- different statistical analysis
- different reporting
- different patient response
EBM workshop
Meta analysis
A.A.Haghdoost
An important controversy has arisen over
whether the primary objective a metaanalysis should be the estimation of an
overall summary or average effect across
studies (a synthetic goal)
or the identification and estimation of
differences among study-specific effects
(analytic goal)
EBM workshop
Meta analysis
A.A.Haghdoost
Test of Homogeneity
This is a test that observed scatter of study
outcomes is consistent with all of them
estimating the same underlying effect.
Q= X2homo=∑i=1nwi (mi -M)2
df=n-1
wi =weight
M=meta analytic estimate of effect
mi =effect measure of each study
EBM workshop
Meta analysis
A.A.Haghdoost
Dealing with statistical heterogeneity
The studies must be examined closely to
see if the reason for their wide variation in
effect. If it’s found the analysis can be
stratified by that factor.
Subgroup analysis
Exclusion of study
Choose another scale
Random effect model
Meta-regression
EBM workshop
Meta analysis
A.A.Haghdoost
Random effect model
Assume there are two component of
variability:
1)Due to inherent differences of the effect
being sought in the studies (e.g. different
design, different populations, different
treatments, different adjustments ,etc.)
(Between study)
2)Due to sampling error (Within study)
EBM workshop
Meta analysis
A.A.Haghdoost
Random effect model
There are two separable effects that can be
measured
The effect that each study is estimating
The common effect that all studies are
estimating
Observed study effect=study specific (random )effect +
error
EBM workshop
Meta analysis
A.A.Haghdoost
Random effect model
This model assumes that the study specific
effect sizes come from a random distribution
of effect sizes with a fixed mean and
variance.
There are five approach for this model:
Weighted least squares
Un-weighted least squares
Maximum likelihood
Restricted Maximum likelihood
Exact approach to random effects of binary data.
EBM workshop
Meta analysis
A.A.Haghdoost
Random effect
Xi= θi + ei
ei is N (0,δ2 )
Xi = Observed study effect
θi = Random effect specific to each study
θi =U+di
U=Grand mean (common effect)
di is N (0, ‫ד‬2 ) – Random term
EBM workshop
Meta analysis
A.A.Haghdoost
Weighted least squares for Random Effect
Ŵ=∑wi/k
S2w=1/k-1(∑wi2-k Ŵ2)
U=(k-1)(Ŵ-S2w/kŴ)
‫ד‬2=0 if Q<k-1
‫ד‬2=(Q-(k-1))/U if Q>k-1
wi* = 1/var.+ ‫ד‬2 var.=within study variances
EBM workshop
Meta analysis
A.A.Haghdoost
Weighted least squares for Random Effect
(WLS)
Ť.RND=∑ wi* Ti/ ∑ wi*
Var(Ť.RND)=1/ ∑ wi*
Where Ti is an estimate of effect size and θi
is the true effect size in the ith study
Ti = θi +ei
ei is the error with which Ti
estimates θi
var(Ti)= ‫ד‬θ2 +vi
EBM workshop
Meta analysis
A.A.Haghdoost
random versus fixed effect models
Neither fixed nor random effect analysis
can be considered ideal.
Random effect models has been criticized
on grounds that unrealistic distributional
assumption have to be made.
Random effect models are consistent with
the specific aims of generalization.
EBM workshop
Meta analysis
A.A.Haghdoost
Peto’s advocates
He suggested a critical value .01 instead of
usual .05 to decide whether a treatment
effect is statistically significant for a fixed
effect model.
This more conservative approach has the
effect of reducing the differences between
fixed and random effect models.
EBM workshop
Meta analysis
A.A.Haghdoost
Meta-regression
If more than two groups of studies have
been formed and the characteristic used
for grouping is ordered, greater power to
identify sources of heterogeneity may be
obtained by regressing study results on
the characteristic .
With meta-regression, it is not necessary or
even desirable to groups the studies.
The individual study results can be entered
directly in the analysis.
EBM workshop
Meta analysis
A.A.Haghdoost
Meta-Regresion
1- meta-Regression model( extension of
fixed effect model)
2- Mixed model( extension of random effect
model)
EBM workshop
Meta analysis
A.A.Haghdoost
Fixed-effects regression
Θi=B0+B1xi1+...+Bpxip
It’s the covariate predictor variables that
are responsible for the variation not a
random effect; the variation is
predictable, not random.
EBM workshop
Meta analysis
A.A.Haghdoost
Mixed model
Θi=B0+B1xi1+...+Bpxip+ui
This model assumes that part of the
variability in true effects is unexplainable
by the model.
EBM workshop
Meta analysis
A.A.Haghdoost
Between studies variation
You can and should allow for real
differences between studies–
heterogeneity–in the magnitude of the
effect.
– The τ2 statistic quantifies % of variation due
to real differences.
EBM workshop
Meta analysis
A.A.Haghdoost
Fixed effects model and heterogeneity
In fixed-effects meta-analysis, you do so by
testing for heterogeneity using the Q
statistic.
If p<0.10, you exclude "outlier" studies and
re-test, until p>0.10.
When p>0.10, you declare the effect
homogeneous.
But the approach is unrealistic, limited, and
suffers from all the problems of statistical
significance.
EBM workshop
Meta analysis
A.A.Haghdoost
Random effects model and
heterogeneity
In random-effect meta-analysis, you assume there
are real differences between all studies in the
magnitude of the effect.
The "random effect" is the standard deviation
representing the variation in the true magnitude
from study to study.
You need more studies than for traditional metaanalysis.
The analysis is not available in a spreadsheet.
EBM workshop
Meta analysis
A.A.Haghdoost
Concept of analysis in random versus
fixed effect models
Fixed effects models: within-study variability
– "Did the treatment produce benefit on average
in the studies at hand?"
Random effects models: between-study and
within-study variability
– "Will the treatment produce benefit ‘on
average’?"
EBM workshop
Meta analysis
A.A.Haghdoost
Limitations
It's focused on mean effects and
differences between studies. But what
really matters is effects on individuals.
(Aggression bias)
A meta-analysis reflects only what's
published or searchable.
EBM workshop
Meta analysis
A.A.Haghdoost
Aggregation bias
Relation between group rates or and means
may not resemble the relation between
individual values of exposure and
outcome.
This phenomenon is known as aggregation
bias or ecologic bias.
EBM workshop
Meta analysis
A.A.Haghdoost
BP
Ecological fallacy
Education
EBM workshop
Meta analysis
A.A.Haghdoost
Meta-analysis of neoadjuvant chemotherapy for
cervical cancer
Hand Searching
14%
Word of Mouth
14%
Trial Registers
14%
Medline/Cancerlit
58%
EBM workshop
Meta analysis
A.A.Haghdoost
Type of reporting
Unpublished
24%
Ongoing
5%
Published in full
47%
Published as abstract
24%
EBM workshop
Meta analysis
A.A.Haghdoost
Selection bias in Meta analysis
English language bias
Database bias
Publication bias
Bias in reporting of data
Citation bias
Multiple publication bias
Sample size
EBM workshop
Meta analysis
A.A.Haghdoost
Publication bias
The results of a meta-analysis may be
biased if the included studies are a biased
sample of studies in general.
The classic form of this problem is
publication bias, a tendency of journals to
accept preferentially papers reporting an
association over papers reporting no
association
EBM workshop
Meta analysis
A.A.Haghdoost
Publication bias
If such a bias is operating, a meta-analysis
based on only published reports will yield
results biased away from the null.
Because small studies tend to display more
publication bias, some authors attempt to
avoid or minimize the problem by
excluding studies below a certain size.
EBM workshop
Meta analysis
A.A.Haghdoost
Some meta-analysts present the effect
magnitude of all the studies as a funnel
plot, to address the issue of publication
bias.
A plot of 1/(standard error) vs effect
magnitude has an inverted funnel shape.
Asymmetry in the plot can indicate nonsignificant studies that weren’t published.
EBM workshop
Meta analysis
A.A.Haghdoost
Funnel plot
1/SE
“funnel” of
unbiased
studies
0
EBM workshop
effect
Meta analysis
magnitude
A.A.Haghdoost
Funnel plot
EBM workshop
Meta analysis
A.A.Haghdoost
Measures of Funnel Plot Asymmetry
1- Linear Regression Approach (Egger’s
method)
SND=a + b. precision
SND=OR/SE
The intercept “a” provides a measure of
asymmetry- the larger its deviation from
zero the more pronounced the
asymmetry.
EBM workshop
Meta analysis
A.A.Haghdoost
Measures of Funnel Plot Asymmetry
2- A rank correlation test
This method is based on association
between the size of effect estimates and
their variance. If publication bias is
present, a positive correlation between
effect size and variance emerges because
the variance of the estimates from smaller
studies will also be large.
EBM workshop
Meta analysis
A.A.Haghdoost
Funnel plot
EBM workshop
Meta analysis
A.A.Haghdoost
Key Messages
Funnel plot asymmetry was found in 38% of
meta-analyses published in leading
general medicine journals and in 13% of
reviews from the Cochrane Database of
Systematic Reviews.
Critical examination of systematic reviews
for publication and related biases should
be considered a routine procedure.
EBM workshop
Meta analysis
A.A.Haghdoost
Sources of Funnel Plot asymmetry
Selection Bias
True Heterogeneity
Size of effect differs according to study size:
–
–
–
–
–
–
–
–
–
Intensity of interventions
Difference on underlying risk
Data irregularities
Poor methodological design of small studies
Inadequate analyses
Fraud
Artefactual
Choice of effect measure
Chance
EBM workshop
Meta analysis
A.A.Haghdoost
Sample size as source of bias
Consider a hypothetical literature summary
stating, “of 17 studies to date, 5 have found a
positive association,11 have found no
association, and 1 has found a negative
association; thus, the preponderance of
evidence favors no association”.
Mere lack of power might cause most or all of the
study results to be reported as null.
EBM workshop
Meta analysis
A.A.Haghdoost
Quality score
Some meta-analysts score the quality of a study.
– Examples (scored yes=1, no=0):
• Published in a peer-reviewed journal?
• Experienced researchers?
• Research funded by impartial agency?
• Study performed by impartial researchers?
• Subjects selected randomly from a population?
• Subjects assigned randomly to treatments?
• High proportion of subjects entered and/or finished the
study?
• Subjects blind to treatment?
• Data gatherers blind to treatment?
• Analysis performed blind?
EBM workshop
Meta analysis
A.A.Haghdoost
Quality score
Use the score to exclude some
studies, and/or…
Include as a covariate in the metaanalysis, but…
Some statisticians advise caution
when using quality.
EBM workshop
Meta analysis
A.A.Haghdoost
Quality scoring
A very common practice is to weight
studies on a quality score usually based
on some subjective assignment .
For example, 10 quality points for a cohort
design, 8 points for a nested case control
design, and 4 points for a population
based case control design.
EBM workshop
Meta analysis
A.A.Haghdoost
Quality scoring
Quality scoring submerges important
information by combining disparate study
features into a single score.
It also introduces an unnecessary and
somewhat arbitrary subjective element in
to the analysis.
EBM workshop
Meta analysis
A.A.Haghdoost
Quality scores as weighing factors
study weight=1/var.
Quality adjusted weight= quality score /var.
EBM workshop
Meta analysis
A.A.Haghdoost
Quality scores
The judgment that the studies should or
should not be combined should be stated
and justified explicitly.
There is some of a tendency to make this
judgment on the basis of the quantitative
results, but it’s critical to make a
qualitative judgment.
EBM workshop
Meta analysis
A.A.Haghdoost
What is an IPD Meta-analysis?
Involves the central collection, checking
and analysis of updated individual patient
data
Include all properly randomised trials,
published and unpublished
Include all patients in an intention-to-treat
analysis
EBM workshop
Meta analysis
A.A.Haghdoost
IPD Meta-analysis
Individual patient data used
Analysis stratified by trial
IPD does not mean that all patients are
combined into a single mega trial
EBM workshop
Meta analysis
A.A.Haghdoost
IPD Analyses
Collect raw data from related studies,
whether or not the studies collaborated at
the design stage, exposures measures
and other covariates that can be applied
uniformly across the studies combined.
The major advantage of a IPD over an MA is
the use of individual-based rather than
group-based data.
EBM workshop
Meta analysis
A.A.Haghdoost
sensitivity analysis
In sensitivity analysis, the sensitivity of
inference to variations in or violations of
certain assumptions is investigated.
For example, the sensitivity of inference to
the assumption about the bias produced
by failure to control for smoking can be
checked by repeating the meta-analysis
using other plausible values of the bias.
EBM workshop
Meta analysis
A.A.Haghdoost
sensitivity analysis
If such reanalysis produces little change in
an inference, one can be more confident
that the inference is insensitive to
assumptions about confounding by
smoking.
In influence analysis, the extent to which
inferences depend on a particular study
or group of studies is examined; this can
be accomplished by varying the weight of
that study or group.
EBM workshop
Meta analysis
A.A.Haghdoost
sensitivity analysis
Thus , in looking at the influence of a
study, one could repeat the meta-analysis
without the study, or perhaps with half its
usual weight .
If change in weight of a study produces
little change in an inference, inclusion of
the study can not produce a serious
problem, even if unquantified biases
exist in the study
EBM workshop
Meta analysis
A.A.Haghdoost
Sensitivity and influence analysis
On the other hand, if an inference
hinges on a single study or group of
studies, one should refrain from
making that inference
EBM workshop
Meta analysis
A.A.Haghdoost
conclusion
Most meta-analysis will require from each
study both a point estimate of effect and
an estimate of its standard error .
A point estimate accompanied only by a P
value will generally not provide for
accurate computation of a standard error
estimate, and should not be considered
sufficient for reporting purposes.
EBM workshop
Meta analysis
A.A.Haghdoost
Over conclusion
Like large epidemiologic studies, metaanalysis run the risk of appearing to give
results more precise and conclusive that
warranted.
The lager number of subjects contributing
to a meta-analysis will often lead to very
narrow confidence intervals for the effect
estimate.
EBM workshop
Meta analysis
A.A.Haghdoost