Transcript Is a meta-analysis right for me?

Is a meta-analysis right for me?
Jaime Peters
[email protected]
June 2014
What is a meta-analysis? (1)
Gene Glass 1976:
“Meta-analysis refers to the analysis of analyses”
Quantitative synthesis of data extracted from a systematic
review
Systematic review
Study 1
Study 2
Study 3
Study 4
Meta-analysis
….
Study N
What is a meta-analysis? (2)
A weighted
average
Framework for investigation into
– Heterogeneity/differences between study results
– Biases associated with the review (publication/reporting)
– Assess sensitivity of results
Why is a meta-analysis useful?
• Meta-analysis > single study: therefore greater power to
detect an effect if one exists
• Quantify effect sizes and their uncertainty
– “average effect is X”, rather than “most studies report a positive
effect”
• Facilitate synthesis of large number of studies
– Summarising tens, sometimes hundreds studies
• Allows exploration of heterogeniety
• Obtain results that inform evidence-based medicine
– E.g. input directly into cost-effectiveness analyses
What might affect your decision to do a
meta-analysis?
Reporting or
publication biases
Study quality
Heterogeneity
Importance of a protocol
Incomplete
reporting
Example: Arsenic systematic review
Study quality
• Different study designs can be subject to different types
of bias, e.g. Blinding in RCTs; recall bias in case-controls
• Quality assessment, e.g. Cochrane risk of bias tool
• But, what to do with this information?
– Garbage in, garbage out
– Precise meta-analysis results, but how useful?
• Use a pre-defined cut-off for the meta-analysis?
• Assess sensitivity of results to low quality studies
• Down-weight low quality studies?
Example: Arsenic systematic review
• Quality appraisal:
http://www.ephpp.ca/PDF/Quality%20Assessment%20T
ool_2010_2.pdf
Strong n=1
• 51 studies
– Weak
– Moderate
– Strong
Weak
n=50
Heterogeneity
Methodological
Clinical
Populations
age
sex
Study design
RCT, cohort, case-control,
etc
Setting
geography
primary/ secondary care
Outcome measure
mean difference, odds ratio,
hazard ratio, etc
Statistical
Consequence of clinical and/or methodological heterogeneity
Observed effects from each study more different than we would expect by chance
Protocol
Work-arounds
Q-test, I2
Sub-groups
Average effect
Individual patient
data
Meta-regression
Example: Arsenic systematic review
• Interventions
– 51 studies
– 50 interventions!
• Study design
–
–
–
–
RCTs
Pre/post
Non-randomised
Cross-over
• Follow-up times
• Country
• Unit of analysis
– Samples from multiple sites
– Repeat samples from one
site
– Repeat samples from
multiple sites
Reporting and publication biases
• Interesting, positive, statistically significant
results more likely to be published and
reported
• Without consideration, meta-analysis just replicates,
sometimes exaggerates, this bias
• Avoidance: comprehensive search of grey literature
• Methods to assess presence of reporting and/or
publication biases
• Funnel plots, tests, methods for adjustment
Example: Arsenic systematic review
• Comprehensive search, including grey literature
• Due to heterogeneity, no attempt to investigate presence
of reporting or publication bias
• Mention possibility of reporting bias as limitation
• Methods for identifying or adjusting for reporting and
publication bias are not ideal
• Require >10? studies and perform poorly in presence of
heterogeneity
Incomplete reporting
• Word, table, figure limits
• Poor reporting
• Contact authors
• Individual patient data
Example: Arsenic systematic review
• Poor quality reporting
• Could not extract necessary outcome data
– “effectiveness evidence inconclusive”
• Did not contact authors – time and resources running
out!
• Major limitation of the systematic review
• Did we conduct a meta-analysis?
Other difficulties
• Too few studies: testing for heterogeneity and
publication/reporting biases, meta-regression and subgroups
• Multiple reports of same study
• Outcomes measured at different time-points
• Most issues not restricted to meta-analysis
• Meta-analysis: systematic and transparent framework
• Protocol considering possibilities very important
• Cochrane Handbook great resource:
handbook-cochrane.org
Useful refs
AJ Sutton et al (2000) Methods for Meta-analysis in Medical Research, Wiley.
[Systematic review of trials and other studies, HTA, 1998 Vol 2 No 19
http://www.journalslibrary.nihr.ac.uk/hta/volume-2/issue-19]
JPT Higgins and S Green (2011) Cochrane Handbook for Systematic Reviews
of Interventions. Version 5.1.0 (Cochrane website)
M Egger et al (2001) Systematic Reviews in Health Care: Meta-analysis in
Context, BMJ.
M Borenstein et al (2009) Introduction to Meta-analysis (Statistics in Practice),
Wiley.
Questions? Comments?
[email protected]