Transcript Systemic Review and Meta-Analysis in Cancer Epidemiology
Systemic Review and Meta Analysis in Cancer Epidemiology
Chun Rebecca Chao, Ph.D.
Kaiser Permanente Southern California Department of Research and Evaluation 1
Overview
Introduction of systematic review and meta analysis Methods of systemic review and meta analysis 2
What is Systemic Review?
A review that has been prepared using a systemic approach to minimize biases and random errors (which is documented in a materials and methods section)*.
Review in which there is a comprehensive search for ALL relevant studies on a specific topic, and those identified are then appraised and synthesized according to a predetermined and explicit method.
Systemic review vs. Narrative review Narrative Review: traditional expert review, subjective, no formal rules in selecting studies, no standard statistical methods for combining studies.
3 *Chalmers and Altman. Systemic Reviews, BMJ Publishing Group, 1995.
What is Meta-Analysis?
A statistical analysis of the results from independent studies, which generally aims to produce a single estimate of a treatment effect. A systemic review may or may not include a meta analysis.
It is always appropriate and desirable to systemically review a body of data, but it is sometimes inappropriate to statistically pool results from different studies. 4
The Need for Systemic Review
Health Care Professional: number of biomedical publication has been increasing rapidly. Guideline and Policy Makers: Evidence based medicine is the trend for patient care, clinical guideline development and policy making.
Researchers: future research direction can be guided by systemic reviews. Consumers 5
Strength of Evidence Concerning Efficacy of Treatment
Case report Case series without controls Series with literature controls Series with historical controls Case control studies Cohort studies Randomized controlled trials (RCTs) SR/Meta-analysis of RCTs Prospective meta-analysis of RCTs with individual data 6
Strengths of Systemic Review
To provide a more objective appraisal of the evidence Contribute to resolve uncertainty when original research, reviews and editorials disagree.
Meta-analysis of the BCG vaccine trials Graph adopted from Egger, Smith and Altman. Systemic Reviews in Health Care. BMJ 7
Strengths of Systemic Review (Cont.)
To provide a more objective appraisal of the evidence Contribute to resolve uncertainty when original research, reviews and editorials disagree.
To reduce the probability of false negative results To explore treatment effects in subgroups of patients To explore and explain heterogeneity between study results To guide the direction of future studies To understand current gap and limitation in the literature To generate new research questions to be addressed.
8
Advantages of Use of Meta-Analysis to Combine Studies
When individual trials or studies are too small to give reliable answers When large trials or studies are impractical or impossible Potentially lead to more timely introduction of effective treatment When there have been many trials or studies showing small effects may be important 9
Number of Publications about Meta Analysis, 1985-2005
2500 2000 1500 1000 500 0 '8 5 '8 7 '8 9 '9 1 '9 3 '9 5 '9 7 '9 9 '0 1 '0 3 '0 5 Results from MEDLINE search using MeSH and text word “meta-analysis” 10
Potential Limitation of Conducting Systemic Review
Bias can be introduced in reviews in several ways.
Problems associated with design or reporting of original studies Limitations of using published data Publication bias 11
Publication Bias – A real threat for systemic review
Studies with significant results are more likely to be published More likely to be published without delay (lag time) More likely to be published in English More likely to be cited More likely to be published more than once Outcome reporting bias Significant outcomes are more likely to be reported than non-significant outcomes.
Should unpublished data be included in systemic review? Pre-specified inclusion (quality) criteria are recommended. 12
A Demonstration of Publication Bias
Graph adopted from Egger, Smith and Altman. Systemic Reviews in Health Care. BMJ 13
Predictors of Publication
Graph adopted from Egger, Smith and Altman. Systemic Reviews in Health Care. BMJ 14
Language Bias
Graph adopted from Egger, Smith and Altman. Systemic Reviews in Health Care. BMJ 15
Publication/ Reporting Bias?
Pan Z et al. PLoS Med. 2005 Dec;2(12):e334 16
Meta-Analysis of Observational Studies
In observational studies, bigger is not necessarily better. This is a danger that meta-analysis of observational data produces precise but spurious results.
A careful systemic review and examination of source of heterogeneity are more important than combining results.
Statistical combination of data should not be the main component of systemic review of observational data.
It is often desirable to have individual level data for this purpose. Confounding and effect modifiers.
17
Example 1. Fat Intake and Risk of Breast Cancer
Recall bias might play a role in the findings from case-control studies. Graph adopted from Egger, Smith and Altman. Systemic Reviews in Health Care. BMJ 18
Example 2: Intermittent Sunlight Exposure and Risk of Melanoma
Lack of blinding of the hypothesis to the subjects may introduce recall bias. Graph adopted from Egger, Smith and Altman. Systemic Reviews in Health Care. BMJ 19
Example 3. Formaldehyde Exposure and Lung Cancer
Dose-response or selection bias?
Graph adopted from Egger, Smith and Altman. Systemic Reviews in Health Care. BMJ 20
Carotenoids and the Risk of Developing Lung Cancer: A Systemic Review*.
Objective: Systemic review of the association between carotenoids and lung cancer. Included both RCT and prospective observational studies (smoking adjusted).
Examined total carotenoids, β-carotene, α-carotene, β-cyrptoxanthin, lycopene, and lutein-zeaxanthin. Examined carotenoid supplements, dietary intake of carotenoids, and serum carotenoid concentrations. 21 *Gallicchio L et al, Am J Clin Nutr 2008; 88: 372-83.
β-carotene Supplement Use in RCT
Gallicchio et al, Am J Clin Nutr 2008; 88: 371-83 22
Dietary Total Carotenoid Intake: Prospective Cohort Studies
Gallicchio et al, Am J Clin Nutr 2008; 88: 371-83 23
Dietary β-carotene Intake: Prospective Cohort Studies
Gallicchio et al, Am J Clin Nutr 2008; 88: 371-83 24
Serum Total Carotenoid Concentrations: Prospective Cohort Studies.
Gallicchio et al, Am J Clin Nutr 2008; 88: 371-83 25
Dose Response
Gallicchio et al, Am J Clin Nutr 2008; 88: 371-83 26
Subgroup Analysis
Gallicchio et al, Am J Clin Nutr 2008; 88: 371-83 27
Method for Systemic Review
1. Define research questions 2. Define inclusion/exclusion criteria 3. Search the literature 4. Select articles 5. Evaluate the internal/external validity of the studies 6. Extract/Abstract Data 7. Calculate effect size and standard error 8. Examine heterogeneity 9. Assess publication bias 10. Combining study results if appropriate 11. Influence analysis, sensitivity analysis 12. Interpretation of results 13. Reporting 28
Stages of a Systemic Review
Stage 1 Planning Identification of the need for a systemic review Development of a proposal Stage 2 Conducting Identification of the research Selection of studies Stage 3 Reporting The report and recommendations Study quality assessment Data extraction Data synthesis
29
1. Clearly Formulated Question
The research question is extremely important! Get feedback about your specific research question from many people (content expert, savvy clinician in the field, methodologist) The question should be clearly specified What is the study objective To validate results in a large population To guide new studies Pose questions in both biologic and health care terms specifying with operational definitions Population Intervention/exposure Outcomes (both beneficial and harmful) 30
Study Protocol
Develop a protocol of what will be done Background: why is this important?
Specify the research question Research question defines the following criteria Provide an overview of the methods, including search strategy, inclusion/exclusion criteria, quality assessment, how data will be abstracted, and an analysis plan Specify any subgroup analyses or sensitivity analyses (best if these are
a priori
) Initially, inclusion criteria should be overly broad E.g., search all alcohol when specifically interested in wine.
E.g., search all dietary, anthropometric related papers when 31 interested in carotenoids.
2. Develop Inclusion Criteria
Use clinical/scientific judgment to enhance validity and homogeneity Validity (Study quality) Homogeneity Similar study design Similar patient populations Similar interventions/exposures Similar outcomes 32
Typical Inclusion Criteria
Study Design (e.g., RCT, prospective cohort) Population (risk group) Interventions (dosage, regimen)/exposure of interest Outcomes (definitions, follow-up period) 33
Practical Considerations in Defining Eligibility for a Systemic Review
Study designs to be included Years of publication or study conduct Languages Choice among multiple publications Restrictions due to sample size or follow-up duration Similarity of treatment and/or exposure Completeness of information 34
List of excluded studies Chao, C. et al. Am. J. Epidemiol. 2008 168:471-480; doi:10.1093/aje/kwn160
Copyright restrictions may apply.
35
3. Comprehensive Literature Search
Need a well formulated and coordinated effort Seek guidance from a librarian Specify language constraints Requirements for comprehensiveness of the search depends on the field and question to be addressed 36
Searching the Literature
Database searches: MEDLINE, EMBASE, ISI Web of Science, PsychLit, CancerLit, Cochrane, Dissertations online Reference lists of retrieved articles Manual searching of related journals, conference proceedings, textbooks Experts in the field Granting agencies Study/trial registries Industry (device manufacturers, pharmaceutical companies) 37
How to Develop a Search Strategy ( Example )
Study Question What is the relationship between consumption of different types of alcoholic beverage (beer, wine and liquor) and risk of lung cancer?
Break down the question into facets (not all may be needed for searching) Exposure: Alcoholic beverage consumption Outcome: Lung cancer Study design: Observational studies with internal comparison group (case-control or cohort) 38
How to Develop a Search Strategy (Cont.)
Identify synonyms, spelling variants, and subject headings associated with each facet Text terms (title/abstract) alcohol ethanol alcoholic alcoholics beer wine liquor MeSH alcohol drinking ethanol 39
How to Develop a Search Strategy (Cont.)
Text terms (title/abstract) lung in combination with cancer neoplasm carcinoma tumor incidence mortality MeSH lung neoplasm 40
4. Study Selection
Unbiased Study Selection Two independent reviewers select studies Based on a priori specification of the population, intervention, outcomes and study design Differences are resolved by consensus Specify reasons for rejecting studies Keep a record of what is excluded and reason (journals may require this and
the search must be reproducible
) 41
Figure 1.
Flow Diagram of Study Selection Process
Potentially relevant references identified after liberal screening of the electronic search (n=#) Excluded by Title/Abstract (n=#) List the reasons Articles retrieved for more detailed evaluation (n=#) Articles excluded after evaluation of full text (n=#) List the reasons Relevant studies included in the meta analysis (n=#) 42
Carotenoids and Lung Cancer Systemic Review Gallicchio et al, Am J Clin Nutr 2008; 88: 371-83 43
Study Quality Assessment
Quality of study conduct vs. quality of reporting Controversial since this process is subjective and can introduce bias No standard method to assess quality of a study Scales and checklists exist but can be challenging to incorporate for observational studies Pre-specify algorithm for quality assessment Assess quality of each study in uniform, systematic and complete manner Once quality is assessed then what to do with that information Consider weighting each study result by quality score Exclude studies with ‘poor’ quality Stratifying by methodological quality component (component approach) 44
5. Data Extraction
Develop an easy-to-fill-out form for collecting data from each article Study methods (quality) Study description (population) Results (outcomes, side effects) Two independent reviewers extract data Should be explicit, unbiased, and reproducible Include all relevant measures of benefit and harm of the intervention Contact investigators of the studies for clarification in published methods, data Differences in data extraction are resolved by consensus 45
Data Extraction: Study Characteristics
Types of publication (journal article, abstract or unpublished data) Publication year and country of origin Study participants (sample size, age, gender, race, health status) Design details (case-control, cohort, parallel or cross-over, randomization, blinding) Nature of treatment and control Study duration Measurement of compliance Definition and measurement of outcome Other confounders 46
Data Extraction: Study Outcome
Continuous data Outcomes summarized as means (blood pressure, weight) Dichotomous or binary data Each individual must be in one of two states (dead/alive, smoking/not smoking) These data are summarized using odds ratios, risk ratios, or risk differences Survival or time to event data Outcome of interest is the time to the occurrence of an event Usually summarized using hazard ratios Other data Some outcomes may be Short ordinal scales (pain scales: individuals’ rate their pain as none, mild, moderate, severe) for which it is not sensible to calculate a mean Event counts (number of asthma attacks per month) 47
Gallicchio et al, Am J Clin Nutr 2008; 88: 371-83 48
6. Calculating Effect Size and Standard Error Perform a narrative, qualitative summary when data are too sparse, or too low quality or too heterogeneous to proceed with a meta-analysis The results from each study are converted into an Odds Ratio (OR) or Effect Size (ES) 95% confidence intervals (CI) are calculated for each study-specific OR or ES For a meta-analysis, If only confidence intervals are given, computation will be required to obtain estimates of the standard error 49
7. Examining Heterogeneity
Statistical test for heterogeneity Visual inspection/Graphical approach Forest plot, Galbraith plot Meta-regression Unit of regression: study Dependent variable: study-specific effect estimate Independent variables: study-specific characteristics (e.g., study design, geographic location, length of follow-up) 50
Examining Forest Plot for Heterogeneity
Graph adopted from Egger, Smith and Altman. Systemic Reviews in Health Care. BMJ 51
Meta-regression STATA Output
Type 2 diabetes and risk of NHL: systemic review and meta analysis ---------------------------------------------------------------------------------------- | Coef. Std. Err. z P>|z| [95% Conf. Interval] ---------------------------------------------------------------------------------------- design | .4256531 .203504 2.09 0.036 .0267926 .8245136 _cons | .167641 .083123 2.02 0.044 .0047228 .3305591 -------------------------------------------------------------------------------------- -- Reference group: case-control design ---------------------------------------------------------------------------------------- | Coef. Std. Err. z P>|z| [95% Conf. Interval] ---------------------------------------------------------------------------------------- europe | .1077322 .1625632 0.66 0.508 -.2108859 .4263503 asia | .4499953 .1966564 2.29 0.022 .064556 .8354347 _cons | .107974 .1095071 0.99 0.324 -.106656 .3226041 --------------------------------------------------------------------------- ------------- Reference group: US studies 52
8. Assessing Publication Bias
Results because negative studies are less likely to be submitted or published Can bias the results of a meta-analysis toward a positive finding Can evaluate publication bias graphically (funnel plot) or through statistical analysis Egger’s test, Begg’s test 53
Begg’s Funnel Plot
Graph adopted from Egger, Smith and Altman. Systemic Reviews in Health Care. BMJ 54
Begg's funnel plot with pseudo 95% confidence limits for assessment of publication bias Chao, C. et al. Am. J. Epidemiol. 2008 168:471-480; doi:10.1093/aje/kwn160
Copyright restrictions may apply.
55
9. Meta-Analysis for Calculating a Summary Effect Estimate
Several methods are available for combining study results Inverse variance method M-H methods Peto’s odds ratio method Fixed effect vs. random effect 56
Subgroup Analyses
Pre-specify hypothesis-testing subgroup analyses and keep few in number Label all posteriori subgroup analyses When subgroup differences are detected, interpret in light of whether they were: Established a priori Few in number Supported by plausible mechanisms Important (qualitative vs. quantitative) Consistent across studies Statistically significant (adjusted for multiple testing) 57
10. Influence Analysis
Examine how each study influence the summary statistic by removing one study at a time and re-calculate the combined estimate. A graphically display can be used for visual inspection of influential studies.
58
Influence Analysis STATA Output
Wine consumption and risk of lung cancer: systemic review and meta-analysis ------------------------------------------------------------------------------ Study ommited | e^coef. [95% Conf. Interval] ------------------------------------------------------------------------------ Bendera 1992 | .831761 .67413002 1.0262506 De Stefani 1993 | .78909296 .65476894 .95097321 Carpenter 1998 | .81794792 .66839695 1.0009602 De Stefani 2002 | .82806259 .67865324 1.0103652 Hu 2002 | .82721388 .67337066 1.0162053 Freudenheim 2003 | .82455081 .67135686 1.0127014 Benedetti I 2006 | .82831144 .67354399 1.0186415 Benedetti II 2006 | .81791788 .66001374 1.0135996 Benedetti II 2006 | .82721388 .67337066 1.0162053 Pollack 1984 | .78993553 .67183363 .92879862 Prescott 1999 | .84944367 .70591372 1.0221567 Prescott 1999 | .83032089 .69030547 .99873579 Freudenheim 2005 | .81104547 .65682006 1.001484 Freudenheim 2005 | .78433573 .64188826 .9583950 ------------------------------------------------------------------------------- Combined | .81770466 .67530395 .99013327 ------------------------------------------------------------------------------- 59
Influence Analysis STATA Output
Graph adopted from Egger, Smith and Altman. Systemic Reviews in Health Care. BMJ 60
10. Sensitivity Analysis
Test robustness of results relative to key features of the studies and key assumptions and decisions Include tests of bias due to retrospective nature (e.g., with/without studies of lower methodological quality) 61
Component Approach
Graph adopted from Egger, Smith and Altman. Systemic Reviews in Health Care. BMJ 62
11. Interpretation of Findings
Interpret results in context of clinical practice State methodological limitations of the individual studies included and in the meta-analysis Consider size of effect in studies and meta-analysis, consistency of effect sizes and any dose-response relationship Interpret results in light of other valuable evidence Make recommendations clear and practical Propose future research agenda (clinical and methodological requirements) 63
12. Describe Studies and Findings
Guideline for reporting systemic review The QUOROM statement (Quality Of Reporting Of Meta-analysis) Moher D, et al. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet 1999, 354(9193):1896-900. 64
QUOROM Check List
Moher D, et al. Lancet 1999, 354(9193):1896 900. 65
Summary
A well conducted systemic review allows for a more objective appraisal of the evidence than traditional narrative reviews Systemic review may contribute to resolve uncertainties and disagreements in original research Meta-analysis may enhance the precision of estimates of treatment effects Exploratory analyses (i.e., subgroups who are likely to respond well to a treatment) may guide cost effective treatment decisions Systemic review may demonstrate areas where the evidence is inadequate and thus identity areas where further research is needed 66