Systematic Reviews and Meta-Analyses

Ritz Kakuma, MSc (PhD Candidate) Department of Epidemiology & Biostatistics McGill University

Problems with Today's Medical Literature



Most studies are too small

 Inconclusive, often conflicting results 

Traditional reviews are unstructured & subjective

 Biased conclusions

WHO CAN KEEP UP?

 Over 345,000 trials published to date  Over 20,000 new trials published annually

Help!

For General Physicians to keep current:

Read 19 new articles per day which appear in medical journals 19 x 2 hrs (Critical Appraisal) = 38 hrs per day Davidoff F et al. (1995)

EBM; A new journal

to help doctors identify the information they need. BMJ 310:1085-86.

Statistical Quality of Medical Research gets low marks!

1996 G. WELCH, S. GABBE

Review of Statistics Usage in the Amer J Obstet Gynecol 175;1138-41

 All clinical papers: Jan - June 1994 (Vol. 170, No. 1 to 6)  31.7% (46/145) had inappropriate statistics  27 of 46 papers had serious flaws RESULTS???

Problems with Standard Reviews

      Lack of scientific purpose (question) Undocumented methods of literature search Unstated criteria for selecting studies No methodological assessment of selected studies Inadequate assessment of inter-study differences in results No attempt at quantitative synthesis (pooling) to take advantage of increased power

Why Systematic Reviews?

 Help to deal with the volume of literature  Help resolve conflicting results  Scientific rather than subjective summarization of literature  Can reveal new evidence  Identify knowledge gaps   More reliable evidence with which to aid decision making Guide clinical research by providing new hypotheses

The Cochrane Collaboration - origins

• Archie Cochrane

–

“It is surely a great criticism of our profession that we have not organized a critical summary, by specialty or subspecialty, adapted periodically, of all relevant randomized controlled trials.”

• Pregnancy and childbirth work - 1980s

Canadian Cochrane Network & Centre

Aims and objectives of the CC

“ The Cochrane Collaboration is an international organization that aims to help people make wellinformed decisions about healthcare by preparing, maintaining and promoting the accessibility of systematic reviews of the effects of healthcare interventions” • Founded 1993

Canadian Cochrane Network & Centre

CC built on 10 Principles

• collaboration • building on the enthusiasm of individuals • avoiding duplication • minimizing bias • keeping up to date • striving for relevance • promoting access • ensuring quality • continuity • enabling wide participation

Canadian Cochrane Network & Centre

Organization of the CC

• Cochrane Centres (n=12) • Collaborative Review groups (n=51) • Fields (n=9) • Networks (n=1) • Methods working groups (n=10)

Canadian Cochrane Network & Centre

Cochrane Centres

• Australasian CC • Brazilian CC • Canadian CC • Chinese CC • Dutch CC • German CC •Iberoamerican CC • Italian CC • Nordic CC • South African CC • UK CC • US CC (Rhode Island, Boston, San Francisco Branches)

Canadian Cochrane Network & Centre

Cochrane Review Groups

Acute Respiratory Infections Group Airways Group Anaesthesia Group Back Group Breast Cancer Group Colorectal Cancer Group Consumers and Communication Group Cystic Fibrosis and Genetic Disorders Group Dementia and Cognitive Improvement Group Depression, Anxiety and Neurosis Group Developmental, Psychosocial & Learning Problems Group Drugs and Alcohol Group Ear, Nose and Throat Disorders Group Effective Practice and Organisation of Care Group Epilepsy Group Eyes and Vision Group Fertility Regulation Group Gynaecological Cancer Group Haematological Malignancies Group Heart Group Hepato-Biliary Group HIV/AIDS Group Hypertension Group Incontinence Group Infectious Diseases Group Inflammatory Bowel Disease Group Injuries Group Lung Cancer Group Menstrual Disorders and Subfertility Group Metabolic and Endocrine Disorders Group Methodology Review Group Movement Disorders Group Multiple Sclerosis Group Musculoskeletal Group Musculoskeletal Injuries Group Neonatal Group Neuromuscular Disease Group Oral Health Group Pain, Palliative and Supportive Care Peripheral Vascular Diseases Group Pregnancy and Childbirth Group Prostatic Diseases and Urologic Cancers Group Renal Group Schizophrenia Group Sexually Transmitted Diseases Group Skin Group Stroke Group Subfertility Group (see Menstrual Disorders) Tobacco Addiction Group Upper Gastrointestinal & Pancreatic Diseases Grp Wounds Group

Cochrane Fields & Network

Canadian Cochrane Network & Centre • Cancer Network • Child Health • Complementary Medicine • Health Care of Older People • Health Promotion and Public Health • Neurological Network • Primary Health Care • Rehabilitation and Related Therapies • Vaccines Field • Consumer Network

Cochrane Methods Groups

Canadian Cochrane Network & Centre • Applicability and Recommendations • Health Economics • Health-Related Quality of Life • Individual Patient Data Meta-Analyses • Non-randomised Studies • Prospective Meta-Analysis • Qualitative Methods • Reporting Bias Methods • Screening and Diagnostic Tests • Statistical Methods

Canadian Cochrane Network & Centre

Cochrane Activities

• Produce and update systematic reviews • Hand search for RCTs • Medline (and others) enhancement • Review methodology  COCHRANE LIBRARY

Canadian Cochrane Network & Centre

When should you use the CLIB?

For questions on effectiveness     What is the effectiveness of treatment x What is an effective treatment for y Is z effective in treating y Is z better than x at treating y

Canadian Cochrane Network & Centre

When not to use the Cochrane Library

General healthcare questions • causal, prognosis, epidemiology, etc.

• Statistics (prevalence and incidence) • Primary research other than RCTs • Guidelines • Current research

Canadian Cochrane Network & Centre

The Cochrane Database of Systematic Reviews (CDSR) Complete Reviews

 full text, regularly updated systematic reviews of the effects of health care (1,837 reviews)  prepared and maintained by the Collaboration Review Groups

Protocols

 protocols of reviews currently being prepared, incl. expected date of completion (1,344 protocols)  includes background, objectives and methods sections

The Database of Abstracts of Reviews of Effectiveness (DARE)

Canadian Cochrane Network & Centre prepared by the National Health Services Centre for Reviews and Dissemination at the University of York, UK.

Abstracts of quality assessed systematic reviews

 structured abstracts assessing the quality of previously published SRs & summarizing findings (3,484 reviews)

Other reviews: bibliographic details only

 references to published SRs NOT assessed for quality (800 reviews)

The Cochrane Central Register of Controlled Trials (CENTRAL)

Canadian Cochrane Network & Centre

References

 Reference list of ALL identified published randomized trials  Latest issue 378,160 RCTs (Medline: 81,705 articles identified as RCT publication type)

The Cochrane Database of Methodology Reviews (CDMR)

Canadian Cochrane Network & Centre

Complete Reviews

 Full-text SRs of methodological studies (9 reviews)  Highly structured and systematic, covering a specific and well-defined area of methodology  prepared and maintained by the Cochrane Methodology Review Groups

Protocol

 protocols of reviews currently being prepared, incl. expected date of completion (7 protocols)  includes background, objectives and methods sections

Cochrane Review Methodology Database (CRMD)

Canadian Cochrane Network & Centre

References

 published reports of empirical studies of methods used in reviews (4,626 reports)  methodological studies directly relevant to conducting a review

Searching the CLIB

Canadian Cochrane Network & Centre Some basics… ALL contents of ALL records in ALL databases are searched Ignores: punctuation & numbers Boolean terms: AND, OR, NEXT, NEAR – within 6 words both ways, NOT Restricting searches  ‘Options’ page and choose desired restrictions  At the end of the term, add: :AU – Author :AB – Abstract :TI – Title :KY - Keywords :ME - MeSH terms

Searching the CLIB

Canadian Cochrane Network & Centre MeSH  Keywords drawn from MeSH thesaurus of U.S. NLM    Accompanies some but not all records Organized hierarchically in ‘trees’ Permuted Index – an index of all words that appear in the MeSH thesaurus  used to located specific MeSH terms

Canadian Cochrane Network & Centre

At McGill University and affiliated sites:

The pathway to the CLIB is: McGill HSL library homepage > Databases > > Cochrane Library > web (no login or password nec.) http://www.cochranelibrary.com/enter/

Useful Materials

Canadian Cochrane Network & Centre       The Cochrane Library: Self Training Guide Interpretation of Odd-ratio diagrams WWW links, HTA database Cochrane Handbook User manual Glossary, etc...

DOWNLOAD

http://www.cochrane.org/resources/training.htm

Quality of Cochrane Reviews

• Comparison of 36 Cochrane reviews with 39 paper-based journals  Cochrane reviews less prone to bias (Jadad et al. 1998) – Explicit reporting of eligibility criteria (35/36 vs 18/39) – Assessed trial quality (36/36 vs 7/39) – No language restriction (36/36 vs 32/39)

Quality of Cochrane Reviews

• Also not perfect (Olsen et al. 2001) • 52 Cochrane reviews from 1998 • 18% - conclusions not backed up by evidence – All overestimated effect of intervention

Discordant Reviews

• Direction of effect • Significance • Magnitude of effect • Interpretation of results  Numerous possible reasons

Sources of discordance

• Different research question – Target population – Intervention being studied – Outcome measures – Setting • Study selection – Search strategy – Eligibility criteria

Sources of discordance

• Data extraction – Methods of measuring outcomes and endpoints – Extent of human error • Quality assessment method – 25 scales and 9 checklists avail for assessing RCT quality (Moher et al. 1996) • Inconsistent quality depending on instrument used – QUOROM (Quality of Reporting of Meta-Analyses) = most comprehensive (Moher et al. 1999, Shea et al. 2010)

Sources of discordance

• Analysis – Appropriateness of combining results – Method of synthesis: Descriptive, meta-analysis – Statistical methods • Bayesian • Meta-regression • Frequentist • Interpretation of evidence

Decision Algorithm for interpreting discordant reviews

Screening for breast cancer with mammography among women aged 50-69 years

Report prepared for the Breast Cancer Screening Unit,

Cancer Branch, Health Canada

Ritsuko Kakuma March 2002

MAs on mammography screening for breast cancer:

• Fletcher et al. Report of the International Workshop on Screening for Breast Cancer. J Natl Cancer Inst 1993; 85:1644-1656.

• Kerlikowske et al. Efficacy of screening mammography. A meta-analysis. JAMA 1995; 273:149-154.

• Olsen O & Gøtzsche PC

.

Is screening for breast cancer with mammography justifiable? Lancet 2000.

Fletcher et al. review

Research Question: • To assess current state of knowledge about BRCA screening, identify knowledge gaps Study Sample: • Women aged 40-79 years stratified into 3 groups: 40 49, 50-69, 70+

Fletcher et al. review

Methodological issues:

• Int’l Workshop on screening for BRCA in Feb 1993 (by NCI) • W/S participant characteristics not reported • Study selection method not described • Data source: published and unpublished data provided by the w/s participants • Validity of trials assessed: – Randomization, Compliance, Contamination – Quality and frequency of screening test – Adequacy of FU – Validity of outcome assessment – Generalizability of results

Fletcher et al. review

Results/Conclusions: • 7 trials identified • For age 50-69 age group, only 5 trials provided data – New York, Two-County, Malmo, Edinburgh and Canada • No statistical pooling of data • Together, Fletcher et al. concluded that there was substantial benefit of screening on BRCA mortality  reduction of 30% after 10-12 years of FU

Kerlikowske et al. review

Research Question: • To determine efficacy of screening in reducing BRCA mortality by age, # MM views per screen, screening interval and duration of FU Sample: • Women aged 40-74 years stratified into 3 groups: 40-49, 40-74, 50-74

Kerlikowske et al. review

Methodology: • Search: MEDLINE, manual (reference lists) and consultations with colleagues and experts • English language restriction • Explicit inclusion criteria • Data extraction – 2 indep extractors & 3 rd for resolving discrep’s – Published and unpublished data used • Quality assessment not reported..not done?

• Analysis – Pooled data of all included trials using fixed effects

Kerlikowske et al. review

Results / Conclusions: • 9 studies (7 RCTs & 2 CCS) have data on 50-74 year age group • All studies – beneficial effect of screening (4 statistically significant) • All trials included in the analysis (not CCSs) • MM screening reduced BRCA mortality by 27% in this age group after 7- 9 years and 24% after 10-12 years regardless of # MM views, screening interval, duration of screening or addition of CBE

Olsen & Gøtzsche review

Research Question: • To assess effect of MM screening for BRCA on mortality and morbidity • Review methodological quality of trials – focus on 3 most important sources of bias – Randomization – blind outcome assessment – post-Rx exclusions Sample: • Women without previously diagnosed BRCA on women aged <50 years and 50+ years

Olsen & Gøtzsche review

Methodology: • Search: MEDLINE, common author names search, reference lists, specialized trials registers, letters, abstracts, grey literature • Explicit inclusion criteria: – RCT / quasi-RCT on MM screening vs. no MM screening • Two indep reviewers for study selection, quality assessment & data extraction • Quality assessment: High, Medium, Poor or Flawed – Rx process, baseline comparability, post-Rx exclusions, Consistency of number of women randomized • Analysis – Intent-to-Treat analysis, with fixed effect unless heterogeneous – Stratified by quality (excluded flawed)

Olsen & Gøtzsche review

Results / Conclusions: • 7 trials identified, none with high quality (2 medium, 3 poor, 2 flawed) • Total mortality – no impact of screening after 7- and 13 yrs FU for both medium and poor quality trials • BRCA mortality – no beneficial effect of screening in medium quality trials – Beneficial effect among poor quality trials pooled RR=0.69, CI 0.55-0.80 at 7 years pooled RR=0.64, CI 0.54-0.76 at 13 years • Interventions for screened group higher – Over-diagnosis, increased use of more aggressive treatment such as mastectomies and tumorectomies (20-30%)

Decision Algorithm for interpreting discordant reviews

Identified Trials

Trials: Methodological issues

Canadian CBE in control group may dilute effect of screening (MM+CBE vs. CBE) Imbalance in rate of advanced cancers at Rx Malm Stockholm G ö öteborg Inconsistent numbers across multiple reports – missing unaccounted for Date of entry different for the 2 groups (date of Rx vs. date of invitation to screening) Contamination: 24% of controls had MM Continuation of study: not randomized, no baseline information Inconsistent numbers across multiple reports – missing unaccounted for Non-independence of results of 2 sub-trials (2 smaller trials with overlapping controls) Imbalanced post-Rx exclusions, Contamination (Screening for controls began as early as year 3) Information on Rx not described adequately Contamination (Screening for controls began 3 rd and 6 th years)

Two County

Trials: Methodological issues

Age imbalance of 5-months – younger in screening group (clinically signif?) Cluster Rx method not described adequately Contamination (Screening for controls began after a few years into study) New York Baseline imbalance in previous lump, menopause and education Post-Rx exclusion ID method of current BRCA differed --self-report vs. screen Outcome assessment 72% non-blind (differential misclassification or true effect?) Edinburgh Inadequate Rx  Baseline imbalances – screened group had higher SES Change in allocation status after Rx Imbalanced post-Rx exclusions, Contamination (Screening for controls began in year 10 of FU) Differential eligibility criteria and entry dates for the 2 groups

Impact on effect measure

• Contamination & Compliance : underestimate • Post-Rx exclusions: overestimate – Identification of BRCA at baseline more complete for screened group..first round of screening used to exclude – Dropouts and losses: equally distributed?

• Adequacy of Randomization: overestimate – Typically overestimate treatment effect by 30-40% (Schulz et al. 1995; Moher et al. 1998; Kjaergard et al. 2001) • Outcome Assessment: overestimate – Misclassification – Determination of cause of death may be difficult in light of multiple illnesses

Comparability of Trials

• Same research question?

– Effectiveness vs. efficacy

• Same Target population?

• Appropriate study sample? • Intervention?

– Screening modality, frequency of screening

• Outcome assessment

– Cause of death identified the same way?

• Analytical methods

Fletcher

  No precise research question Inadequate description of methods used – Search strategy, trial selection, data extraction  Acknowledgement of difference in quality of trials but conclude based on ALL evidence  Source of some data unclear  NOT synthesizing data = appropriate • But evidence does not support their conclusion that BRCA mortality is reduced by 30% as a result of screening

Kerlikowske

  Clear research question Explicit reporting of methods used    No indication that quality assessment was done Some numbers extracted don’t match reference cited Missing key articles for trials (I.e., original report) • Despite good reporting of methods, caution needed in interpreting results because of the questionable nature of data used

Olsen and Gotzsche

    Clear research question Explicit reporting of methods used Strong methodology Addressed important question: Reduced BRCA mortality not necessarily equal to improved survival  Did not assess baseline comparability stratified by age groups, which is how the data was analyzed • Most rigorous and properly done review of the 3 • Brought potential adverse effects of screening to forefront

Conclusions:

• All existing evidence has methodological issues • No conclusive interpretations can be made • US and Canada continue to recommend MM screening for women in this age group