Transcript Systematic Reviews and Meta-Analyses

Systematic Reviews and
Meta-Analyses
Ritz Kakuma, MSc (PhD Candidate)
Department of Epidemiology & Biostatistics
McGill University
Outline
•
•
•
•
Background on SR/MA
Cochrane Collaboration
Cochrane Library
Example on Mammography Screening
for Breast Cancer
Problems with Today's
Medical Literature
 Most studies are too small

Inconclusive, often conflicting results
 Traditional reviews are unstructured &
subjective

Biased conclusions
WHO CAN KEEP UP?
 Over 425,000 trials
published to date
 Over 20,000 new trials
published annually
Help!
For General Physicians
to keep current:
Read 19 new articles per day which appear in
medical journals
19 x 2 hrs (Critical Appraisal) = 38 hrs per day
Davidoff F et al. (1995)
EBM; A new journal to help doctors identify
the information they need. BMJ 310:1085-86.
Statistical Quality of Medical
Research getting better…
2002 G. WELCH, S. GABBE
Statistics Usage in the Amer J Obstet Gynecol: has
anything changed? 180;584-6
 All clinical papers: Jan - June 1994 (Vol. 170, No. 1 to 6)
vs. Jan – June 1999 (Vol. 180, No. 1 to 6)
 Inappropriate statistics used in 31.7% (46/145) in 1994
and 9.8% (19/195) in 1999.
RESULTS???
Problems with Standard Reviews
 Lack of scientific purpose (question)
 Undocumented methods of literature search
 Unstated criteria for selecting studies
 No methodological assessment of selected
studies
 Inadequate assessment of inter-study
differences in results
 No attempt at quantitative synthesis (pooling)
to take advantage of increased power
Why Systematic Reviews?
 Help to deal with the volume of literature
 Help resolve conflicting results
 Scientific rather than subjective summarization of




literature
Can reveal new evidence
Identify knowledge gaps
More reliable evidence with which to aid decision
making
Guide clinical research by providing new
hypotheses
The Cochrane Collaboration origins
• Archie Cochrane
– “It is surely a great criticism of our profession that
we have not organized a critical summary, by
specialty or subspecialty, adapted periodically, of
all relevant randomized controlled trials.”
• Pregnancy and childbirth work - 1980s
• Founded 1993
Aims and objectives of the CC
Canadian Cochrane
Network & Centre
“The Cochrane Collaboration is an international
organization that aims to help people make
wellinformed decisions about healthcare by
preparing, maintaining and promoting the
accessibility of systematic reviews of the
effects of healthcare interventions”
CC built on 10 Principles
Canadian Cochrane
Network & Centre
•
•
•
•
•
•
•
•
•
•
collaboration
building on the enthusiasm of individuals
avoiding duplication
minimizing bias
keeping up to date
striving for relevance
promoting access
ensuring quality
continuity
enabling wide participation
Organization of the CC
Canadian Cochrane
Network & Centre
•
•
•
•
•
Cochrane Centres (n=12)
Collaborative Review groups (n=51)
Fields (n=11)
Networks (n=1)
Methods working groups (n=10)
Canadian Cochrane
Network & Centre
Cochrane
Centres
• Australasian CC
• Brazilian CC
• Italian CC
• Canadian CC
• Nordic CC
• Chinese CC
• South African CC
• Dutch CC
• UK CC
• German CC
• US CC (Rhode Island,
•Iberoamerican CC
Boston, San Francisco
Branches)
Cochrane Review Groups
Canadian Cochrane
Network & Centre
Acute Respiratory Infections Group
Airways Group
Anaesthesia Group
Back Group
Breast Cancer Group
Colorectal Cancer Group
Consumers and Communication Group
Cystic Fibrosis and Genetic Disorders Group
Dementia and Cognitive Improvement Group
Depression, Anxiety and Neurosis Group
Developmental, Psychosocial & Learning
Problems Group
Drugs and Alcohol Group
Ear, Nose and Throat Disorders Group
Effective Practice and Organisation of Care
Group
Epilepsy Group
Eyes and Vision Group
Fertility Regulation Group
Gynaecological Cancer Group
Haematological Malignancies Group
Heart Group
Hepato-Biliary Group
HIV/AIDS Group
Hypertension Group
Incontinence Group
Infectious Diseases Group
Inflammatory Bowel Disease Group
Injuries Group
Lung Cancer Group
Menstrual Disorders and Subfertility Group
Metabolic and Endocrine Disorders Group
Methodology Review Group
Movement Disorders Group
Multiple Sclerosis Group
Musculoskeletal Group
Musculoskeletal Injuries Group
Neonatal Group
Neuromuscular Disease Group
Oral Health Group
Pain, Palliative and Supportive Care
Peripheral Vascular Diseases Group
Pregnancy and Childbirth Group
Prostatic Diseases and Urologic Cancers Group
Renal Group
Schizophrenia Group
Sexually Transmitted Diseases Group
Skin Group
Stroke Group
Subfertility Group (see Menstrual Disorders)
Tobacco Addiction Group
Upper Gastrointestinal & Pancreatic Diseases Grp
Wounds Group
Cochrane Fields & Network
Canadian Cochrane
Network & Centre
•
•
•
•
•
•
•
•
•
•
•
Cancer Network Field
Child Health Field
Complementary Medicine Field
Health Care of Older People Field
Health Promotion and Public Health Field
Neurological Network Field
Occupational Health Field
Prehospital & Emergency Health Field
Primary Health Care Field
Rehabilitation and Related Therapies Field
Vaccines Field
• Consumer Network
Cochrane Methods Groups
Canadian Cochrane
Network & Centre
•
•
•
•
•
•
•
•
•
•
Applicability and Recommendations
Health Economics
Health-Related Quality of Life
Individual Patient Data Meta-Analyses
Non-randomised Studies
Prospective Meta-Analysis
Qualitative Methods
Reporting Bias Methods
Screening and Diagnostic Tests
Statistical Methods
Cochrane Activities
Canadian Cochrane
Network & Centre
•
•
•
•
Produce and update systematic reviews
Hand search for RCTs
Medline (and others) enhancement
Review methodology
 COCHRANE LIBRARY
The Cochrane Library
Canadian Cochrane
Network & Centre
• unique source of reliable and up-to-date
information on the effects of
interventions in health care
• To provide information and evidence to
support decisions taken in health care
and to inform those receiving care
• Published on a quarterly basis
When should you use the CLIB?
Canadian Cochrane
Network & Centre
For questions on effectiveness
 What is the effectiveness of treatment x
 What is an effective treatment for y
 Is z effective in treating y
 Is z better than x at treating y
Canadian Cochrane
Network & Centre
When not to use the Cochrane
Library
General healthcare questions
•
•
•
•
•
causal, prognosis, epidemiology, etc.
Statistics (prevalence and incidence)
Primary research other than RCTs
Guidelines
Current research
Canadian Cochrane
Network & Centre
The Cochrane Database of Systematic
Reviews (CDSR)
Complete Reviews
 full text, regularly updated systematic reviews of the
effects of health care (2,170 reviews)
 prepared and maintained by the Collaboration Review
Groups
Protocols
 protocols of reviews currently being prepared, incl.
expected date of completion (1,500 protocols)
 includes background, objectives and methods sections
The Database of Abstracts of
Reviews of Effectiveness (DARE)
Canadian Cochrane
Network & Centre
prepared by the National Health Services Centre for Reviews and
Dissemination at the University of York, UK.
Abstracts of quality assessed systematic reviews
 structured abstracts assessing the quality of previously
published SRs & summarizing findings (4,118 reviews)
Other reviews: bibliographic details only
 references to published SRs NOT assessed for quality
(800 reviews)
Canadian Cochrane
Network & Centre
The Cochrane Central Register of
Controlled Trials (CENTRAL)
References
 Reference list of ALL identified published randomized
trials
 Latest issue 427,807 RCTs
(Medline: 97,827 articles identified as RCT publication type)
Canadian Cochrane
Network & Centre
The Cochrane Database of
Methodology Reviews (CDMR)
Complete Reviews
 Full-text SRs of methodological studies (10 reviews)
 Highly structured and systematic, covering a specific and
well-defined area of methodology
 prepared and maintained by the Cochrane Methodology
Review Groups
Protocol
 protocols of reviews currently being prepared, incl.
expected date of completion (8 protocols)
 includes background, objectives and methods sections
Canadian Cochrane
Network & Centre
Cochrane Methodology Register
(CMR)
References
 published reports of empirical studies of methods used in
reviews (5,968 reports)
 methodological studies directly relevant to conducting a
review
Canadian Cochrane
Network & Centre
Health Technology Assessment
Database (HTA)
References
 HTA covers prevention and rehabilitation, vaccines,
pharmaceuticals and devices, medical and surgical
procedures and the systems within which health is
protected and maintained
 Ongoing projects and completed publications from HTA
organizations (4,395 citations)
Canadian Cochrane
Network & Centre
NHS Economic Evaluation Database
(NHS EED)
References
 Structured abstracts of articles reporting economic
evaluations of health care interventions
 Bibliographic details of articles on relevant topics (i.e.,
burden of illness, economic methodology, reviews of
economic evaluations)
 N=15,041 citations
Searching the CLIB
Canadian Cochrane
Network & Centre
Some basics…
ALL contents of ALL records in ALL databases are searched
Ignores: punctuation & numbers
Boolean terms: AND, OR, NEXT, NEAR – within 6 words both
ways, NOT
Restricting searches
 ‘Options’ page and choose desired restrictions
 At the end of the term, add:
:AU – Author
:TI – Title
:ME - MeSH terms
:AB – Abstract
:KY - Keywords
Searching the CLIB
Canadian Cochrane
Network & Centre
MeSH
 Keywords drawn from MeSH thesaurus of U.S. NLM
 Accompanies some but not all records
 Organized hierarchically in ‘trees’
 Permuted Index – an index of all words that appear in the
MeSH thesaurus  used to located specific MeSH terms
Canadian Cochrane
Network & Centre
At McGill University and
affiliated sites:
The pathway to the CLIB is:
McGill HSL library homepage > Databases > >
Cochrane Library > web
(no login or password nec.)
Useful Materials
Canadian Cochrane
Network & Centre
 The Cochrane Library: Self Training Guide
 Interpretation of Odd-ratio diagrams
 WWW links, HTA database
 Cochrane Handbook
 User manual
 Glossary, etc...
DOWNLOAD
http://www.cochrane.org/resources/training.htm
Quality of Cochrane Reviews
• Comparison of 36 Cochrane reviews with 39
paper-based journals
 Cochrane reviews less prone to bias
(Jadad et al. 1998)
– Explicit reporting of eligibility criteria
(35/36 vs 18/39)
– Assessed trial quality (36/36 vs 7/39)
– No language restriction (36/36 vs 32/39)
Quality of Cochrane Reviews
• Also not perfect (Olsen et al. 2001)
• 52 Cochrane reviews from 1998
• 18% - conclusions not backed up by evidence
– All overestimated effect of intervention
Discordant Reviews
•
•
•
•
Direction of effect
Significance
Magnitude of effect
Interpretation of results
 Numerous possible reasons
Sources of discordance
• Different research question
–
–
–
–
Target population
Intervention being studied
Outcome measures
Setting
• Study selection
– Search strategy
– Eligibility criteria
Sources of discordance
• Data extraction
– Methods of measuring outcomes and endpoints
– Extent of human error
• Quality assessment method
– 25 scales and 9 checklists avail for assessing RCT
quality (Moher et al. 1996)
• Inconsistent quality depending on instrument used
– Additional 8 instruments (Juni et al. 1999)
– QUOROM (Quality of Reporting of Meta-Analyses)
= most comprehensive
(Moher et al. 1999, Shea et al. 2001)
Sources of discordance
• Analysis
– Appropriateness of combining results
– Method of synthesis: Descriptive, meta-analysis
– Statistical methods
• Bayesian
• Meta-regression
• Frequentist
• Interpretation of evidence
Decision Algorithm for interpreting discordant
reviews
Jadad, Cook & Browman. A guide to interpreting discordant systematic reviews. CMAJ. 1997 May
15;156(10):1411-6.
Screening for breast cancer
with mammography among
women aged 50-69 years
Report prepared for the Breast Cancer Screening Unit,
Cancer Branch, Health Canada
Ritsuko Kakuma
March 2002
SRs on mammography screening for
breast cancer
• Fletcher et al. Report of the International Workshop
on Screening for Breast Cancer. J Natl Cancer Inst
1993; 85:1644-1656.
• Kerlikowske et al. Efficacy of screening
mammography. A meta-analysis. JAMA 1995;
273:149-154.
• Olsen O & Gøtzsche PC. Is screening for breast
cancer with mammography justifiable? Lancet 2000.
Fletcher et al. review
Research Question:
• To assess current state of knowledge about BRCA
screening, identify knowledge gaps
Study Sample:
• Women aged 40-79 years stratified into 3 groups: 4049, 50-69, 70+
Fletcher et al. review
Methodological issues:
• Int’l Workshop on screening for BRCA in Feb 1993 (by
NCI) – characteristics of participants?
• Search strategy not reported
• Study selection method not described
• Data source: published and unpublished data
provided by the w/s participants
• Validity of trials assessed:
–
–
–
–
–
Randomization, Compliance, Contamination
Quality and frequency of screening test
Adequacy of FU
Validity of outcome assessment
Generalizability of results
Fletcher et al. review
Results/Conclusions:
• 7 trials identified
• For age 50-69 age group, only 5 trials provided data
– New York, Two-County, Malmo, Edinburgh and Canada
• No statistical pooling of data
• Together, Fletcher et al. concluded that there was
substantial benefit of screening on BRCA mortality
 reduction of 30% after 10-12 years of FU
Kerlikowske et al. review
Research Question:
• To determine efficacy of screening in reducing BRCA
mortality by age, # MM views per screen, screening
interval and duration of FU
Sample:
• Women aged 40-74 years stratified into 3 groups: 4049, 40-74, 50-74
Kerlikowske et al. review
Methodology:
• Search: MEDLINE, manual (reference lists) and
consultations with colleagues and experts
– Search terms, dates covered
•
Study Selection
– English language restriction, explicit inclusion criteria
– Not restricted to RCTs
• Data extraction
– 2 indep extractors & 3rd for resolving discrepancies
– Published and unpublished data used
• Quality assessment not reported..not done?
• Analysis
– Pooled data of all included trials using fixed effects
Kerlikowske et al. review
Results / Conclusions:
• 9 studies (7 RCTs & 2 CCS) have data on 50-74 year
age group
Comparison*
Relative Risks
RCT vs. Case-control studies
0.77 (0.69-0.87) vs. 0.45 (0.29-0.70)
No. of MM views: 2-views vs. 1-view
0.70 (0.58-0.84) vs. 0.83 (0.71-0.97)
Screening interval: 12 mo. vs. 18-33 mo.
0.77 (0.59-1.0) vs. 0.77 (0.68-0.88)
FU duration: 7-9 yrs vs. 10-12 yrs
0.73 (0.63-0.84) vs. 0.76 (0.67-0.87)
Screening duration: 3-5 yrs vs. 8-10 yrs
0.76 (0.62-0.95) vs. 0.78 (0.67-0.90)
CBE + MM vs. MM alone
0.80 (0.66-0.98) vs. 0.76 (0.65-0.88)
Study start date: Pre-1980 vs. Post-1980
0.76 (0.67-0.87) vs. 0.83 (0.63-1.10)
Kerlikowske et al. review
Results / Conclusions:
• All studies – beneficial effect of screening
(4 statistically significant)
• All trials included in the analysis (not CCSs)
• MM screening reduced BRCA mortality by 27% in this
age group after 7- 9 years and 24% after 10-12 years
regardless of # MM views, screening interval,
duration of screening or addition of CBE
Olsen & Gøtzsche review
Research Question:
• To assess effect of MM screening for BRCA on
mortality and morbidity
• Review methodological quality of trials – focus on 3
most important sources of bias
– Randomization
– blind outcome assessment
– post-Rx exclusions
Sample:
• Women without previously diagnosed BRCA on
women aged <50 years and 50+ years
Olsen & Gøtzsche review
Methodology:
• Search: MEDLINE, common author names search,
reference lists, specialized trials registers, letters,
abstracts, grey literature
– Search terms, dates covered
• Study Selection:
– Explicit inclusion criteria (RCT / quasi-RCT on MM screening vs.
no MM screening)
• Two indep reviewers for study selection, quality
assessment & data extraction
– Authors of studies contacted for missing information
• Quality assessment: High, Medium, Poor or Flawed
– Rx process, baseline comparability, post-Rx exclusions,
Consistency of number of women randomized
• Analysis
– Intent-to-Treat analysis, with fixed effect unless heterogeneous
– Stratified by quality (excluded flawed)
Olsen & Gøtzsche review
Results / Conclusions:
• 7 trials identified, none with high quality (2 medium, 3
poor, 2 flawed)
• Total mortality
– no impact of screening after 7- and 13 yrs FU for both medium
and poor quality trials
• BRCA mortality
– no beneficial effect of screening in medium quality trials
– Beneficial effect among POOR quality trials
pooled RR=0.69, CI 0.55-0.80 at 7 years
pooled RR=0.64, CI 0.54-0.76 at 13 years
• Interventions for screened group higher
– Over-diagnosis, increased use of more aggressive treatment
such as mastectomies and tumorectomies (20-30%)
Decision Algorithm for interpreting discordant
reviews
Identified Trials
•Sampling Randomization: Canadian study recruited volunteers, other used population-based lists
•Assessment of pre-existing BRCA: pre- vs. post-Rx
Trials: Methodological issues
Canadian
Efficacy trial, not effectiveness (women = screened, not ‘invited’ to be screened)
CBE in control group may dilute effect of screening (MM+CBE vs. CBE)
Imbalance in rate of advanced cancers at Rx
Malmö
Inconsistent numbers across multiple reports – missing unaccounted for
Date of entry different for the 2 groups (date of Rx vs. date of invitation to
screening)
Contamination: 24% of controls had MM
Continuation of study: not randomized, no baseline information
Stockholm
Inconsistent numbers across multiple reports– missing unaccounted for
Non-independence of results of 2 sub-trials (2 smaller trials with overlapping
controls)
Imbalanced post-Rx exclusions, Contamination (Screening for controls began as
early as year 3)
Göteborg
Information on Rx not described adequately
Contamination (Screening for controls began 3rd and 6th years)
No separate data on 50+ age group
Trials: Methodological issues
TwoCounty
Age imbalance of 5-months – older in screening group (clinically signif?)
Cluster Rx method not described adequately
Contamination (Screening for controls began after a few years into study)
New York
Baseline imbalance in previous lump, menopause and education
Post-Rx exclusion ID method of current BRCA differed --self-report vs. screen
Outcome assessment (cause of death): 72% non-blind (differential
misclassification or true effect?)
Edinburgh
Inadequate Rx  Baseline imbalances – screened group had higher SES
Change in allocation status after Rx
Imbalanced post-Rx exclusions
Contamination (Screening for controls began in year 10 of FU)
Differential eligibility criteria and entry dates for the 2 groups
Impact on effect measure
• Contamination & Compliance : underestimate
– Introduction of screening in controls
– Screened vs. invited to be screened
• Post-Rx exclusions: overestimate
– Identification of BRCA at baseline more complete for
screened group..first round of screening used to exclude
– Dropouts and losses: equally distributed?
• Adequacy of Randomization: overestimate
– Typically overestimate treatment effect by 30-40%
(Schulz et al. 1995; Moher et al. 1998; Kjaergard et al. 2001)
• Outcome Assessment: overestimate
– Misclassification
– Determination of cause of death may be difficult in light of
multiple illnesses
Comparability of Trials
• Same research question?
– Effectiveness vs. efficacy
• Same Target population?
• Appropriate study sample?
• Intervention?
– Screening modality, frequency of screening
• Outcome assessment
– Cause of death identified the same way?
• Analytical methods
Fletcher et al.
 No precise research question
 Inadequate description of methods used
– Search strategy, trial selection, data extraction
 Acknowledgement of difference in quality of trials but
conclude based on ALL evidence
 Source of some data unclear
 Quality assessment carried out
 NOT synthesizing data = appropriate
• But evidence does not support their conclusion that
BRCA mortality is reduced by 30% as a result of
screening
Kerlikowske et al.
 Clear research question
 Explicit reporting of methods used
 No indication that quality assessment was done
 Some numbers extracted don’t match reference cited
 Missing key articles for trials (I.e., original report)
• Despite good reporting of methods, caution needed
in interpreting results because of the questionable
nature of data used
Olsen and Gotzsche




Clear research question
Explicit reporting of methods used
Strong methodology
Addressed important question:
Reduced BRCA mortality not necessarily equal to
improved survival
 Did not assess baseline comparability stratified by
age groups, which is how the data was analyzed
• Most rigorous and properly done review of the 3
• Brought potential adverse effects of screening to
forefront
Conclusions of assessment:
• All existing evidence has methodological issues
• No conclusive evidence exists to support MM
• Other issues:
– Accuracy of MM (false positives?)
– Detection of slow-growing tumors that may never develop
into invasive cancers
– Psychological impact of positive results
• US and Canada continue to recommend MM
screening for women in this age group
Overall conclusions
• MA not appropriate in many cases
• Must be able to identify the potential
sources of bias in both the systematic
review as well as the studies included in
the review
• Cochrane reviews are not always
perfect but generally better than nonCochrane reviews