Transcript Systematic Reviews - Royal Devon and Exeter Hospital

Objectives
 Identify the key elements of a good
randomised controlled study
 To clarify the process of meta analysis
and developing a systematic review
 To use this to critically evaluate a
Cochrane review
 Answer a clinically relevant problem
using information from a systematic
review
RCT – key issues
• Allocation – method of randomisation
(to minimise confounding)
• Blinding – patient, clinician and study
personnel
(to minimise observer and recall bias)
• Follow-up (adequate; complete; differences
between those followed up and those lost to
follow-up)
• Intention to treat analysis
Checklist for appraising RCTs
• Was assignment of patients to treatment
randomised?
• Was randomisation concealed?
• Were the groups similar at the start of the
trial?
• Was follow-up of patients sufficiently long and
complete?
• Were all patients analysed in the groups to
which they were randomised (intention to
treat)?
• Were patients, clinicians and study personnel
kept blind to treatment?
• Were groups treated equally, apart from the
experimental therapy?
Overview of a systematic review
• ASK a clear and focussed question
– How easy is this?
• SEARCH for relevant studies
– ALL studies, anywhere, ever? Does systematic
have to be exhaustive?
• APPRAISE the quality and relevance
– Structure depends on the type of studies
• SYNTHESISE
– Meta-analysis or narrative synthesis
• INTERPRET
– In relation to the question asked (policy or clinical)
Correspondingly…
critical appraisal
• Did the review ask a clearly focussed
question?
• Are the findings valid?
–
–
–
–
Did they find all the relevant studies?
Was quality appraised?
Were the study findings combined appropriately?
Were the study results consistent?
• What are the findings?
• Will the findings help me with my patient (or
policy problem)?
Finding studies…
• Electronic databases
– General
• Medline
• Embase
– Methodologically specific
• Cochrane Library (Reviews and CCTR)
• DARE, HTA database, INAHTA, NHS EED
– Disease area specifics
• Psychlit
• Cancerlit
– “Grey literature”
• Hand searching
• Citation searching
Getting the right studies…
• PUBLICATION BIAS: Systematic exclusion of
studies with (usually) negative findings
• DUPLICATE PUBLICATION BIAS: Multiple
publications of positive trials leads to doublecounting in reviews
• CITATION BIAS: Positive studies are cited more
frequently
• LANGUAGE BIAS: English and positive results
• TIME LAG BIAS: Negative studies take
longer to be published
• RETRIEVAL BIAS: Method of
searching produces a systematic error –
high impact journals are more likely to
report positive findings and to be listed
in databases
Assessing publication (and
similar) bias(es)
• Were the searches comprehensive?
• Evidence of “missing studies”
• Look at the FUNNEL PLOT
Biased pooled effect size
Study
size
Treatment better
Control better
Effectiveness
Critical appraisal
•
Not going to cover in detail
• Some explicit method is required
– General “systematicity”
• Transparency
• Reducing errors and bias
– Lots of issues!
• One or two reviewers?
• Measure agreement?
• Scores or not?
• A tension:
– Identify all studies? But…
– Include only the best quality??
• Methodological quality is important as
possible reason for differences between
studies
Findings of review
Forest Plot of Amenorrhoea
following Microwave Endometrial Ablation
Odds ratio
(95% CI)
Study
Cooper
0.96 (0.58,1.59)
Microsulis
1.47 (0.92,2.34)
Meyer
0.56 (0.30,1.04)
Romer
1.56 (0.24,9.91)
Soysal
0.58 (0.18,1.93)
Favours control
Favours treatment
0.1
1
10
Odds ratio
Heterogeneity
• Are the studies all measuring the
same thing?
• If not, then the pooled estimate may not
be informative
• Heterogeneity can due to a variety of
reasons
Sources of heterogeneity
• Chance (i.e. sampling error)
• Clinical
– Different doses or duration of treatment or
co-interventions
– Different populations (i.e. risk of events)
– Different outcome measures
• Methodological
– Blinding
– Methods of analysis
– Publication bias
Statistical significance and
heterogeneity
• Studies can be tested statistically for
heterogeneity to see if results are
significantly different
• P value of 0.05 or lesser indicates
statistical significance for heterogeneity
• If significant for heterogeneity, then
need to consider whether results can be
pooled
Summary of critical appraisal
of a systematic review
• Was there a clear question?
• Did they find all the relevant studies?
• Was the quality of the studies considered?
Was the quality so poor that the review is
suspect?
• What are the results?
• Is there important heterogeneity? What does
it mean?
• What are the implications of the review
findings for my clinical or policy question?
Are they good questions?
 Appropriate to compare with non opiate
analgesics or sedatives?
 Outcome measures appropriate?
 Which outcomes most relevant (in
order)?
 Which outcomes easiest to measure?
 Any difficulties with some of these
outcomes?
Appropriate search
strategy?
Were the Cochrane neonatal group
guidelines followed in relation to




Blinding of randomisation
Blinding of intervention
Completeness of follow up
Blinding of outcome measurement –
see handout
Appropriate search
strategy?
 What is a quasi randomised study?
 What is Cochrane’s central register of
controlled studies?
 Were non English studies included?
 Was there risk of publication bias? How
to overcome this.
 Was more than 1 reviewer involved?
How were differences resolved?
Data Analysis/ Results
 Look at meta-analysis of PIPP scores (figure 01.01)
Do you think the all studies result shows a
satisfactory forest plot?
 What is your interpretation of the results of the other
methods of assessing pain (figure 01.04)
 What is the relevance of a reduction of 2 in the PIPP?
(see handout of PIPP)
 What are the differences in execution and reporting
of trials which authors say will interfere with applying
this?
Data Analysis/ Results
 What do they mean that heterogeneity
was high in all analyses of pains?
 What are sources of heterogeneity for
PIPP?
 What are the other sources of
heterogeneity?
 Do you think there is sufficient evidence
to recommend morphine over
midazolam for sedation?
Application
 How important are the results?
 How applicable are the results?
 What do you do?