Transcript Changing Your World with the National Programme for IT

Critical Appraisal Skills
quantitative reviews
Pippa Orr
Knowledge Support Librarian
With acknowledgements to CASP for their slides
North Cumbria Informatics Service
Critical Appraisal Skills
Programme (CASP)
Critical appraisal is the
process of weighing up
evidence to see how
useful
it is in decision making
http://www.phru.nhs.uk/casp/critical_appraisal_tools.htm
North Cumbria Informatics Service
Effectiveness of Health Care
• doing the right thing
•
•
•
•
•
to the right patient
in the right way
at the right time
at the right cost
in the right place
North Cumbria Informatics Service
Kinds of evidence
• Descriptive
– cross-sectional, longitudinal
• Analytic
– case-control study
– cohort study
• Experimental
– randomized controlled trial
North Cumbria Informatics Service
Hierarchy of evidence
North Cumbria Informatics Service
Why does good evidence from
research fail to get into practice??
- 75% cannot understand the
statistics
- 70% cannot critically appraise a
research paper
Using Research for Practice: A UK Experience of the barriers scale
Dunn V, Crichton C, Williams K, Roe B, Seers K
North Cumbria Informatics Service
Critical appraisal helps the reader
of research ………...
• Decide how trustworthy a piece of
research is (validity)
• Determine what it is telling us (results)
• Weigh up how useful the research
will be (relevance)
North Cumbria Informatics Service
Primary Research Evidence:
Randomised Controlled Trials (RCT)
Robust randomisation procedures:
•to ensure that the variables are equal
in both groups
•to remove all bias
•to ensure that the results are
generalisable
North Cumbria Informatics Service
Randomised controlled trial
new treatment
group 1
Outcome
group 2
Outcome
population
control treatment
North Cumbria Informatics Service
Blinding
Blinding = participants don’t
know what intervention they are
getting
Double blinding = those giving
the intervention don’t know what
the participant is receiving
North Cumbria Informatics Service
Loss to follow-up
It is important to ensure that all
those that are randomised into the
trial are followed up to the trial’s
conclusion
North Cumbria Informatics Service
Intention to treat analysis
Analysing people, at the end of the
trial, in the groups to which they
were randomised, even if they did
not receive the intended
intervention.
North Cumbria Informatics Service
Types of review:
Reviews
Systematic reviews
Meta-analysis
North Cumbria Informatics Service
Publication bias
Papers with "interesting" results are (or may be)
more likely to be:
• submitted for publication
• accepted for publication
• published in a major journal and in English
Language
• quoted by authors
• quoted in newspapers
North Cumbria Informatics Service
Odds Ratio, Relative Risk
Measures of risk
The likelihood of something
happening
V
The likelihood of something not
happening
North Cumbria Informatics Service
Odds ratio (OR)
• The odds of an event happening
in the experimental group
expressed as a proportion of the
odds of an event happening in
the control group
• The closer the OR is to 1, the
smaller the difference in effect,
i.e. no effect: OR = 1
North Cumbria Informatics Service
Confidence intervals/ limits
• Presents the range of likely effects
• The 95% confidence interval, for example,
includes 95% of results from studies of the
same size and design in the same
population
• This is close, but not identical, to saying
that the true size of effect (never exactly
known) has 95% chance of falling within
the confidence interval
• The narrower/ shorter the confidence
interval, the more precise/ confident we
can be about the estimate
North Cumbria Informatics Service
Forest plots
• Common approach to presenting
the results of a meta-analysis
• Also known as a ‘blobbogram’ or
‘odds ratio diagram’
• Graphical representation of
individual trial results included
in a review, together with the
combined meta-analysis result
North Cumbria Informatics Service
line of no effect
confidence interval
meta-analysis result
North Cumbria Informatics Service
p-value
• The probability (ranging from 0 to 1) that
the results observed in a study (or results
more extreme) could have occurred by
chance if in reality the null hypothesis was
true, ie if you did nothing.
• If this probability is less than 1/20 (which
is when the p value is less than 0.05), then
the result is conventionally regarded as
being “statistically significant”.
North Cumbria Informatics Service
The p-value in a nutshell
Could the result have occurred by chance?
The result is likely
to be due to chance
The result is unlikely
to be due to chance
0
1
p < 0.05
p > 0.05
a statistically
significant result
not a statistically
significant result
p = 0.05
1
20
or 1 in 20
result fairly unlikely to
be due to chance
North Cumbria Informatics Service
p = 0.5
1
2
or 1 in 2
result quite likely to be
due to chance
Number needed to treat
Is the number of people you would
need to treat with a specific
intervention to see one additional
occurrence of a specific beneficial
outcome.
North Cumbria Informatics Service
Critical appraisal:
questions to apply to reviews
• is it trustworthy?
validity
• what does it say?
results
• will it help?
relevance
North Cumbria Informatics Service