Transcript Critical appraisal of quantitative research

Introduction to
Critical Appraisal :
Quantitative Research
South East London
Outreach Librarians
January 2008
Learning objectives
• Understand the principles of critical
appraisal and its role in evidence based
practice
• Be able to appraise quantitative research
and judge its validity
• Be able to assess the relevance of
published research to your own work
Daily Mail exercise
• Would you treat a patient based on this
article? Why?
• Validity
• Reliability
• Transferable to practice
What is evidence based
practice?
Evidence-based practice is the integration of
• individual clinical expertise
with the
• best available external clinical evidence
from systematic research
and
• patient’s values and expectations
The evidence-based
practice process.
• Decision or question arising from a
patient’s care.
• Formulate a focused question.
• Search for the best evidence.
• Appraise the evidence.
• Apply the evidence.
Why does evidence from
research fail to get into practice?
• 75% cannot understand the statistics
• 70% cannot critically appraise a
research paper
•
Using research for Practice: a UK experience of the barriers scale.
Dunn, V. et al.
What is critical appraisal?
• Weighing up evidence to see how useful it
is in decision making
• Balanced assessment of benefits and
strengths of research against its flaws and
weaknesses
• Assess research process and results
• Skill that needs to be practiced by all
health professionals as part of their work
What critical appraisal is NOT
• Negative dismissal of any piece of
research
• Assessment of results alone
• Based entirely on statistical analysis
• Only to be undertaken by researchers/
statisticians
Why do we need to
critically appraise?
• “It usually comes as a surprise to students
to learn that some (the purists would say
99% of) published articles belong in the
bin and should not be used to inform
practice” (Greenhalgh 2001)
• Find that 1% - save time and avoid
information overload
How do I appraise?
• Mostly common sense.
• You don’t have to be a statistical expert!
• Checklists help you focus on the most
important aspects of the article.
• Different checklists for different types of
research.
• Will help you decide if research is valid
and relevant.
Research methods
Quantitative
• Uses numbers to
describe and analyse
• Useful for finding
precise answers to
defined questions
Qualitative
• Uses words to
describe and analyse
• Useful for finding
detailed information
about people’s
perceptions and
attitudes
Levels of quantitative
evidence.
(In order of decreasing scientific validity.)
•
•
•
•
•
•
Systematic reviews.
Randomized controlled trials.
Prospective studies (cohort studies).
Retrospective studies (case control).
Case series and reports
Opinions of respected authorities.
Systematic Reviews.
• Thorough search of literature carried
out.
• All RCTs (or other studies) on a
similar subject synthesised and
summarised.
• Meta-analysis to combine statistical
findings of similar studies.
Randomised Controlled
Trials (RCTs)
• Normal treatment/placebo versus new
treatment.
• Participants are randomised.
• If possible should be double-blinded.
• Intention to treat analysis
Cohort studies
•
•
•
•
•
prospective
groups (cohorts)
exposure to a risk factor
followed over a period of time
compare rates of development of an
outcome of interest
• Confounding factors and bias
Case control studies
• Retrospective
• Subjects confirmed with a disease (cases)
are compared with non-diseased subjects
(controls) in relation to possible past
exposure to a risk factor.
• Confounding factors and bias
Appraising original research
Are the results valid?
• Is the research question focused?
• Was the method appropriate?
• How was it conducted, e.g. randomisation,
blinding, recruitment and follow up?
What are the results?
• How was data collected and analysed?
• Are they significant?
Will the results help my work with patients?
Appraising systematic
reviews.
In addition to the above:
• Was a thorough literature search
carried out ?
• Publication bias - papers with more
‘interesting’ results are more likely to be:
– Submitted for publication
– Accepted for publication
– Published in a major journal
– Published in the English language
Reviews in general
medical journals
• 50 reviews in 4 major journals 1985-6
• No statement of methods
• Summary inappropriate
• “Current systematic reviews do not
routinely use scientific methods to identify,
assess and synthesise information”
(Mulrow, 1987)
Is the research
question focused?
•
•
•
•
Patient
Intervention
Comparison
Outcome
(e.g. child)
(e.g. MMR vaccine)
(e.g. single vaccines)
(e.g. autism)
Are results significant?
•
•
•
•
How was data collected?
Which statistical analyses were used?
How precise are the results?
How are the results presented?
Intention to treat analyses
• Analysing people, at the end of the trial, in
the groups to which they were
randomised, even if they did not receive
the intended intervention
Statistical analyses
Odds ratios, absolute and relative
risks/benefits
• The likelihood of something happening vs
the likelihood of something not happening
Numbers needed to treat (NNT)
• The number of people you would need to
treat to see one additional occurrence of a
specific beneficial outcome
Odds Ratio Diagrams.
(Blobbograms or Forest Plots.)
Odds Ratio Diagrams
• Line of no effect – no difference between
treatment and control group
• Result (blob) to the Left of the line of no
effect = Less of the outcome in the
treatment group.
• Result to the Right of the line = More of
the outcome.
• BUT - Is the outcome good or bad?
Cardiac deaths –
Less = good
Smoking cessation –
More = good
Confidence Intervals.
• Longer confidence interval = less
confident of results – wider range.
• Shorter confidence interval = more
confident – narrower range.
• Crosses line of no effect/no
significance = Inconclusive results.
Confidence intervals
P Values.
• P stands for probability - how likely is the
result to have occurred by chance?
• P value of less than 0.05 means likelihood
of results being due to chance is less than
1 in 20 = “statistically significant”.
• P values and confidence intervals should
be consistent
Number Needed to Treat
• The number of people you would need to
treat to see one additional occurrence of a
specific beneficial outcome.
• The number of patients that need to be
treated to prevent one bad outcome.
• The NNT can be calculated by finding the
Absolute Risk Reduction (ARR)
Events or outcomes are used for reporting results. The
event rate is the proportion of patients in a group in
whom the event is observed
Outcome event
Total
Yes
No
Experimental group
a
b
a+b
Control group
c
d
c+d
a+c
b+d
a+b+c+d
Total
CER and EER
• Control Event Rate (CER) is the proportion of
patients in the control group in whom an
event is observed.
CER = c/(c+d)
• Experimental Event Rate (EER) is the
proportion of patients in the experimental in
whom an event is observed.
EER = a/(a+b)
AAR & NNT
• Absolute Risk Reduction is the difference
between the Control Event Rate (CER) and
the Experimental Event Rate (EER).
ARR = CER – EER
• Number needed to treat (NNT)
NNT = 1/ARR
Outcome
event
Total
Yes
No
Experimental
group
3
7
10
Control group
5
5
10
Total
8
12
20
Answers
•
•
•
•
•
•
•
•
•
•
What is the event ?
Lack of concentration and sleeping
What is the control event rate (CER)?
5/10 = 0.50
What is the experimental event rate (EER)?
3/10 = 0.30
Calculate the absolute risk reduction (ARR)
0.50 – 0.30 = 0.20
What is the number needed to treat (NNT)?
1.00/0.20 = 5
Are results relevant?
• Can I apply these results to my own
practice?
• Is my local setting significantly different?
• Are these findings applicable to my
patients?
• Are findings specific/detailed enough to be
applied?
• Were all outcomes considered?
The good news!
• Some resources have already been
critically appraised for you.
• An increasing number of guidelines
and summaries of appraised
evidence are available on the
internet.
Summary.
• Search for resources that have already been
appraised first, e.g. Guidelines, Cochrane
systematic reviews.
• Search down through levels of evidence, e.g.
systematic reviews, RCTs.
• Use checklists to appraise research.
• How can these results be put into practice?