Transcript How to review a diagnostic study

How to review a
diagnostic study
CCBS Study Afternoon
12 August 2008
[email protected]
What is a diagnostic study?
• Accuracy of an index
test compared to
reference standard
• Accuracy very often
given as sensitivity
and specificity
Disease
positive
Disease
negative
Test
positive
TP = no. of
true
positives
FP = no. of
false
positives
Test
negative
FN = no. of
false
negatives
TN = no. of
true
negatives
TP + FN =
no. of
disease +ve
patients
FP + TN =
no. of
disease -ve
patients
Sensitivity = TP/No. of disease +ve patients
Specificity = TN/No. of disease –ve patients
Factors affecting
sensitivity and specificity
intrinsic
performance
of test
patients
sensitivity
and
specificity
Bias due to
(de)selection of
patients
Bias due to methods
of conducting tests
Bias due to
misanalysis of
data
How patient selection goes wrong
Recruitment stage
patients selected
because they do or
do not have the disease
Ideal patients =
people who would
get the test in
real life to
establish
diagnosis
Patients included
in the analysis
patients selected
for convenience
Patients included
in reference standard
testing depending on
result of index test, i.e.
Verification bias
Testing stage
How testing goes wrong
Problem
Affect on sens
and spec
Lack of blinding
Overestimation
Workup bias – patients not getting the same Depends
tests (cf verification bias)
Incorporation bias – reference standard
uses info from index test
Overestimation
Inaccurate reference standard
Depends
Disease progression – time between tests
allows disease to change
Underestimation
Knowing / not knowing clinical data
Depends
How the analysis goes wrong (1)
Reference standard results
Reference standard results
Pos
Neg
TP
FN
Neg
Pos
TP
FP
???
???
???
Neg
FN
TN
Neg
FP
TN
Index test results
Index test results
Pos
Pos
How analysis goes wrong (2)
Problem
Consequence
Not correcting for
verification bias
overestimate sensitivity,
underestimate specificity
Not correcting for
inaccurate reference
standard
Overestimation or
underestimation of both
sensitivity and specificity
The methods for dealing with
these are complicated.
Another problem
• “However, evaluations were hampered
because many reports lacked information
on key elements of design, conduct and
analysis of diagnostic studies.” Bossuyt et al.
The STARD Statement
http://www.stard-statement.org/
Practicalities of reviewing (1)
Abstract inclusion/exclusion criteria
Full-text inclusion/exclusion criteria
Practicalities of reviewing (2)
QUADAS checklist for diagnostic studies
QUADAS:
See Whiting P et al. BMC Med
Res Methodol 2003;3:25
http://www.biomedcentral.com/14
71-2288/3/25
Data extraction
Document everything
100%
80%
no
60%
unclear
40%
yes
20%
0%
1
2
3
4
5
6
7
8
9 10 11 12 13 14
QUADAS Item
Meta-analysis
Aim:
to summarise the
two-dimensional
data
FREE
SOFTWARE!
http://www2.napier.ac.uk/depts/fhls/diagmeta/
Example 1
Example 2
References
Loong T-W. Understanding sensitivity and specificity with the right
side of the brain. BMJ 2003;327:716-719.
STARD Statement
http://www.stard-statement.org/
QUADAS checklist
Whiting P et al. BMC Med Res Methodol 2003;3:25
(Also available in NCC HTA monograph)
The Magnificent ROC
http://www.anaesthetist.com/mnm/stats/roc/Findex.htm
Summary ROC curves
http://srdta.cochrane.org/Files/Website/Presentations/
4_AdvancedAnalysis_SaoPaulo.pdf
DiagMeta website
http://www2.napier.ac.uk/depts/fhls/diagmeta/
[email protected]