Transcript Evaluating Diagnostic Tests - Kashan University of Medical

Evaluating
Diagnostic Tests
Payam Kabiri, MD. PhD.
Clinical Epidemiologist
Tehran University of Medical Sciences
Seven question to evaluate the
utility of a diagnostic test
Can the test be reliably performed?
 Was the test evaluated on an appropriate
population?
 Was an appropriate gold standard used?
 Was an appropriate cut-off value chosen
to optimize sensitivity and specificity?

Seven question to evaluate the
utility of a diagnostic test
What are the positive and negative
likelihood ratios?
 How well does the test perform in specific
populations?
 What is the balance between cost of the
disease and cost of the test?

Which one of these test is the best
for SLE Dx?
Test
Sensitivity%
Specificity%
ANA
99
80
dsDNA
70
95
ssDNA
80
50
Histone
30-80
50
Nucleoprotein
58
50
Sm
25
99
RNP
50
87-94
PCNA
5
95
Diagnostic Tests Characteristics
 Sensitivity
 Specificity
 Predictive
Value
 Likelihood Ratio
5
Validity of Screening Tests
True Disease Status
+
-
+
a
b
-
c
d
Sensitivity: The probability of testing
positive if the disease is truly present
Sensitivity = a / (a + c)
6
Validity of Screening Tests
True Disease Status
+
-
+
a
b
-
c
d
Specificity: The probability of screening
negative if the disease is truly absent
Specificity = d / (b + d)
7
Two-by-two tables can also be used for
calculating the false positive and false
negative rates.
 The false positive rate = false positives /
(false positives + true negatives). It is also
equal to 1- specificity.

The false negative rate = false negatives /
(false negatives + true positives). It is also
equal to 1 – sensitivity.
 An ideal test maximizes both sensitivity
and specificity, thereby minimizing the
false positive and false negative rates.

Validity of Screening Tests
Breast Cancer
Physical Exam
and Mammography
+
-
+
-
132
983
45
63650
Sensitivity: a / (a + c)
Sensitivity =
Specificity: d / (b + d)
Specificity =
10
Validity of Screening Tests
Breast Cancer
Physical Exam
and Mammography
+
-
+
-
132
983
45
63650
Sensitivity: a / (a + c)
Sensitivity = 132 / (132 + 45) = 74.6%
Specificity: d / (b + d)
Specificity = 63650 / (983 + 63650) = 98.5%
11
2 X 2 table
+
Disease
-
Positive
predictive
value
+
Test
Sensitivity
Natural Frequencies Tree
Population
100
In Every 100 People, 4 Will Have The Disease
Population
100
Disease +
Disease -
4
96
If these 100 people are representative of the population at
risk, the assessed rate of those with the disease (4%)
represents the PREVALENCE of the disease – it can also be
considered the PRE-TEST PROBABILITY of having the disease
OF THE 4 PEOPLE WITH THE DISEASE, THE TEST
WILL DETECT 3
Population
100
Disease +
Disease -
4
96
Test +
Test -
3
1
In other words, the
sensitivity is 75%
AMONG THE 96 PEOPLE WITHOUT THE DISEASE, 7
WILL TEST POSITIVE
In other words, the
specificity is 93%
Population
100
Disease +
Disease -
4
96
Test +
Test -
Test +
Test -
3
1
7
89
AMONG THOSE WHO TEST POSITIVE, 3 IN 10 WILL
ACTUALLY HAVE THE DISEASE
This is also the
POST-TEST PROBABILITY of having
the disease
Population
100
Disease +
Disease -
4
96
Test +
Test +
3
7
POSITIVE
PREDICTIVE
VALUE = 30%
Test -
Test -
1
89
AMONG THOSE WHO TEST NEGATIVE, 89 OF 90 WILL
NOT HAVE THE DISEASE
Population
100
Disease +
Disease -
4
96
Test +
Test +
3
7
NEGATIVE
PREDICTIVE
VALUE = 99%
Test -
Test -
1
89
CONVERSELY, IF SOMEONE TESTS NEGATIVE, THE
CHANCE OF HAVING THE DISEASE IS ONLY 1 IN 90
Population
100
Disease +
Disease -
4
96
Test +
Test +
3
7
Test -
Test -
1
89
PREDICTIVE VALUES AND CHANGING PREVALENCE
Population
1000
Disease +
Disease -
4
996
Prevalence reduced by an order
of magnitude from 4% to 0.4%
PREDICTIVE VALUE AND CHANGING PREVALENCE
Population
Sensitivity and
Specificity
unchanged
1000
Disease +
Disease -
4
996
Test +
Test +
3
70
Test -
Test -
1
926
POSITIVE PREDICTIVE VALUE AT LOW PREVALENCE
Population
Previously, PPV
was 30%
1000
Disease +
Disease -
4
996
Test +
Test +
3
70
POSITIVE
PREDICTIVE
VALUE = 4%
Test -
Test -
1
926
NEGATIVE PREDICTIVE VALUE AT LOW PREVALENCE
Population
Previously, NPV
was 99%
1000
Disease +
Disease -
4
996
Test +
Test +
3
70
NEGATIVE
PREDICTIVE
VALUE >99%
Test -
Test -
1
926
Prediction Of Low Prevalence
Events
Even highly specific tests, when applied to
low prevalence events, yield a high
number of false positive results
 Because of this, under such
circumstances, the Positive Predictive
Value of a test is low
 However, this has much less influence on
the Negative Predictive Value

Relationship Between
Prevalence and Predictive Value
Predictive Value
1
Difference between
PPV and NPV
relatively small
0.8
0.6
0.4
PPV
NPV
0.2
Difference between
PPV and NPV
relatively large
0
0.05
0.2
0.4
0.6
0.8
0.95
Pre-test Probability (Prevalence)
Based on a test with 90% sensitivity and 82% specificity
100
90
80
70
60
50
40
30
20
10
0
PPV
NPV
0.
05
0.
0. 1
15
0.
0. 2
25
0.
0. 3
35
0.
0. 4
45
0.
0. 5
55
0.
0. 6
65
0.
0. 7
75
0.
0. 8
85
0.
0. 9
95
Predictive Value
Relationship Between
Prevalence And Predictive Value
Prevalence
Based on a test with 75% sensitivity and 93% specificity
Performance of A Test With
Changing Prevalence
1
A : Sensitivity =
Specificity = 0.9
LR+ = 9.0
PROBABILITY
POST-TEST POST-TEST
0.9
0.8
0.7
B : Sensitivity =
Specificity = 0.7
LR+ = 3.0
0.6
0.5
0.4
C : Sensitivity =
Specificity = 0.5
LR+ = 1.0
A (90%)
0.3
B (70%)
0.2
C (50%)
0.1
0
0
0.2
0.4
0.6
PRE-TEST PROBABILITY
0.8
1
2 X 2 table
Yes
No
Yes
3
7
10
a+b
No
1
4
89
96
90
c+d
TEST
DISEASE
a+c
b+d
Total
a b
c D
d
Total
100
a+b+c+d
Sensitivity
Yes
No
Yes
3
7
10
a+b
No
1
4
89
96
90
c+d
TEST
DISEASE
a+c
b+d
Total
a b
c D
d
Total
FALSE
NEGATIVES
100
a+b+c+d
Sensitivity
The proportion of people with the diagnosis (N=4) who are
correctly identified (N=3)
Sensitivity = a/(a+c) = 3/4 = 75%
Specificity
Yes
No
Yes
3
7
10
a+b
No
1
4
89
96
90
c+d
TEST
DISEASE
a+c
b+d
Total
a b
c D
d
Total
FALSE
POSITIVES
100
a+b+c+d
Specificity
The proportion of people without the diagnosis (N=96) who
are correctly identified (N=89)
Specificity = d/(b+d) = 89/96 = 93%
Value of a diagnostic test depends
on the prior probability of disease






31
Prevalence
(Probability) = 5%
Sensitivity = 90%
Specificity = 85%
PV+ = 24%
PV- = 99%
Test not as useful
when disease unlikely






Prevalence
(Probability) = 90%
Sensitivity = 90%
Specificity = 85%
PV+ = 98%
PV- = 49%
Test not as useful
when disease likely
A Test With Normally
Distributed Values
% of Group
Test cut-off
Assessing the performance
of the test assumes that
these two distributions
remain constant. However,
each of them will vary
(particularly through
spectrum or selection bias)
NON-DESEASED
DISEASED
Positive
Negative
Degree of ‘positivity’ on test
Performance of A Diagnostic
Test
FALSE
NEGATIVES
NON-CASES
CASES
Test cut-off
% of Group
FALSE
POSITIVES
NON-DESEASED
DISEASED
Positive
Negative
Degree of ‘positivity’ on test
Minimising False Negatives: A
Sensitive Test
NON-CASES
CASES
Test cut-off
Cut-off shifted to minimise
false negatives ie to
optimise sensitivity
% of Group
CONSEQUENCES:
- Specificity reduced
NONDESEASED
- A Negative result from a
seNsitive test rules out the
diagnosis - snNout
DISEASED
Positive
Negative
Degree of ‘positivity’ on test
Minimising False Positives: A
Specific Test
Test cut-off
Cut-off shifted to
minimise false positives
ie to optimise specificity
CONSEQUENCES:
% of Group
- Sensitivity reduced
NON-DESEASED
- A Positive result from
a sPecific test rules in
the diagnosis - spPin
DISEASED
Positive
Negative
Degree of ‘positivity’ on test
Receiver Operating Characteristics (ROC)
Non-diseased
Diseased
Threshold
Evaluation Result Value
Or
Subjective Judgment Of Likelihood That Case Is Diseased
Non-diseased
Centers
Diseased
Centers
Threshold
Test result value
or
subjective judgment of likelihood that case is diseased
more typically:
Non-diseased
Centers
Diseased
Centers
Cutoff point
more typically:
Non-diseased
cases
Diseased
cases
FP rate
more typically:
Non-diseased
Centers
TP rate
Diseased
Centers
Threshold
Diseased
Centers
TPF, sensitivity
Non-diseased
Centers
less aggressive
mindset
FPF, 1-specificity
Threshold
Diseased
cases
TPF, sensitivity
Non-diseased
cases
moderate
mindset
FPF, 1-specificity
Threshold
Diseased
cases
TPF, sensitivity
Non-diseased
cases
more
aggressive
mindset
FPF, 1-specificity
Non-diseased
cases
Threshold
Diseased
cases
TPF, sensitivity
Entire ROC curve
FPF, 1-specificity
TPF, sensitivity
Entire ROC curve
FPF, 1-specificity

Check this out:
http://www.anaesthetist.com/mnm/stats/ro
c/Findex.htm
Likelihood Ratios
Pre-test & post-test probability
Pre-test probability of disease can be
compared with the estimated later
probability of disease using the information
provided by a diagnostic test.
 The difference between the previous
probability and the later probability is an
effective way to analyze the efficiency of a
diagnostic method.

It tells you how much a positive or
negative result changes the likelihood that
a patient would have the disease.
 The likelihood ratio incorporates both the
sensitivity and specificity of the test and
provides a direct estimate of how much a
test result will change the odds of having a
disease


The likelihood ratio for a positive result
(LR+) tells you how much the odds of the
disease increase when a test is positive.

The likelihood ratio for a negative result
(LR-) tells you how much the odds of the
disease decrease when a test is negative.
Positive & Negative Likelihood
Ratios

We can judge diagnostic tests: positive
and negative likelihood ratios.

Like sensitivity and specificity, are
independent of disease prevalence.
Likelihood Ratios (Odds)
The probability of a test result in those
with the disease divided by the
probability of the result in those without
the disease.
 How many more times (or less) likely a
test result is to be found in the disease
compared with the non-diseased.

52
Positive Likelihood Ratios
This ratio divides the probability that a
diseased patient will test positive by the
probability that a healthy patient will test
positive.
 The positive likelihood ratio
+LR = sensitivity/(1 – specificity)

False Positive Rate
The false positive rate = false positives /
(false positives + true negatives). It is also
equal to 1- specificity.
 The false negative rate = false negatives /
(false negatives + true positives). It is also
equal to 1 – sensitivity.

Positive Likelihood Ratios
It can also be written as the
true positive rate/false positive rate.
 Thus, the higher the positive likelihood
ratio, the better the test (a perfect test has
a positive likelihood ratio equal to infinity).

Negative Likelihood Ratio
This ratio divides the probability that a
diseased patient will test negative by the
probability that a healthy patient will test
negative.
 The negative likelihood ratio
–LR = (1 – sensitivity)/specificity.

False Negative Rate
The false negative rate = false negatives /
(false negatives + true positives).
 It is also equal to 1 – sensitivity.

Negative Likelihood Ratio
It can also be written as the
false negative rate/true negative rate.
 Therefore, the lower the negative
likelihood ratio, the better the test (a
perfect test has a negative likelihood ratio
of zero).

Positive & Negative Likelihood
Ratios

Although likelihood ratios are independent
of disease prevalence, their direct validity
is only within the original study population.
Probability of Disease
Pre-test probability of disease = disease
prevalence
 Post-test probability of disease =

 If
normal, c/(c+d)
 If negative, a/(a+b)
60
Disease absent,
gold standard
Disease present,
gold standard
Test result positive
True positives (a)
False positives (b)
Test result negative
False negatives (c)
True negatives (d)
Bayes Theorem
Post-test Odds =
Likelihood Ratio X Pre-test Odds
Using Likelihood Ratios to Determine PostTest Disease Probability
P re-test
probability
of disease
P re-test
odds of
disease
Likelihood
ratio
62
P ost-test
odds of
disease
P ost-test
probability
of disease
Pre-test & post-test probability
“Post-test probability” depends on the
accuracy of the diagnostic test and the
pre-test probability of disease
 A test result cannot be interpreted without
some knowledge of the pre-test probability

Where does “pre-test
probability” come from?
Clinical experience
 Epidemiological data
 “Clinical decision rules”
 Guess

what is the likelihood that this
patient has the disease?
A disease with a prevalence of 30% must
be diagnosed.
 There is a test for this disease.
 It has a sensitivity of 50% and a specificity
of 90%.

Likelihood Ratios
FNA Biopsy
Sensitivity
Specificity
88%
82%
From: J Clin End & Metab. 2006;
91(11):4295-4301.
Sensitivity
1 – Specificity
= 0.88 / (1 – 0.82)
= 4.89
This means that Anne’s positive FNA biopsy will
be approx. 5 times as likely to be seen with, as
opposed to without, thyroid cancer.
Prevalence of 30%
Sensitivity of 50%
Specificity of 90%
Disease +ve
30
15
70
63
15
100
Disease -ve
70 – 63 = 7
22 positive
tests in
total of
which 15
have the
disease
About 70%
Likelihood
Population
100
Disease +
4
Test +
3
Test 1
The likelihood that
someone with the
disease will have a
positive test is ¾ or
75%
This is the same as
the sensitivity
Likelihood II
Population
100
Disease 96
The likelihood that
someone without
the disease will
have a positive test
is 7/96 or 7%
This is the same as
the (1-specificity)
Test +
7
Test 89
Likelihood Ratio
Likelihood Ratio =
=
Likelihood of Positive Test Given
The Disease
Likelihood of Positive Test
in the Absence of the Disease
Sensitivity
1- Specificity
=
0.75
= 10.7
0.07
A Likelihood Ratio of 1.0 indicates an uninformative test
(occurs when sensitivity and specificity are both 50%)
The higher the Likelihood Ratio, the better the test
(other factors being equal)
Diagnostic Odds Ratio
Yes
No
Yes
3
7
10
a+b
No
1
4
89
96
90
c+d
TEST
DISEASE
a+c
b+d
Total
a b
c D
d
Total
100
Potentially useful as an
overall summary
measure, but only in
conjunction with other
measures (LR,
sensitivity, specificity)
a+b+c+d
The Diagnostic Odds Ratio is
the ratio of odds of having the
diagnosis given a positive test
to those of having the
diagnosis given a negative test
DOR 
3
1
7
89
0.429

 38.2
0.011
Is there an
easier way?
Likelihood Ratio And Pre- And
Post-test Probabilities
For a given test with a
given likelihood ratio, the
post-test probability will
depend on the pre-test
probability (that is, the
prevalence of the condition
in the sample being
assessed)
Sensitivity Analysis of A
Diagnostic Test
Value
Pre-test
probability
35%
95% CI
26% to
44%
Sensitivity Analysis of A
Diagnostic Test
Value
95% CI
Pre-test
probability
35%
26% to 44%
Likelihood
ratio
5.0
3.0 to 8.5
Applying the 95% confidence
intervals above to the
nomogram, the post-test
probability is likely to lie in the
range 55-85%
Applying A Diagnostic Test In
Different Settings
 The
Positive Predictive Value of a test will vary
(according to the prevalence of the condition in the
chosen setting)
 Sensitivity
and Specificity are usually considered
properties of the test rather than the setting, and
are therefore usually considered to remain
constant
 However,
sensitivity and specificity are likely to be
influenced by complexity of differential diagnoses
and a multitude of other factors (cf spectrum bias)
Likelihood Ratios (Odds)

77
This is an alternative way of describing the
performance of a diagnostic test. Similar to S
and S, and can be used to calculate the
probability of disease after a positive or
negative test (predictive value). Advantage of
this is that it can be used at multiple levels of
test results.
What is this second fraction?
Likelihood Ratio Positive
 Multiplied by any patient’s pretest odds
gives you their posttest odds.
 Comparing LR+ of different tests is
comparing their ability to “rule in” a
diagnosis.
 As specificity increases LR+ increases
and PPV increases (Sp P In)

78
Clinical interpretation of posttest probability
P robability of disease:
Don't
treat for
disease
Do further
diagnostic
testing
Treat for
disease
0
1
Testing
threshold
Disease
ruled out
79
Treatm ent
threshold
If you are here, Test
will help you to go
toward one end of
this probability,
either 0 or 1 to get
the final decision.
Disease
ruled in
Values of Positive and Negative
Likelihood Ratios (LR)
LR
Poor-fair
Good
Excellent
Positive
likelihood
ratio
2.1-5
5.1-10
>10
Negative
likelihood
ratio
0.5-0.2
0.19-0.1
<0.1
Likelihood Ratios & You


Allows us to determine the accuracy with which
a test identifies the target disorder
As the LR becomes larger, the likelihood of the
target disease increases:
Likelihood ratio
Interpretation
>10
Strong evidence to rule in disease
5-10
Moderate evidence to rule in disease
2-5
Weak evidence to rule in disease
0.5-2
No significant change in the likelihood of disease
0.2-0.5
Weak evidence to rule out disease
0.1-0.2
Moderate evidence to rule out disease
<0.1
Strong evidence to rule out disease
Advantages of LRs





82
The higher or lower the LR, the higher or lower
the post-test disease probability
Which test will result in the highest post-test
probability in a given patient?
The test with the largest LR+
Which test will result in the lowest post-test
probability in a given patient?
The test with the smallest LR-
Advantages of LRs

83
Clear separation of test characteristics
from disease probability.
Likelihood Ratios - Advantage
Provide a measure of a test’s ability to rule
in or rule out disease independent of
disease probability
 Test A LR+ > Test B LR+

 Test

Test A LR- < Test B LR Test
84
A PV+ > Test B PV+ always!
A PV- > Test B PV- always!
Predictive Values
Alternate formulations:Bayes’ Theorem
PV+ =
Se  Pre-test Prevalence
Se  Pre-test Prevalence + (1 - Sp)  (1 - Pre-test Prevalence)
High specificity to “rule-in” disease
PV- =
Sp  (1 - Pre-test Prevalence)
Sp  (1 - Pre-test Prevalence) + (1 - Se)  Pre-test Prevalence
High sensitivity to “rule-out” disease
85
Clinical Interpretation: Predictive Values
PV+ And PV-1 Of Electrocardiographic Status2
For Angiographically Verified3 Coronary Artery
Disease, By Age And Sex Of Patient
Sex
Age
PV+ (%)
PV- (%)
F
F
F
<40
40-50
50+
32
46
62
88
80
68
M
M
M
<40
40-50
50+
62
75
85
68
54
38
1. Based on statistical smoothing of results from 78 patients referred to NC
Memorial Hospital for chest pain. Each value has a standard error of 6-7%.
2. At least one millivolt horizontal st segment depression.
3. At least 50% stenosis in one or more main coronary vessels.
86
If Predictive value is more
useful why not reported?
Should they report it?
 Only if everyone is tested.
 And even then.
 You need sensitivity and specificity from
literature. Add YOUR OWN pretest
probability.

87
So how do you figure pretest
probability?






88
Start with disease prevalence.
Refine to local population.
Refine to population you serve.
Refine according to patient’s presentation.
Add in results of history and exam (clinical
suspicion).
Also consider your own threshold for testing.
Pretest Probability: Clinical
Significance
Expected test result means more than
unexpected.
 Same clinical findings have different
meaning in different settings
(e.g.scheduled versus unscheduled visit).
Heart sound, tender area.
 Neurosurgeon.
 Lupus nephritis.

89
What proportion of all patients
will test positive?
Diseased X sensitivity
+ Healthy X (1-specificity)
 Prevalence X sensitivity +
(1-prevalence)(1-specificity)
 We call this “test prevalence”
 i.e. prevalence according to the test.

Some Examples
Diabetes mellitus (type 2)

Check out this:
Some Examples from
Essential Evidence Plus
Disease
Link Address
Diabetes Mellitus
(type 2)
http://www.essentialevidenceplus.com/content/eee/127
Deep Vein
Thrombosis
http://www.essentialevidenceplus.com/content/eee/28
Arrhythmia (Atrial
Fibrillation & Flutter)
http://www.essentialevidenceplus.com/content/eee/13
http://www.essentialevidenceplus.com/
Which one of these test is the best
for SLE Dx?
Test
Sensitivity
Specificity
LR(+)
ANA
99
80
4.95
dsDNA
70
95
14
ssDNA
80
50
1.6
Histone
30-80
50
1.1
Nucleoprotein
58
50
1.16
Sm
25
99
25
RNP
50
87-94
3.8-8.3
PCNA
5
95
1
Was it clear enough !
Key References
Sedlmeier P and Gigerenzer G. Teaching Bayesian
reasoning in less than two hours. Journal of Experimental
Psychology: General. 130 (3):380-400, 2001.
Knotternus JA (ed). The Evidence Base of Clinical
Diagnosis. London: BMJ Books, 2002.
Sackett DL, Haynes RB, Guyatt G, and Tugwell P. Clinical
Epidemiology : A Basic Science for Clinical Medicine.
Boston, Mass: Little, Brown & Co, 1991.
Loong TW. Understanding sensitivity and specificity with
the right side of the brain. BMJ 2003: 327: 716-19.
! ‫ بزنید‬Email ‫اگر میل داشتید‬
[email protected]