Transcript STATISTICAL EVALUATON OF DIAGNOSTIC TESTS
STATISTICAL EVALUATION OF DIAGNOSTIC TESTS
Describing the performance of a new diagnostic test
Physicians are often faced with the task of evaluation the merit of a new diagnostic test. An adequate critical appraisal of a new test requires a working knowledge of the properties of diagnostic tests mathematical relationships between them.
and the
The gold standard test: Assessing a new diagnostic test begins with the identification of a group of patients known to have the disorder of interest, using an accepted reference test known as the gold standard.
Limitations: 1) The gold standard is often the most risky, technically difficult, expensive, or impractical of available diagnostic options.
2) For some conditions, no gold standard is available.
The basic idea of diagnostic test interpretation is to calculate the probability a patient has a disease under consideration given a certain test result. A 2 by 2 table can be used for this purpose. Be sure to label the table with the test results on the left side and the disease status on top as shown here: Test Positive Negative Disease Present True Positive Absent False Positive False Negative True Negative
The sensitivity
of a diagnostic test is the probability that a diseased individual will have a positive test result. Sensitivity is the true positive rate (TPR) of the test.
Sensitivity = P(T + |D + )=TPR diseased with positive test all diseased = TP / (TP+FN)
The specificity
of a diagnostic test is the probability that a disease-free individual will have a negative test result. Specificity is the true negative rate (TNR) of the test.
Specificity=P(T |D ) = TNR disease free with negative test all disease free =TN / (TN + FP).
False-positive rate: The likelihood that a nondiseased patient has an abnormal test result.
FPR = P(T + |D-)= disease free with positive test all diseased free = FP / (FP+TN)
False-negative rate: The likelihood that a diseased patient has a normal test result.
FNR = P(T |D + )=
diseased with negative test all diseased
= FN / (FN+TP)
Pretest Probability
is the estimated likelihood of disease before the test is done.
It is the same thing as
prior probability
and is often estimated. If a defined population of patients is being evaluated, the pretest probability is equal to the
prevalence
of disease in the population. It is the proportion of total patients who have the disease.
P(D + ) = (TP+FN) / (TP+FP+TN+FN)
Sensitivity and specificity describe how well the test discriminates between patients with and without disease. They address a different question than we want answered when evaluating a patient, however.
What we usually want to know is: given a certain test result, what is the probability of disease? This is the
predictive value
of the test.
Predictive value of a positive test
is the proportion of patients with
positive
tests who have disease.
PVP=P(D + |T + ) = TP / (TP+FP) This is the same thing as
posttest probability
of disease given a positive test. It measures how well the test rules in disease.
Predictive value of a negative test
is the proportion of patients with
negative
tests who
do not
have disease. In probability notation: PVN = P(D |T ) = TN / (TN+FN) It measures how well the test rules out disease. This is posttest probability of non-disease given a negative test.
Evaluating a 2 by 2 table is simple if you are methodical in your approach.
Test Positive Present TP Disease Absent FP Negative FN Total with disease TN Total with out disease Total positive Total negative Grand total
Bayes’ Rule Method
Bayes’ rule is a mathematical formula that may be used as an alternative to the back calculation method for obtaining unknown conditional probabilities such as PVP or PVN from known conditional probabilities such as sensitivity and specificity.
General form of Bayes’ rule is
P
(
A B
)
P
(
A
)
P
(
P
(
B A
)
A
)
P
(
B P
(
A
)
A
)
P
(
B A
) Using Bayes’ rule, PVP and PVN are defined as
PVP
P
(
D
T
)
p
(
D
)
p TPR
(
D
) ( 1
TPR p
(
D
))
FPR
PVN
P
(
D
T
)
p
(
D
)
p
(
TNR D
) ( 1
TNR p
(
D
))
FNR
Example The following table summarizes results of a study to evaluate the dexamethasone suppression test (DST) as a diagnostic test for major depression.
The study compared results on the DST to those obtained using the gold standard procedure (routine psychiatric assessment and structured interview) in 368 psychiatric patients.
1.
2.
What is the prevalence of major depression in the study group?
For the DST, determine a-Sensitivity and specificity b-False positive rate (FPR) and false negative rate (FNR) c-Predictive value negative (PVN) positive (PVP) and predictive value
DST Result + Total + 84 Depression 5 131 215 148 153 Total 89 279 368 P(D + ) =215/368 =0.584
Sensitivity = P(T + |D + )=TPR=TP/(TP+FN)=84/215=0.391
Specificity=P(T |D )=TNR=TN / (TN + FP)=148/153=0.967
FPR = P(T + |D )=FP/(FP+TN)=5/153=0.033
FNR = P(T |D + )=FN/(FN+TP)=131/215=0.609
PVP=P(D + |T + ) = TP / (TP+FP)=84/89=0.944
PVN = P(D |T ) = TN / (TN+FN)=148/279=0.53
FNR=1-Sensitivity=1-0.391=0.609
FPR=1-Specificity=1-0.967=0.033
ROC (Receiver Operating Characteristic ) CURVE
The ROC Curve is a graphic representation of the relationship between sensitivity and specificity for a diagnostic test. It provides a simple tool for applying the predictive value method to the choice of a positivity criterion.
ROC Curve is constructed by plottting the true positive rate (sensitivity) against the false positive rate (1-specificty) for several choices of the positivity criterion.
1,00
ROC Curve
,75 ,50 ,25 0,00 0,00 ,25 ,50 ,75 1,00 1 - Specificity Diagonal segments are produced by ties.
Disease-free Positivity criterion Diseased TP FP 2 Test negative
x i
1 Test positive FN TN
z
1
z
2
x i x i
1 1 2 2
Example: One of the parameters which are evaluated for the diagnosis of CHD, is the value of “HDL/Total Cholesterol”.
Consider a population consisting of 67 patients with CHD, 93 patients without CHD. The result of HDL/Total Cholesterol values of these two groups of patients are as follows.
CHD+ Hdl/Total Cholestrol CHD Hdl/Total Cholestrol 0,29 0,26 0,39 0,16 .
.
.
0,25 0,36 0,30 0,20 .
.
.
HDL/Total Cholestrol Descriptive Statistics GROUP CHD CHD+ Mean ,2926 ,2301 SD ,066 ,048 Min ,16 ,06 Max ,52 ,34
To construct the ROC Curve, we should find sensitivity and specificity for each cut off point. We have two alternatives to find these characteristics.
• Cross tables • Normal Curve
If HDL/Total Cholestrol is less than or equal to 0,26, we classify this group into diseased.
RATIO 0,26> Total Count 0,26<= Count Count CHD 64 68,8% 29 31,2% 93 100,0% + 15 22,4% 52 77,6% 67 100,0% Total 79 49,4% 81 50,6% 160 100,0% Specificity Sensitivity
Best cutoff point Cutoff 0,000 0,093 0,129 0,142 0,156 0,158 0,162 0,168 0,171 0,173 0,175 .
.
.
0.26
.
.
0,393 0,402 0,407 0,420 0,446 0,493 1,000
TPR
0,000 0,015 0,030 0,045 0,060 0,075 0,075 0,104 0,119 0,119 0,119 .
.
.
0.78
.
.
1,000 1,000 1,000 1,000 1,000 1,000 1,000
FPR
0,000 0,000 0,000 0,000 0,000 0,000 0,011 0,011 0,011 0,022 0,032 .
.
.
0.31
.
.
0,935 0,946 0,957 0,968 0,978 0,989 1,000
Let cutoff=0,171 RATIO 0,171< 0,171>= Total CHD 92 98,9% 1 1,1% 93 100% + 59 88% 8 12% 67 100% Total 151 94% 9 5,6% 160 100%
ROC Curve 1,0 ,9 ,8 ,7 ,6 ,5 ,4 ,3 ,2 ,1 0,0 0,0 1 Seçicilik ,1 ,2 ,3 ,4 ,5 ,6 ,7 ,8 ,9 1,0 1-Specificity Cutoff=0.26
TPR=0.78
FPR=0.31
TNR=0.69
FNR=0.22
Area Under the Curve
Tes t Result Variable(s): ORAN Area ,778 Std. Error a ,036 Asymptotic Sig.
b ,000 Asymptotic 95% Confidence Interval Lower Bound ,708 Upper Bound ,849 The test result variable(s): ORAN has at least one tie between the pos itive actual state group and the negative actual state group. Statis tics may be bias ed.
a. Under the nonparametric as sumption b. Null hypothesis : true area = 0.5
If Patients with CHD and without CHD are normally distributed, we can easily find sensitivity and specificity from the area under these normal curves. Sensitivity and specificity are calculated for each different cotoff points
CHD+ CHD-
TP FP FN TN
0,23 0,29 Cutoff=0,28
If we take cut off point=0.28, the characteristics of test are:
CHD-
Z CHD+ =(0.28-0.23)/0.048=1.04
CHD+
TPR=0.5+0.3508=0.8508
FNR=1-TPR=0.1492
Z CHD =(0.28-0,29)/0.066=-0.15 TNR=0.5+0.0596=0.5596
FPR=1-TNR=0.4404
0,23 0,29 Cutoff=0,28
Cutoff 0,10 0,15 0,20 0,25 0,28 0,30 0,35 0,45
TPR
0,00 0,05 0,27 0,66 0,85 0,93 0,99 1,00 FNR 1,00 0,95 0,73 0,34 0,15 0,07 0,01 0,00 TNR 1,00 0,98 0,91 0,73 0,56 0,44 0,18 0,00
FPR
0,00 0,02 0,09 0,27 0,44 0,56 0,82 1,00 1,00 0,90 0,80 0,70 0,60 0,50 0,40 0,30 0,20 0,10 0,00 0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 FPR 1