Transcript Cattle Diagnostics David R. Smith DVM, PhD –Lincoln University of Nebraska

Cattle Diagnostics
David R. Smith DVM, PhD
Dipl. ACVPM (Epidemiology)
University of Nebraska–Lincoln
Diagnostic
Approaches in
Bovine Medicine
Population
approaches to
BVDV control
David R. Smith, DVM, PhD
University of Nebraska-Lincoln
Applying population dynamics
to the diagnosis and control of
Johne's disease
Strategies for controlling
neonatal diarrhea in
cowcow-calf herds:
David R. Smith, DVM, PhD
The Sandhills
Calving System
Department of Veterinary and Biomedical Sciences
University of Nebraska–Lincoln
David R. Smith, DVM, PhD
University of Nebraska–
Nebraska–Lincoln
Population Diagnostics
• Test performance and interpretation
• Sampling strategies in population
diagnostics
Test performance and
interpretation
Diseased
Not Dis.
Test +
True
Positive
False
Positive
Test -
PrePre-test probability
False
Negative
True
Negative
Specificity
Sensitivity
PV+ = p(TP|T+)
PVPV- = p(TN|Tp(TN|T-)
Making decisions in the face of
uncertainty
Making risk-based decisions to solve problems
3
1
Making risk-based decisions to solve problems
3
1
2
Making risk-based decisions to solve problems
3
1
2
Draw Partner
Making risk-based decisions to solve problems
Witness: Blue Taxi
Accuracy 80%
85% of taxis are
green
Is the witness credible?
Blue
Green
Blue
120
170
290
Green
Witness
Truth
30
680
710
150
850 1,000
FeLV
• 9wk old DSH kitten
• Presented for
“shots”
• PE
•
•
•
•
•
B/A T101.5
Mm pink, refill 1sec
N/R Resp/HR
Ear mites
Fecal flotation
++Ascarids
• FeLV + CITE test
“Whenever there is a
discrepancy between tests,
one must repeat the tests to
be sure that consistent
results are obtained.”
“Finally, when two tests fail
to agree, there is a slim
possibility that one of them
is incorrect. This is not
something to base all your
hopes on, but it is true that
the tests are not 100
percent accurate, and once
in a great while you might
receive an incorrect result.”
FELINE LEUKEMIA VIRUS
CITE FeLV test
CITE FeLV test kit is a monoclonal antibody
based, membrane ELISA designed to detect the
presence of the feline leukemia virus antigen
p27. Serum, plasma or whole blood may be
tested with the kit. A color change greater than
that of the negative control is considered a
positive result.
http://web.vet.cornell.edu/PUBLIC/epidemiology/disease/cite%20felv%20test.htm
FELINE LEUKEMIA VIRUS
CITE FeLV test
…Hawks et.al., tested the CITE/FeLV kit using blood from
specific pathogen free kittens that were either vaccinated
against FeLV or exposed to FeLV at the start of the the
study.
He found the sensitivity to be 0.93 when kittens that were
IFA positive (established FeLV infection) were tested. When
kittens who were IFA negative, but viral isolation positive
(early FeLV infection), were tested he found the sensitivity
to be 0.60. He found the specificity of the CITE test to be
0.86.
http://web.vet.cornell.edu/PUBLIC/epidemiology/disease/cite%20felv%20test.htm
FeLV testing (healthy kitten)
Test -
Test +
Diseased
60
40
Not Dis.
1386 1446
PV+
=4%
8514 8554
PV=99.5%
AP=14.5%
100
9900 10,000
pD = 1%
Diagnostic tests
• Objective of diagnostic tests
• Reflect disease status of an individual (or herd)
• Reflect infection status of an individual (or herd)
• We often make inferences on the disease status of an
individual by using tests that measure infection status
(not always a good assumption).
• Diagnostic tests should lead to the solution to a
problem
Diagnostic tests
• What is a diagnostic
test?
• Medical history
• Physical examination
• Laboratory tests
• Objective
measurements that
we use as predictors
of disease [infection]
status
Diagnostic tests
• The problem is that no test is infallible.
• Test results can be wrong!
Diagnostic tests
• If diagnostic tests are not perfect then we
would like to know how often, and under what
circumstances, a diagnostic test might give us a
wrong answer.
• Test Performance
• Sensitivity / Specificity
• Likelihood ratios
• Kappa statistic (Test agreement beyond chance)
• Test Interpretation
Test performance
Sensitivity =p(T+|D+)
Specificity =p(T-|D-)
Test +
True
Positive
Test +
False
Positive
False
Negative
Test -
Not Dis.
Test -
Diseased
True
Negative
Test performance
ELISA values >0.50
F re q u e n c y
10
9
8
7
6
5
4
3
2
1
0
-0.06
0.04
0.14
0.24
0.34
0.44
ELISA Value (PACELISA)
BCV positive samples
BCV negative samples
Cut-off points
• Many tests are measures of continuous or
ordinal outcomes.
• The tests results are reported as positive or
negative based on a cut-off point .
• “any test value above X considered positive”
• The sensitivity and specificity of the test
depends on where that cut-off point is.
Cut-off points
ELISA values >0.50
F re q u e n c y
10
9
8
7
6
5
4
3
2
1
0
-0.06
0.04
0.14
0.24
0.34
0.44
ELISA Value (PACELISA)
BCV positive samples
BCV negative samples
Cut-off points
ELISA values >0.50
F re q u e n c y
10
9
8
7
6
5
4
3
2
1
0
-0.06
0.04
0.14
0.24
0.34
0.44
ELISA Value (PACELISA)
BCV positive samples
BCV negative samples
Cut-off points
ELISA values >0.50
F re q u e n c y
10
9
8
7
6
5
4
3
2
1
0
-0.06
0.04
0.14
0.24
0.34
0.44
ELISA Value (PACELISA)
BCV positive samples
BCV negative samples
Cut-off points
ELISA values >0.50
F re q u e n c y
10
9
8
7
6
5
4
3
2
1
0
-0.06
0.04
0.14
0.24
0.34
0.44
ELISA Value (PACELISA)
BCV positive samples
BCV negative samples
Cut-off points
P e rc e n ta g e
100%
80%
60%
40%
20%
0%
-0.06
0.04
0.14
0.24
0.34
ELISA Values (PACELISA)
Sensitivity
Specificity
0.44
Cut-off points
P e rc e n ta g e
100%
80%
60%
40%
20%
0%
-0.06
0.04
0.14
0.24
0.34
ELISA Values (PACELISA)
Sensitivity
Specificity
0.44
Cut-off points
P e rc e n ta g e
100%
80%
60%
40%
20%
0%
-0.06
0.04
0.14
0.24
0.34
ELISA Values (PACELISA)
Sensitivity
Specificity
0.44
Diagnostic Test Evaluation
Part II
Other ways to evaluate tests:
Likelihood ratios and Kappa
Not Dis.
Test +
True
Positive
False
Positive
Test -
Likelihood ratio
Diseased
False
Negative
True
Negative
LR = Sensitivity / (1- Specificity)
True positive rate
False positive rate
Like the statistics that it is
derived from, the likelihood ratio
is independent of pre-test
probability
Likelihood ratios
• Provides more information on test
performance than sensitivity or
specificity when tests results are
reported quantitatively
• Don’t lose information by dichotomizing
(+/-)
• The concept of the STRONG or WEAK
positive
Likelihood ratio
ELISA value
90 or more
80 - 89
70 - 79
60 - 69
50 - 59
40 - 49
30 - 39
20 - 29
10 - 19
<10
Total
Positive
26
14
9
12
15
20
14
11
16
3
140
Negative Likelihood ratio Sensitivity
1
49.0
0.186
1
37.7
0.286
3
18.5
0.350
5
11.5
0.436
7
8.4
0.543
11
6.5
0.686
33
3.4
0.786
73
1.7
0.864
91
1.1
0.979
39
1.0
1.000
264
Specificity
0.996
0.992
0.981
0.962
0.936
0.894
0.769
0.492
0.148
0.000
ELISA value reported as percent of positive control. Positive = fecal
culture positive, Negative = fecal culture negative
Adapted From: Prev Vet Med 13:197-204
Disease status and test
performance
• Up to now we have assumed that the true
disease [infection] status of the individual was
known –for determining test performance
• In real life, we rarely are perfectly sure of the
true disease [infection] status
• It is important to know how true disease
[infection] status was represented before
evaluating the test.
Disease status
• We often evaluate new tests by comparing
them to other “reliable” tests that we believe
represent disease [infection] status.
• We call these “reliable” tests “gold standards”
• Few are.
Inter-rater agreement
• The agreement between two tests beyond the
role of chance may be evaluated by the
kappa statistic.
• Values range from -1 to +1
• Similar interpretation to correlation coefficient
• No assumption of a gold-standard is made.
• Assumes that agreement is a good thing.
• But a superior test may rightly disagree with an
inferior test
Test Interpretation
• What is the probability that the
test result reflects the TRUE
disease status of the animal?
• Post-test probability
• Predictive value of a positive, or
negative, test
Post-test probability
So you have the test result in your
hand… What else do you need?
• Clinical judgment
• What was the
probability of disease
BEFORE you conducted
the test?
• Pre-test probability
Clinical judgement
What is the probability that this animal
(herd) has disease X?
History
Farm
Individual
Prior test results
Clinical signs
Physical exam
Literature
prevalence estimates
Post-test probability
So you have clinical judgment and the test result
in your hand… What else do you need?
• Knowledge of test
performance
• Sensitivity
• Specificity
• Likelihood ratios
Test interpretation
Diseased
Not Dis.
Test +
True
Positive
False
Positive
Test -
Clinical judgment
False
Negative
True
Negative
Specificity
Sensitivity
PV+ = p(TP|T+)
PV- = p(TN|T-)
PV+ =
Test +
Post-test probability
Predictive value
Not Dis.
True
Positive
False
Positive
Test -
Test interpretation
Diseased
False
Negative
True
Negative
pD x Sensitivity
pD x Sensitivity + (1-Spec) x (1-pD)
PV- =
(1-pD) x Specificity
(1-pD) x Specificity + (1-Sens) x pD
100%
Predictive Value
Positive Test
96% False +
Sensitivity 60%,
Specificity 86%
80%
60%
40%
20%
0%
0
0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.1
Pre-test probability
Sensitivity 60%, Specificity 86%
Predictive Value
100%
80%
60%
40%
20%
0%
0
0.2
0.4
0.6
0.8
Pre-test probability
PV+
PV-
1
Calculating test
performance and
interpretation
Test Interpretation Using
Likelihood ratios
• Provides more information on test
performance than sensitivity or
specificity when tests results are
reported quantitatively
• Don’t lose information by dichotomizing
(+/-)
• The concept of the STRONG or WEAK
positive
Likelihood ratio
ELISA value
90 or more
80 - 89
70 - 79
60 - 69
50 - 59
40 - 49
30 - 39
20 - 29
10 - 19
<10
Total
Positive
26
14
9
12
15
20
14
11
16
3
140
Negative Likelihood ratio
1
49.0
1
37.7
3
18.5
5
11.5
7
8.4
11
6.5
33
3.4
73
1.7
91
1.1
39
1.0
264
Sensitivity
0.186
0.286
0.350
0.436
0.543
0.686
0.786
0.864
0.979
1.000
Specificity
0.996
0.992
0.981
0.962
0.936
0.894
0.769
0.492
0.148
0.000
ELISA value reported as percent of positive control. Positive = fecal
culture positive, Negative = fecal culture negative
Prev Vet Med 13:197-204
Post-test probability =
P(D+|Tx)
• Still need to interpret the results based on what
you know about the animal
• clinical judgment, literature, etc
• Pre-test odds x LR = post-test odds
Odds versus probability
• 30 red M&M’s
• 20 green M&M’s
• Odds of selecting a red M&M are 30 to 20
(=3/2 or 1.5)
• Probability of selecting a red M&M is 30 out of
50 (=3/5 or 0.6)
Odds versus probability
• Converting Odds to Probability
• Odds/(Odds +1) = probability
• 1.5/(1.5+1) = 1.5/2.5 = 0.6
• Converting Probability to Odds
• Probability/(1-probability) = odds
• 0.6/(1-0.6) = 0.6/0.4 = 1.5
Post-test probability =
P(D+|Tx)
• General population
(10% of cows infected in 25% of herds)
• Pre-test probability =0.025 or Pre-test odds = 0.026
ELISA value
90 or more
80 - 89
70 - 79
60 - 69
50 - 59
40 - 49
30 - 39
20 - 29
10 - 19
<10
LR
49.0
37.7
18.5
11.5
8.4
6.5
3.4
1.7
1.1
1.0
Post test odds
1.26
0.97
0.47
0.29
0.22
0.17
0.09
0.04
0.03
0.03
Post-test probability
0.56
0.49
0.32
0.23
0.18
0.14
0.08
0.04
0.03
0.03
Post-test probability =
P(D+|Tx)
• From a very high prevalence herd
• Pre-test probability = 0.25 Pre-test odds = .333
ELISA value
90 or more
80 - 89
70 - 79
60 - 69
50 - 59
40 - 49
30 - 39
20 - 29
10 - 19
<10
LR
49.0
37.7
18.5
11.5
8.4
6.5
3.4
1.7
1.1
1.0
Post test odds
16.33
12.57
6.17
3.83
2.80
2.17
1.13
0.57
0.37
0.33
Post-test probability
0.94
0.93
0.86
0.79
0.74
0.68
0.53
0.36
0.27
0.25
Herd-level diagnostics
Do Disease X- infected animals exist in this
herd?
Determining herd-status
based on tests of individuals
• Herd-level sensitivity
• The probability that a herd with diseased [infected]
animals will be correctly classified
• Herd-level specificity
• The probability that a herd truly free of the disease
[infection] will be correctly classified
1
1
0.8
0.8
Probability
Probability
Herd-level test
performance
0.6
0.4
0.6
0.4
0.2
0.2
0
0
10
20
30
40
50
60
70
80
90
As we sample more
animals in the herd
HSENS increases –
10
100 110
20
30
40
50
60
70
90
100 110
Sample size
Sample size
HPV+
HSPEC hypgeom
Sensitivity of the test (individuals)
Specificity of the test (individuals)
True prevalence in a diseased herd
0.95
0.97
0.1
Herd size
Proportion of herds affected
400
0.1
Probability
HSENS hypgeom
80
HPV-
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
WHY?
10
20
30
40
50
60
70
80
90
Sample size
HEFFIC
HAP
HPD
100 110
1
1
0.8
0.8
Probability
Probability
Herd-level test
performance
0.6
0.4
0.6
0.4
0.2
0.2
0
0
10
20
30
40
50
60
70
80
90
As we sample more
animals in the herd
HSPEC decreases –
10
100 110
20
30
40
50
60
70
90
100 110
Sample size
Sample size
HPV+
HSPEC hypgeom
Sensitivity of the test (individuals)
Specificity of the test (individuals)
True prevalence in a diseased herd
0.95
0.97
0.1
Herd size
Proportion of herds affected
400
0.1
Probability
HSENS hypgeom
80
HPV-
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
WHY?
10
20
30
40
50
60
70
80
90
Sample size
HEFFIC
HAP
HPD
100 110
Determining herd-status
based on tests of individuals
Herd-level test interpretation
• Herd-level positive predictive value
• The probability that a herd classified as diseased
[infected] truly has the disease
• Owner’s concern
• Herd-level negative predictive value
• The probability that a herd classified as not diseased
[infected] is truly free of the disease
• Buyers concern… or the government
Herd screening -Post-test
probability
 Sensitivity of the test for individuals
 Specificity of the test for individuals
 Prevalence of reactors in a positive herd
 Prevalence of infected herds
 Sample size
 Number of reactors to classify the herd as
infected
1
1
0.8
0.8
Probability
Probability
Herd-level test
interpretation
0.6
0.4
0.6
0.4
0.2
0.2
0
0
10
20
30
40
50
60
70
80
90
10
100 110
20
30
40
50
60
70
90
100 110
Sample size
Sample size
HPV+
HSPEC hypgeom
Sensitivity of the test (individuals)
Specificity of the test (individuals)
True prevalence in a diseased herd
0.95
0.97
0.1
Herd size
Proportion of herds affected
400
0.1
Probability
HSENS hypgeom
80
HPV-
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
10
20
30
40
50
60
70
80
90
Sample size
HEFFIC
HAP
HPD
100 110
Determining herd-status
based on tests of individuals
• When multiple animals are being tested to
determine a herd’s infection status it is
important that test specificity be as high as
possible
• The probability for misclassification of the herd
due to false positives (1-specificity) is
compounded by the number of animals tested
Improving herd-level specificity
• Serial testing
• Screen herd by sensitive (inexpensive) screening test
• Confirm positive test results with a highly specific test
• Another method to improve specificity?
• Change the cut-off point!
Loss of sensitivity to
increase specificity
Test more animals!
Conclusion
• Optimizing the post-test probability of correctly
classifying a herd’s infection status with tests of
individuals may involve adjustments to
•
•
•
•
improve test-specificity
the number tested
the number of reactors to classify a herd as infected
target high-risk populations
USDA BSE Enhanced Surveillance
On June 24, 2005
USDA reported
that one of
ELISA-positive
sample, from a 12
yr old Texas cow,
was positive for
BSE by Western
blot
Final thoughts…
• It is important to consider post-test probability
before beginning a herd-testing program
• Both positive and negative predictive value are
important to someone
• The costs of incorrect herd classification should
be considered
In the population you find “units” that
are:




Test positive
Test negative
Diseased
Not Diseased
Does test status represent
disease status?
Test result
Disease/Infection status
Likelihood ratios
• A single value that summarizes the information
about sensitivity and specificity at a given
range of test values
• A measure of the likelihood that a sample
within a given range of test values came from
an individual with the target condition.
Should we use SP ratios?
• Yes, more information for decision making
but...
• Not validated LR ratios for many assays
• Complicated issue -Would need to educate
practitioners
• How to use likelihood ratios
• Develop charts to determine post-test probability
Determining herd-status
based on tests of individuals
• Same principles of sensitivity, specificity
and predictive-value apply
• Consequences of misclassifying herd
status
• False sense of security, and further spread of the
disease, if herd is falsely classified as negative
(Buyer’s concern…or the government)
• Undue financial burden if herd is falsely classified
as an infected herd (Owners’s concern)
Frequency Distribution
M. paratuberculosis ELISA serology
Herd
UNL 1
1 animal
positive
(0.9%)
1
Frequency Percent
n = 108
0.75
0.5
0.25
0
0
0.125 0.25 0.375 0.5 0.625 0.75 0.875
S/P Ratio (less than or equal to)
1
1.125
1
1.125
Herd
OSU 2
6 animals
positive
(3.6%)
Frequency Percent
n = 169
1
0.75
0.5
0.25
0
0
0.125 0.25 0.375 0.5 0.625 0.75 0.875
S/P Ratio (less than or equal to)