Transcript Comprehensive version - Tufts University School of Medicine

Is it True? Evaluating Research
about Diagnostic Tests
The Case of Baby Jeff
The Case of Baby Jeff

CPK testing for Muscular Dystrophy

Prevalence: 1 in 5,000 (0.02%)
• Sensitivity: 100%
• Specificity: 99.98%
Prevalence
20
will have=M.D.
1 in 5,00099,980
= .02%– =
no20M.D.
newborn boys
20 correctly
20 false
Sensitivity
positive 100% positive
0 false
negative
99,960 correctly
Specificity
99.98%
negative
40 positive tests
99,960 negative tests
50% truly positive
100% truly negative
50% falsely positive
0 falsely negative
50% PPV
100% NPV
Negative results
Positive results
100,000 newborn boys
Why this is important
http://today.msnbc.msn.com/id/42829175
Other examples

Lyme disease

Echocardiogram as part of executive physical
• Sensitivity= 95%; specificity= 95%
• High prevalence (20%): PPV =83%
• Low prevalence (2%): PPV = 28%
• Prevalence = 10%; PPV = 50%
Technical vs. Clinical Precision
Technical precision
Clinical precision
Sensitivity
Positive predictive value
The percentage of patients with The percentage of patients with
the disease who have a positive a positive test who have the
test
disease
Specificity
The percentage of patients
without disease who test
negative
Negative predictive value
The percentage of patients with
a negative test who are without
disease.
Predictive Values

Positive Predictive Value
• The percentage of patients with a positive test who
have the disease

Negative Predictive Value
• The percentage of patients with a negative test who
don’t have the disease
Let’s practice
Task 1. A serum test screens pregnant women
for babies with Down’s syndrome. The test is a
very good one, but not perfect. Roughly, 1% of
babies have Down’s syndrome. If the baby has
Down’s syndrome, there is a 90% chance that
the result will be positive. If the baby is
unaffected, there is still a 1% chance that the
result will be positive. A pregnant woman has
been tested and the result is positive.
1,000 similar
Positive: 90%
correctly identified
Negative: 99%
correctly identified
Negative results
Positive results
Prevalence = 1% = ___ patients/1,000?
1,000 similar
Positive: 90%
correctly identified
Negative: 99%
correctly identified
Negative results
Positive results
Prevalence = 1% = ___ patients/1,000?
Down’s
Syndrome
1,000 similar patients
9Positive:
correctly
10 false
90%
positive
correctly
identified positive
1 false
negative
980 correctly
Negative:
99%
negative
correctly
identified
19 positive tests
981 negative tests
47.5% truly positive
99.99% truly negative
52.5 falsely positive
0.001% falsely negative
Negative results
Positive results
10 – Downs = 1% = 10 with
Prevalence
990 No
Downs
Downs
Task 2
A 45-year-old woman presents with a sore throat
and cough but without fever, tonsillar exudate, or
cervical nodes. Using a clinical decision rule, you
determine her likelihood of having strep throat is
1%. However, according to your office protocol,
your medical assistant already has performed a
rapid strep (antigen) test, which is positive. What is
the likelihood the patient has strep throat now?
Antigen test -- Sensitivity: 88% Specific: 96%
Strep throat
100,000 similar patients
880 correctly
3,960 false
Sensitivity
positive 88% positive
120 false
negative
95,040 correctly
Specificity
96%
negative
4840 positive tests
95,160 negative tests
18% truly positive
99.87% truly negative
82% falsely positive
0.126% falsely negative
18% PPV
99.87% NPV
Negative results
Positive results
1,000
Prevalence
– Strep = 1% = 1,000
99,000
with
– viral
strep
Adopting new screening/diagnostic
tests


Sensitivity/specificity not enough
Testing as an intervention
• Did the authors study an outcome patients care
about?
Levels of “POEMness” for Diagnostic
Tests
1.
2.
3.
4.
5.
Sensitivity & specificity
Does it change diagnoses?
Does it change treatment?
Does it change outcomes?
Is it worthwhile (to patients and/or society)?
(examples: HbA1C for DM, CPK vs T4/PKU in
newborns, electron beam tomography for CAD)
Fryback DG, Thornbury JR. The efficacy of diagnostic imaging. Med Decis Making 1991; 11:88-94
Screening pulse oximetry for CHD

Diagnostic performance of abnormal pulse oximetry for congenital heart
defects

for all major congenital defects
* sensitivity 49.06%
* specificity 99.016%
* positive predictive value 13.33%
* negative predictive value 99.86%

for critical congenital defects
* sensitivity 75%
* specificity 99.12%
* positive predictive value 9.23%
* negative predictive value 99.97%
Lancet 2011 Aug 27;378(9793):785
Screening pulse oximetry for CHD
Jaundice, terminating breast-feeding, and the vulnerable child
Breast-feeding was more common in the jaundiced group (61% vs
79%). By 1 month, more mothers of jaundiced infants had
completely stopped breast-feeding (19% vs 42%). They were
more likely to have never left the baby with anyone else
(including the father) or left the baby at most one time for less
than 1 hour (15% vs 31%), more well-visits, more ED visits (2% v
11%, not including bili measurements).
Thus, may increase the risk for premature termination of breastfeeding and for development of the VULNERABLE CHILD
SYNDROME.
Pediatrics 1989 Nov;84(5):773-8
Naming is not curing


In the 1600s, astrology dominated medicine as a healing
profession. Neither worked but astrology was much more
popular because it focused on fixing people's problems.
Medicine, on the other hand, focused mainly on
categorizing illnesses (i.e., diagnosing) and not so much on
treatment.
400 years later there is still a priority on categorizing,
regardless of whether it's helpful. A correct diagnosis is only
useful when it results in the selection of a treatment that
benefits the patient; otherwise, it's only a label.
James Burke. The day the Universe Changed. Boston: Little, Brown and Company, 1985, p. 333.
TEST
+
TEST
-
Disease
No disease
True Positive
(TP)
False Positive
(FP)
ab
cd
False Negative
(FN)
True Negative
(TN)
Sensitivity
TEST
+
TEST
-
Disease
No disease
True Positive
(TP)
False Positive
(FP)
ab
cd
False Negative
(FN)
True Negative
(TN)
Specificity
TEST
+
TEST
-
Disease
No disease
True Positive
(TP)
False Positive
(FP)
ab
cd
False Negative
(FN)
True Negative
(TN)
Positive Predictive Value
TEST
+
TEST
-
Disease
No disease
True Positive
(TP)
False Positive
(FP)
ab
cd
False Negative
(FN)
True Negative
(TN)
Negative Predictive Value
TEST
+
TEST
-
Disease
No disease
True Positive
(TP)
False Positive
(FP)
ab
cd
False Negative
(FN)
True Negative
(TN)
Likelihood Ratios

Similar to the concepts of “ruling in” and “ruling out” disease

Pre Test Odds x LR = Post Test Odds

The problem – we don’t think in terms of odds

Clinical decision rules: Do the hard math for us, be we need to
enter the appropriate data and interpret results
II. Are The Results Valid?

Diagnostic test compared with the “Gold standard”
on all patients

Blinded comparison

Independent testing

Consecutive patient enrollment (adequate
spectrum of disease)
(Must have all for LOE = 1b)
II. Are The Results Valid?

What are the results?
•
•
•
Sensitivity, specificity and predictive values
Likelihood ratio calculation
Prevalence of disease in the study population
• Typical?
• Similar to your practice?
Levels of “POEMness” for Diagnostic
Tests
1.
2.
3.
4.
5.
Sensitivity & specificity
Does it change diagnoses?
Does it change treatment?
Does it change outcomes?
Is it worthwhile (to patients and/or society)?
(examples: HbA1C for DM, CPK vs T4/PKU in
newborns, electron beam tomography for CAD)
Fryback DG, Thornbury JR. The efficacy of diagnostic imaging. Med Decis Making 1991; 11:88-94