Transcript Reliability of Screening Tests RELIABILITY: The extent to which the screening test will produce the same or very similar results each time it is.

Reliability of Screening Tests
RELIABILITY:
The extent to which the screening test will
produce the same or very similar results each
time it is administered.
--- A test must be reliable before it can be valid.
--- However, an invalid test can demonstrate
high reliability.
Reliability of Screening Tests
Sources of variability that can affect the
reproducibility of results of a screening test:
1. Biological variation (e.g. blood pressure)
2. Reliability of the instrument itself
3. Intra-observer variability (differences in
repeated measurement by the same
screener)
4. Inter-observer variability (inconsistency in
the way different screeners apply or
interpret test results)
Use of Multiple Screening Tests
Sequential (two-stage) testing:
A less expensive, less invasive, or less
uncomfortable test is performed first…
those who screen positive are referred for
further testing using a test which may have
greater sensitivity and specificity…
reduces false positives, hence an increase in net
specificity.
Use of Multiple Screening Tests
Simultaneous testing:
Multiple tests are used simultaneously…
Person tests “positive” if there is a positive
result on any of the tests employed…
reduces false negatives, hence an increase in
sensitivity … but at the expense of decreased
specificity.
Measuring the performance
(yield) of a screening test
Performance Yield

People with positive screening test
results will also test positive on the
diagnostic test:
Predictive Value Positive (PV+)

People with negative screening test
results are actually free of disease
Predictive Value Negative (PV-)
Performance Yield
True Disease Status
+
Results of +
Screening
Test
-
a
b
c
d
Predictive value positive (PV+): The probability that a
person actually has the disease given that he or she
tests positive.
PV+
=
a / (a + b)
Performance Yield
True Disease Status
+
Results of +
Screening
Test
-
a
b
c
d
Predictive value negative (PV-): The probability that a
person is truly disease free given that he or she
tests negative.
PV-
=
d / (c + d)
Performance Yield
True Disease Status
+
Results of +
Screening
Test
-
400
995
100
98905
Sensitivity: a / (a + c) = 400 / (400 + 100) =
80%
Specificity: d / (b + d) = 98905 / (995 + 98905) =
99%
PV+:
a / (a + b) = 400 / (400 + 995) =
29%
PV-:
d / (c + d) = 98905 / (100 + 98905) =
99%
Performance Yield
True Disease Status
+
Results of +
Screening
Test
PV+:
400
995
100
98905
a / (a + b) = 400 / (400 + 995) = 29%
Among persons who screen positive, 29% are found
to have the disease.
Performance Yield
True Disease Status
+
Results of +
Screening
Test
PV-:
400
995
100
98905
d / (c + d) = 98905 / (100 + 98905) = 99.9%
Among persons who screen negative, 99.9% are found
to be disease free.
Performance Yield
Factors that influence PV+ and PV1.
The more specific the test, the higher
the PV+
2.
The higher the prevalence of preclinical
disease in the screened population, the
higher the PV+
3.
The more sensitive the test, the higher
the PV-
Performance Yield
Prevalence (%)
Sensitivity
Specificity
PV+
0.1
90%
95%
1.8%
1.0
90%
95%
15.4%
5.0
90%
95%
48.6%
50.0
90%
95%
94.7%
Performance Yield
Thus, the PV+ is maximized when used in “high
risk” populations since the prevalence of preclinical disease is higher than in the general
population….
screening a total population for a relatively
infrequent disease can be very wasteful of
resources and may yield few previously
undetected cases.
Effectiveness of Screening
Evaluating if the screening program reduced
morbidity and mortality from the disease:
1.
Overall shift in severity of disease at the
time of diagnosis.
2.
Compare cause-specific mortality from
among those whose disease was picked up
by screening versus those with a diagnosis
related to symptoms.
Effectiveness of Screening
Evaluating if the screening program reduced
morbidity and mortality from the disease:
3.
Reduction in disease-related
complications.
4.
Improvement of quality of life in screened
individuals.
Effectiveness of Screening
In reality, establishing the sensitivity and
specificity of screening tests may be difficult…
often times, data are only available on persons
who screen positive and are referred for
further testing.
a
b
c
d
Data are available for cells “a” and
“b” only.
Permits calculation of PV+ only
Effectiveness of Screening
Sources of bias in evaluating screening
programs:
1. Self-selection bias (volunteer bias)
2. Lead time bias
3. Length bias
4. Over-diagnosis bias
Effectiveness of Screening
1.
Self-selection bias (volunteer bias):
---
Volunteers for screening programs may be
healthier, on average, than persons who do
not participate in screening programs.
On the other hand….
--The “worried well” may be more likely to
participate and may be at overall higher risk
due to family history or lifestyle
characteristics.
Effectiveness of Screening
2.
Lead time bias:
Lead time: The interval between “diagnosis” of
disease at screening and when it would have
been detected from clinical symptoms.
---
Survival may appear to be increased
among screen-detected cases simply
because diagnosis was made earlier in the
course of the disease.
Effectiveness of Screening
3.
Length bias (prognostic selection):
----
The overrepresentation among screendetected cases of those with a long preclinical phase, and thus a more favorable
prognosis.
---
Those with a long pre-clinical phase are
more readily detectable by screening than
more rapidly progressing cases with a
short pre-clinical phase.
Effectiveness of Screening
4.
Over-diagnosis bias:
----
Persons who screen positive and are truly
disease free (false positives), yet are
erroneously diagnosed as having the
disease.
---
Since these persons are truly disease
free, we expect a more favorable longterm outcome – giving the appearance of
an effective screening program.
Effectiveness of Screening
Possible interpretations for null results in a
screening program evaluation include:
----
The disease has an extremely short
detectable pre-clinical phase.
---
Current therapeutic intervention is no
more effective when provided earlier than
at usual time of diagnosis.
---
Inadequacies of care provided to those
who screen positive.
Effectiveness of Screening
Study Designs Used to Evaluate Screening
Programs:
1.
Ecological Studies
2.
Observational Analytic Studies
3.
Randomized Trials (infrequent and
difficult to carry out)