Transcript HOW TO READ A PAPER

How to Use an Article
About a Diagnostic Test
Akbar Soltani. MD, Endocrinologist
Tehran University of Medical Sciences (TUMS)
Endocrine and Metabolism Research Center (EMRC)
Evidence-Based Medicine Research Center (EBMRC)
Shariati Hospital
www.soltaniebm.com
www.ebm.ir
www.avincennact.ir
EBMRC
OBJECTIVES
•Objectives of testing
•Diagnostic research
•Critical appraisal
•Summary
EBMRC
• 2/3 legal claims against GPs
in UK
• 40,000-80,000 US hospital
deaths from misdiagnosis
per year
• Adverse events, negligence
cases, serious disability
more likely to be related to
misdiagnosis than drug
errors
• Diagnosis uses <5% of
hospital costs, but influences
60% of decision making
EBMRC
Objectives of testing
• Increasing certainty of the
presence or absence of disease
• This requires sufficient
discriminative power.
• 2×2 table relating test outcome
to a reference standard.
EBMRC
Objectives of testing
• Supporting clinical
management
• Determining localization, and
shape of arterial lesions is
necessary for treatment
decisions
EBMRC
Objectives of testing
• Assessing prognosis As the starting
point for clinical follow up and
informing patients.
• Monitoring clinical course When a
disease is untreated, or during or after
treatment.
• Measuring fitness For example, for
sporting activity or for employment.
EBMRC
OBJECTIVES
•Objectives of testing
•Diagnostic research
•Critical appraisal
•Summary
EBMRC
Evidence Based Approach
How to Use an Article About
a Diagnostic Test
EBMRC
Appraising diagnostic tests: 3 easy steps
1. Are the results valid?
2. What are the results?
3. How Can the Results
be Applied to Patient
Care?
EBMRC
Some definitions
Present
Test
Result
Positive
Negative
True positive
A
False negative
C
Sensitivity = A / (A+C)
Specificity = D / (B+D)
EBMRC
Disease
Absent
False positive
B
True negative
D
Tip…..
• Sensitivity is useful to me
– ‘The new rapid chlamydia test was positive in 47 out of 56 women with
chlamydia (sensitivity =83.9%)’
• Specificity seems a bit confusing!
– ‘The new rapid chlamydia test was negative in 600 of the 607 women
who did not have chlamydia (specificity = 98.8%)’
• So…the false positive rate is sometimes easier
– False positive rate = 1 – specificity
– So a specificity of 98.8% means that the new rapid test is wrong (or
falsely positive) in 1.2% of women
EBMRC
Some definitions
Present
Test
Result
Positive
Negative
True positive
A
False negative
C
Disease
Absent
False positive
B
True negative
D
PPV = A / (A+B)
Sen*prevalence
Sen*prev+(1-Spec) *(1-prev)
NPV = D / (C+D)
Spec*(1-prev)
(1-Sen)*prev+
Spec*(1-prev)
EBMRC
Basic structure of diagnostic studies
Series of patients
Index test
Reference (“gold”) standard
Compare the results of the
index test with the reference
standard, blinded
EBMRC
Appraising diagnostic tests: 3 easy steps
1. Are the results valid?
2. What are the results?
3. How Can the Results
be Applied to Patient
Care?
EBMRC
• What were the key selection (inclusion
& exclusion) criteria?
• Were they replicable?
• List important selection criteria; e.g.
age group, gender, risk profile, medical
history.
• There should be sufficient information
in the paper to allow the reader to
theoretically select a similar population
EBMRC
• Did selection lead to an appropriate
spectrum of participants (like those
assessed in practice)
• Participants with the range of common
presentations of the target disorder and
with commonly confused diagnosis
EBMRC
1. Was an appropriate spectrum of
patients included? Spectrum bias
• You want to find out how good chest X rays
are for diagnosing pneumonia in the
Emergency Department
• Best = all patients presenting with
difficulty breathing get a chest X-ray
• Spectrum bias = only those patients in
whom you really suspect pneumonia get a
chest X ray
EBMRC
Spectrum bias
• Studies from referral centers
• Patients with negative results are
less likely to referred
EBMRC
2) Did the results of the test being evaluated influence the
decision to perform the reference standard? 2. Were all patients
subjected to the gold standard? Verification (work-up) bias
• What was the gold standard of
diagnosis?
• The validity of the study requires
that there is an accepted, valid and
replicable reference standard of
diagnosis.
EBMRC
gold standard
• 1.Laboratory tests
(Infectious & endocrine diseases)
• 2.Imaging
(DVT, PTE )
• 3.Biopsy
(Cancer , vasculitis)
• 4.Autopsy
(neurologic diseases)
• 5.long-term follow-up
(SLE , MS)
EBMRC
2) Did the results of the test being evaluated influence the
decision to perform the reference standard? 2. Were all patients
subjected to the gold standard? Verification (work-up) bias
• You want to find out how good is exercise ECG
(“treadmill test”) for identifying patients with
angina
• The gold standard is angiography
• Best = all patients get angiography
• Verification (work-up bias) = only patients who
have a positive exercise ECG get angiography
EBMRC
3. Was there an independent, blind or objective
comparison with the gold standard? Observer bias
• You want to find out how good is exercise
ECG (“treadmill test”) for identifying patients
with angina
• All patients get the gold standard
(angiography)
• Observer bias = the Cardiologist who does the
angiography knows what the exercise ECG
showed (not blinded)
• Another example: The pulmonary nodule on
CT, and comparison to CXR
EBMRC
4) Were the methods for performing the test
described in sufficient detail to permit replication?
• This description should cover all issues
that are important in the preparation of
the patient (diet, drugs to be avoided,
precautions after the test), the
performance of the test (technique,
possibility of pain), and the analysis
and interpretation of its results.
EBMRC
Basic structure of diagnostic studies
Series of patients
Index test
Reference (“gold”) standard
Compare the results of the
index test with the reference
standard, blinded
EBMRC
Read this abstract
• Scan in UTI abstract
EBMRC
Gold
standard
• Scan in UTI abstract
Accuracy
Series
of
patients
Index
test
EBMRC
Series of patients
Index test
Reference (“gold”) standard
Compare the results of the
index test with the reference
standard
EBMRC
Women presenting
with history of
recurrent UTIs
Self diagnosis based
on symptoms
Positive urine culture
172 episodes:
Positive urine culture
in 144 (84%)
Do reports meet the standards?
• Between 1978 and 1993 the authors found
112 articles, predominantly in radiological
tests and immunoassays.
• Few of the standards were met consistently
- ranging from 46% avoiding workup bias
down to 9% reporting accuracy in
subgroups.
EBMRC
EBMRC
EBMRC
EBMRC
Appraising diagnostic tests: 3 easy steps
1. Are the results valid?
2. What are the results?
3. How Can the Results
be Applied to Patient
Care?
EBMRC
What are the Results?
Biostatistics Review
• Sensitivity : Of all the people with a particular disease, the
proportion who will test positive for it (PID)
• Specificity : Of all the people without the disease, the
proportion who will test negative for it (NIH)
A gold standard diagnosis is already known, (presumably
without error). The above terms are describing the
properties of a particular TEST.
EBMRC
What are the Results?
Biostatistics Review
Example: Suppose the Prevalence of a particular disease in a
population is 10%. (of 1000 people, 100 have the disease)
Dz (+) Dz (-)
Test(+)
90
90
Test(-)
10
100
810
900
Sensitivity: 90/100 = 90%
Specificity: 810/900 = 90%
EBMRC
What are the Results?
Biostatistics Review
• Positive Predictive Value : Of all the people who tested
positive for a disease, the proportion that actually has it
• Negative Predictive Value : Of all the people who tested
negative for a disease, the proportion that actually does not
have it
In these patients, what you know are their test results, from
which you are trying to determine whether they actually
have the disease.
EBMRC
What are the Results?
Biostatistics Review
Same example: Prevalence of a particular disease in the
population is 10%.
Dz (+) Dz (-)
Test(+)
90
90
180
Test(-)
10
810
820
EBMRC
PPV: 90/180 = 50%
NPV: 810/820 = 99%
Sens: 540/600 = 90%
Spec: 360/400 = 90%
What are the Results?
Note: The PPV and NPV of a test are DEPENDENT on
prevalence.
Suppose the Disease Prevalence is 60%
Dz (+) Dz (-)
Test(+)
540
40
580
Test(-)
60
360
420
600
400
EBMRC
Sens: 540/600 = 90%
Spec: 360/400 = 90%
PPV: 540/580 = 93%
NPV: 360/420 = 86%
What are the Results?
The use of PPV and NPV when describing a diagnostic test
has several drawbacks:
1.
2.
3.
They are dependent on the prevalence of disease
The prevalence of disease in the general population is
not the same as that of the patients you see in clinic/ER.
Not all tests have results that can be categorized as “+”
or “-”.
For these reasons, PPV and NPV may soon be considered
“Old School”.
EBMRC
In My Experience!
Likelihood of disease and the PPV
of diagnostic tests in specialty setting
A test with
Primary care setting
Specialty setting
98% sen &sp
p=2%
ppv=50%
p=30% ppv=95%
EBMRC
Try it



A disease with a prevalence of 4% must be
diagnosed.
It has a sensitivity of 50% and a specificity of
90%.
If the patient tests positive, what is the
chance they have the disease?
EBMRC
Prevalence of 4%, Sensitivity of 50%, Specificity of 90%
Disease +ve
Sensitivity
= 50%
2
4
100
Disease -ve
Testing +ve
9.6
96
False
positive
rate = 10%
EBMRC
11.6 people
test
positive……….
of whom 2
have the
disease
So, chance of
disease is
2/11.6 about
17%
Prevalence of 4%, Sensitivity of 50%, Specificity of 90%
• Doctors with an average of 14 yrs
experience
….answers ranged from 1% to 99%
….half of them estimating the
probability as 50%
Gigerenzer G BMJ 2003;327:741-744
EBMRC
What are the Results?
Likelihood Ratios
Simply stated, the Likelihood Ratio is how much more likely
someone is to get a positive test result if they have the
disease, compared to someone who doesn’t.
In fact, the LR is how much more likely someone is to get
any particular test result (positive, negative, “intermediate
probability, etc) if they have disease, compared to someone
who doesn’t.
EBMRC
What are the Results?
From our previous example:
The LR is (90/100) / (90/900) = 9.0
This is the LR for a (+) result. Someone with disease is 9
times as likely to test positive than someone without it.
Dz (+)
Dz (-)
Test(+)
90
90
Test(-)
10
810
100
900
Note also that :
LR = SENS / 1 - SPEC
WOWO
EBMRC
What are the Results?
The LR for a negative test is:
(10/100) / (810/900) = 0.11
Someone with disease is 0.11 times as likely (1/9) to test
negative than someone without it.
Dz (+)
Dz (-)
Test(+)
90
90
Test(-)
10
810
100
900
EBMRC
Likelihood Ratios
LR = 18
High
Pre-Test
Probability
Post-Test
Probability
Post-Test
Prob. 90%
Mr. A
Pre-Test Prob.
15%
Mrs. B
Pre-Test Prob.
40%
V/Q Scan
Results
Post-Test
Prob. 20%
LR = 1.2
EBMRC
Application: Using Post-test Probability
Above this point,
treat
Disease
ruled IN
Determined by:
Complications
Diseaseof untreated disease
Risks of therapy
not
Complications of tests
ruled
in
Cost
or out
Disease
ruled OUT
Below this point,
no further testing
EBMRC
What do likelihood ratios mean?
LR<0.1 = strong
negative test
result
LR=1
No diagnostic
value
EBMRC
LR>10 = strong
positive test
result
What do likelihood ratios mean?
McGee: Evidence based Physical Diagnosis (Saunders Elsevier)
EBMRC
%
Bayesian
reasoning
Pre test 5%
Post test 20%
? Appendicitis:
McBurney tenderness
LR+ = 3.4
Fagan
nomogram
%
EBMRC
Appraising diagnostic tests: 3 easy steps
1. Are the results valid?
2. What are the results?
3. How Can the Results
be Applied to Patient
Care?
EBMRC
How Can the Results be Applied to
Patient Care?
Will the results change my management
strategy?
• Always ask yourself the above before ordering a test
• What is your treatment threshold?
• What is your threshold for NOT treating?
If you find that most patients are having test results with
a LR near 1.0, the test is not very useful.
EBMRC
How Can the Results be Applied to
Patient Care?
Will patients be better off as a result of the test?
Do the benefits of performing the test outweigh the associated
risk?
If the answer to this is unclear, should a randomized
controlled trial be done to determine if the test actually
gives any benefit?
EBMRC
Summary
•Objectives of testing
•Diagnostic research
•Critical appraisal
EBMRC
•THANK YOU
EBMRC