Statistics in Screening/Diagnosis

Annie Herbert Research & Development Department Salford Royal Hospitals NHS Foundation Trust [email protected]

0161 2064567

Outline

• Intro: Design, recording results • Sensitivity • Specificity • Continuous variables: ROC curves • Predictive values • Likelihood ratio • Bias

Introduction: diagnostic test

Patient enters clinic (Patient has disease). Takes test, e.g. blood sample (Patient doesn’t have disease). Takes test, e.g. blood sample Test result: Positive (right diagnosis, ‘true positive’) Test result: Negative (wrong diagnosis, ‘false negative’) Test result: Positive (wrong diagnosis, ‘false positive’) Test result: Negative (right diagnosis, ‘true negative’)

Introduction: assessing a diagnostic test

All participants Index test Reference test 2 x 2 Table of results 2 x 2 Table of results Reference test + + 90 10 60 240

Introduction: 2x2 table of results

‘TRUTH’ (by Reference test – Gold standard) + Total TEST RESULT (by Index test) + True Positive False Positive Total said to have disease False Negative True Negative Total said not to have disease Total Total with disease Total without disease

Hypothetical example – Breast cancer screening study

• Gold standard: Mammography • Cheaper/more convenient option: GP examination

Breast cancer screening results

TEST RESULT (by GP Exam) + ‘TRUTH’ (by mammography) + 95 45 5 855 Total 100 900 Total 140 860 1000

Sensitivity - Definition

• What proportion of people who have the condition are identified as positive by the test?

• If a test has a high sensitivity, most people with the condition are picked up by the test.

Have condition +ve test

Sensitivity - Calculation

TEST RESULT Total + + ‘TRUTH’ (by gold standard) a c b d a+c b+d Sensitivity = a/(a+c)

Sensitivity - Example

GP Exam Total + Mammography + 95 5 100 45 855 900 Sensitivity = 95/100 = 0.95

I.e. 95% of patients diagnosed as having breast cancer by the mammogram are picked up by GP examination.

Specificity - Definition

• What proportion of people who don’t have the condition are identified as negative by the test?

• If a test has a high specificity, most people without the condition are ruled out by the test.

Don’t have condition -ve test

Specificity - Calculation

TEST RESULT Total + + ‘TRUTH’ (by gold standard) a c b d a+c b+d Specificity = d/(b+d)

Specificity - Example

GP Exam Total + Mammography + 95 5 100 45 855 900 Specificity = 855/900 = 0.95

I.e. 95% of patients diagnosed as not having breast cancer by the mammogram are ruled out by GP examination.

Sensitivity & Specificity - Notes

• It is essential to have a confirmed ‘true’ diagnosis (+ve or -ve). The accuracies of sensitivity and specificity are only as good as that of the gold standard.

• Sensitivity and specificity are estimated from a sample, and so should be accompanied by confidence intervals to convey amount of uncertainty. (StatsDirect:

Analysis

->

Proportions -> Single

)

Tests based on continuous variables (1) • One or more continuous variables can be a marker for a condition, where a very low/high level indicates a low/high likelihood of having the condition.

• A cut-off level can be determined where having higher/lower than that cut-off indicates a positive test result.

• Different cut-off points will give different sensitivity/specificity values.

Tests based on continuous variables (2) E.g. Creatinekinase in patients with unstable angina or acute myocardial infarction angina myocardial infarction Data of Frances Boa, from ‘An introduction to Medical Statistics’ by Martin Bland

Tests based on continuous variables (3) Test Total + + Truth 27 0 27 54 39 93 Sensitivity = 27/27 = 1.0

Specificity = 39/93 = 0.42

angina myocardial infarction Cut-off level at 80

Tests based on continuous variables (4) Test Total + + Truth 26 1 27 35 58 93 Sensitivity = 26/27 = 0.96

Specificity = 58/93 = 0.62

angina myocardial infarction Cut-off level at 100

The trade-off

• Plot sensitivity against (1-specificity) to get the ROC (‘receiver operating characteristic’) curve.

• Ideally want high sensitivity and high specificity (but increase in one is at expense of the other).

• Also requires some clinical judgement, e.g. Likely considered better to send women without breast cancer to have a mammogram than give those with breast cancer the all clear.

• Check sensitivity and specificity values in a new sample.

Sensitivity = 1.0

Specificity = 1.0

Sensitivity 1.00

0.75

0.50

0.25

0.00

0.00

0.25

ROC curve

ROC plot for MI data

0.50

0.75

1-Specificity 1.00

The diagonal line represents sensitivity = specificity, i.e. taking the test is as good as flipping a coin.

Sensitivity = 0.0

Specificity = 0.0

Optimum cut-off

MI data: • ‘Optimum’ cut-off point selected = 302 • Sensitivity (95% CI) = 0.93 (0.76 to 0.99) • Specificity (95% CI) = 0.97 (0.91 to 0.99) Note: ‘optimum’ assumes sensitivity and specificity of equal concern.

Area under the ROC curve

• Area under the ROC curve can be between 0 (sensitivity and specificity always 0.0) and 1 (sensitivity and specificity always 1.0).

• Can be useful for comparing two tests. • MI data: Area under curve is an estimate of ‘probability that creatinekinase of random person with MI will be higher than for random person with angina’.

The difference between sensitivity & specificity and predictive values…

• Sensitivity & Specificity: How good is the test at making the right diagnosis?

• Predictive Values: Once diagnosis has been made, how reliable is it?

Positive Predictive Value - Definition

• Proportion of those with a positive test result that actually have the condition.

• If a test has a high positive predictive value, if someone tests positive for the condition, there is a high probability that they have it.

Test positive Have condition

Positive Predictive Value - Calculation

TEST RESULT + ‘TRUTH’ Total + a b a+b c d c+d PPV = a/(a+b)

Positive Predictive Value - Example

‘TRUTH’ (by mammogram) Total + TEST RESULT (by GP Exam) + 95 5 45 855 140 860 PPV = 95/140 = 0.68

I.e. 68% of patients who test positive for breast cancer by GP examination could be expected to test positive by mammogram.

Negative Predictive Value - Definition

• Proportion of those with a negative test result that really don’t have the condition.

• If a test has a high negative predictive value, if someone tests negative for the condition, there is a high probability that they don’t have it.

Don’t have condition Test negative

Negative Predictive Value - Calculation TEST RESULT + ‘TRUTH’ Total + a b a+b c d c+d NPV = d/(c+d)

Negative Predictive Value - Example

‘TRUTH’ (by mammogram) Total + TEST RESULT (by GP Exam) + 95 5 45 855 140 860 NPV = 855/860 = 0.99

I.e. 99% of patients who test negative for breast cancer by GP examination would be expected to test negative by mammogram.

Prevalence

• What proportion of people in a cohort have the disease? E.g. “The prevalence of breast cancer in females over 40 years of age is approximately 1.5%”.

• ‘Prevalence’ is not the same as ‘incidence’.

• Sensitivity & specificity values are unaffected by prevalence, though predictive values are.

E.g. Test with 95% sensitivity and 95% specificity:

MeReC Briefing: supplement to issue 30

Example: Self administered cognitive screening test (TYM) for detection of Alzheimer’s disease: cross sectional study, Brown et al, June 2009 “A score of 42/50 had a sensitivity of 93% and specificity of 86% in the diagnosis of Alzheimer’s disease. The TYM was more sensitive in detection of Alzheimer’s disease than the mini-mental examination, detecting 93% of patients compared with 52% for the mini-mental state examination. The negative and positive predictive values of the TYM with the cut off of 42 were 99% and 42% with a prevalence of Alzheimer’s disease of 10%.”

Likelihood Ratio - Definition

• How many times more (or less) a patient with the condition is likely to have that particular result than a patient without the disease.

• Can be used to calculate the probability of individual patient having condition based on test results.

Use of Fagan's nomogram for calculating post-test probabilities: Copyright ©2004 BMJ Publishing Group Ltd.

Deeks, J. J et al. BMJ 2004;329:168-169

Bias in studies:

• Is the reference appropriate?

• Was the same reference used for all patients (verification bias)?

• Were assessors blind to case details?

• Was it a ‘diagnostic case-control study’?

• Which population was the test tested in?

Summary

• All patients must have both new test & reference test (gold standard).

• Report 2x2 table and give sensitivity, specificity with precision.

• A good screening test is not necessarily a good diagnostic test.

• Test cut-offs in an independent sample.

• Predictive values vary according to prevalence.

• Consider all potential sources of bias.

Recommended Texts

• BMJ Statistics Notes: – 1) Sensitivity & Specificity – 2) Predictive Values – 3) ROC Curves – 4) Likelihood Ratios • Assessing bias – How to read a paper: Papers that report diagnostic or screening tests by Trisha Greenhalgh; BMJ 1997 315 pg. 540.