Transcript SCreening for Occult REnal Disease (SCORED), a simple

SCreening for Occult REnal
Disease (SCORED)
Simple Algorithms to Predict
Kidney Disease: ready to be used
in the real world?
Heejung Bang, PhD & Madhu Mazumdar, PhD
Division of Biostatistics and Epidemiology
Department of Public Health
Weill Medical College of Cornell University
1
Overview
 Background
 Objectives
 Methods: model development and
validation
 Results
 Discussion
2
Background
Prevalence of Kidney Disease (1999-2004)
6
5.7
5.4
5.4
5
Stage 1 GFR>90
Stage 2 90-60
Stage 3 59-30
Stage 4 29-15
Stage 5 <15
4
3
2
1
0.3
0.1
0
Stages 1 and 2 with kidney damage
3
Background
End-Stage Renal Disease (ESRD) Counts
4
Background
Total Cost of Medicare for ESRD (in billions)
30
95% CL
Projection
Cost
24
28.3
18
14.2
12
6
0
1982
1984
1986
1988
1990
1992
1994
1996
1998
2000
2002
2004
2006
2008
2010
5
Background
 Chronic kidney disease (CKD) is a global health problem.
Low-awareness and late detection are common problems.
 It is progressive disease. Yet, most affected individuals are
asymptomatic with known risk factors and are not routinely
tested.
 Identifying individuals with CKD should be ‘simple’ with
serum creatinine concentration that is widely available and
inexpensive ($10-20), in combination with urinalysis.
 Systematic methods to predict disease in other chronic
conditions such as cardiovascular disease (e.g.,
Framingham, Reynolds scores, stroke instrument), cancer
(e.g., Gail model), diabetes exist but not for CKD.
6
Objectives
 To develop risk prediction model for prevalent CKD
 Important prerequisites in our investigation:
Easy to use but accurate
 Cumulative effects of concurrent risk factors
 Demographic + medical history + modifiable risk factors
 To test the validity of the model internally as well as using
independent large databases (i.e., external validation)
 To compare the performance of the model with the current
clinical practice guidelines
 To develop risk prediction model for incident CKD

7
Kidney Early Evaluation Program (KEEP)
by the National Kidney Foundation
if a persons is ≥ 18 years old and has one or
more of the following:
1. diabetes
2. high blood pressure
3. a family history of diabetes, high blood
pressure or kidney disease
http://www.kidney.org/news/keep/
8
SCreening for Occult
REnal Disease
(SCORED)
Bang et al. (2007)
9
Methods
 Cross-sectional analysis of a nationally
representative population based survey, the
National Health and Nutritional Examination
Surveys (NHANES) 1999-2002
 Adult subjects only (≥20 years old)
 Potential risk factors searched from literature
 Endpoint: CKD stage 3 or higher, i.e., glomerular
filtration rate (GFR) < 60 ml/min/1.73m2 (using the
MDRD formula)
10
Methods (Cont’d)
 Split-sample method to create a development and
validation dataset using a 2:1 ratio.
 Standard diagnostic characteristics: # at high risk,
sensitivity, specificity, positive & negative predictive
values, area under ROC curve
 Multiple logistic regression model (with proper
weighting and complex survey design)
e.g., proc surveylogistic in SAS.
11
Methods (Cont’d)
 ‘Categorical scoring system’ derived by assigning an
integer for the regression coefficients
 ‘Continuous probability’ of having CKD from the fitted
regression model
 External validation using the Atherosclerosis Risk in
Communities (ARIC) Study, Cardiovascular Health Study
(CHS) and NHANES 2003-2004.
 Comparison between SCORED vs. KEEP using standard
diagnostic measures
 A number of sensitivity analyses (e.g., missing info,
different definitions)
--- important to be used in the real world!
12
Results
 NHANES 1999-2002 gave 10,291 individuals
 After exclusions (based on unmeasured or
missing data, etc.), dataset included 8,530
observations
 A total of 601 individuals had CKD (5.4%
weighted proportion)
13
Final SCORED model in development data
(N= 5,666, AUC=0.88)
14
Diagnostic characteristics of SCORED in
internal validation dataset (N=2,864)
(cutpoint ≥4 to define high risk group)
15
Event rate by risk score
16
Fitting SCORED model to ARIC dataset
(N= 12,038, AUC=0.71)
17
Sample questionnaire
18
Advantages of SCORED
 Estimate the cumulative likelihood of having disease
with multiple risk factors
 Accuracy and high sensitivity.
 Simple to use (implemented by the pen & pencil
method) so foresee a variety of uses
e.g., mass screenings
public education initiatives, health fair
medical emergency departments
web-based medical information sites
patient waiting room in clinics
19
Limitations of SCORED
 Inability to assess family history of kidney disease
-- many large national and community studies do not
enquire about history of kidney disease.
 For prevalent disease, not incident disease (a new
risk score is needed, later in this talk)
 Some variables may be commonly missing (e.g.
proteinuria)
 Low PPV (but prediction is HARD!)
 Kidney disease: multiple definitions, different stages
20
Diagnostic performance of SCORED vs. KEEP
using external validation data
(Bang, Mazumdar et al. 2008)
Screening guidelines
% high risk
Sensitivity
Specificity
PPV
NPV
AUC
NHANES
40
95
65
20
99
0.88
ARIC/CHS
51
88
52
14
98
0.78
ARIC/CHS*
53
89
50
13
98
0.79
ARIC/CHS*
53
90
50
13
98
0.80
NHANES
67
90
35
12
97
0.75
NHANES*
69
92
33
12
98
0.77
ARIC
76
88
24
3
98
0.67
ARIC/CHS
77
86
24
9
95
0.65
SCORED
KEEP
21
* some sensitivity analyses
A simple algorithm to predict
incident kidney disease
(aka, SCORED II)
by Kshirsagar, Bang et al. In Press
22
Prediction is very hard, especially
about the future - Yogi Berra
23
Background
 Another important issue is to predict a new disease in
disease-free population.
 In many asymptomatic diseases, both prevalent and
incident diseases are important. (in contrast, for hard
outcomes such as heart attack, only incident disease
makes sense)
 Incident disease is less urgent so less user-friendliness
is acceptable.
--- 3 different models developed: 1) best-fitting
continuous, 2) best-fitting categorical, 3) simplified
categorical.
 Beyond AUC. We also used AIC/BIC.
24
Background (Conti’)
 We need prospective studies to develop the
models.
 Internal validation only using Split-sample, no
external validation.
 Same logistic regression --- so observed
outcome among survivors.
 Cutpoint for high risk group might be less
important.
25
Simplified categorical model
(AUC=0.69, AIC=6295, BIC=6374)
Beta coefficient
(standard error)
Odds Ratio
(95% Cl)
P value
Assigned
score
Age 50-59
0.63 (0.12)
1.9 (1.5, 2.4)
<0.0001
1
60-69
1.33 (0.12)
3.8 (3.0, 5.8)
<0.0001
2
70 or older
1.46 (0.14)
4.3 (3.3, 5.6)
<0.0001
3
Female
0.13 (0.07)
1.1 (1.0, 1.3)
0.05
1
Anemia
0.48 (0.20)
1.6 (1.1, 2.4)
0.02
1
Hypertension
0.55 (0.07)
1.7 (1.5, 2.0)
<0.0001
1
Diabetes mellitus
0.33 (0.10)
1.4 (1.2, 1.7)
0.0006
1
History of cardiovascular
disease
0.26 (0.10)
1.3 (1.1, 1.6)
0.009
1
History of heart failure
0.50 (0.25)
1.6 (1.0, 2.7)
0.04
1
Peripheral vascular disease
0.41 (0.13)
1.5 (1.2, 1.9)
0.002
1
Covariate
26
Risk prediction table for
up to 10 years
Total score
Estimated Risk (%)
≤1
2
3
≤5
8
13
4
5
6
7
20
25
30
35
≥8
≥50
27
Discussion
 Evidence-based medicine = Science (theory) + Data +
Statistics.
 Risk score = Statistics + Art + Reality
--- SCORED is a good example.☺
 Performed well in a variety of different settings.
 Seems to provide the enhanced guidelines upon the current
clinical practice guidelines.
 It started be utilized in the ‘real world’.
 SCORED II yet to be validated but strong consistency/
similarities observed in SCORED I and II.
28
Discussion (Conti’)
 Categorization can be a bad idea (Royston et al. 2005;
Greenland 1995) but is crucial for risk scoring algorithms to be
useful in the real world.
 More than 1 model may be justified and we can let
consumers/users to choose because
All models are wrong, but some are useful ---George Box
 Relying on only 1 measure (e.g., AUC) can be problematic
(Cook et al. 2006; Cook 2007).
 Trade-offs between accurate vs. easy medical terms.
 Risk scores for internet vs. physician’s office vs. Walmart can
be different.
29
Current and future research
 Evaluation of SCORED in vascular patients
because detection of CKD in patients with or at
increased risk of CVD was emphasized by a
science advisory from the American Heart
Association and National Kidney Foundation
(2006).
 Relationships SCORED with other risk scores
 Testing SCORED in community settings
30
References
 Bang, Vupputuri, Shoham et al. (2007). SCreening for Occult REnal




Disease (SCORED). A simple prediction model for chronic kidney disease.
Archives of Internal Medicine.
Bang, Mazumdar, Kern et al. (2008). Validation and Comparison of a novel
prediction rule for kidney disease: KEEPing SCORED. Arch Int Med.
Kshirsagar, Bang, Bomback et al . A simple algorithm to predict incident
kidney disease. In Press. Arch Int Med.
Bang, Mazumdar, Newman et al. Screening for kidney disease in vascular
patients. Submitted.
Building and Using Disease Prediction Models in the Real World.
Roundtable discussion led by H. Bang at JSM, Utah, 2007. Slides at:
http://www.med.cornell.edu/public.health/conference_presentations.htm
31
Exposed to and used by public
 Covered by the CBS Early Show (on World Kidney
Day 2007)
 SCORED questionnaire is posted in some health
information websites
 Distributed by ESRD network, KidneyTrust, Am
Kidney Fund, UK Dept of Health, and UNC Kidney
Center for Kidney Education Outreach Program
 “Research Highlights” in Nature Clinical Practice
Nephrology (2007)
 Lead Story in Physician’s Weekly (2007)
32