Transcript Combining severity scores and organ dysfunction models

Combining severity scores and
organ dysfunction models
improves the accuracy of
prediction
JF Timsit, MD
Réanimation médicale et
infectieuse
Hôpital Bichat, Paris
Slides available on http://www.outcomerea.org
Combining Organ dysfonction and
severity scores

Is stupid (multicolinearity)

However the colinearity is not so important

It works : TRIO score
Using 2 scores….
Selection of variables is based on data reduction principle
–
–

Overfitting phenomenon:
–
–
–
–

Simple
Numerically stable, easily generalized
The estimates become unstable
Large SD of the beta estimates
Insufficient fit
Low applicability in an external data set
The mechanical selection is not sufficient…
Combining Organ dysfonction and
severity scores




Is stupid (multicolinearity)
However the colinearity is not so important
We have already the data collected
It works : TRIO score
Because variables enclosed in these
composite scores are different
SAPSII
Age
Type of adm.
Chronic HS
MV
Vasopressors
Temp
K, NA, CO3HPao2/fiO2
HR
BP
WBC
Platelets
PTT
Bilirubin
Urea
Creatinine
Urine output
GCS
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
LOD
SOFA
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
Because cutpoints are different


Optimized cutpoint for a particular continuous
variable depend to the distribution of the
variable in the training set
Then:
–
–
If various cutpoints of a prognostic variable are
tested, it influenced the other independent variables
selected in the final model.
The same biological parameter transformed as 2
dummy variables with different cutpoints and
weights could be partly non colinear
Buettner et al – J Clin Epidemiol 1997; 1201
Glasgow coma score
GCS
3
4
5
6
7
8
9
10
11
12
13
14
15
APACHE II
12
11
10
9
8
7
6
5
4
3
2
1
0
SAPS II
26
26
26
13
13
13
7
7
5
5
5
0
0
LOD
5
5
5
3
3
3
1
1
1
1
1
0
0
SOFA
4
4
4
3
3
3
3
2
2
2
1
1
0
MODS
4
4
4
4
3
3
3
2
2
2
1
1
0
Point and cutpoints of Glagow coma scale in various scores
Neurologic dysfonction
Mean(SD)
LOD
SOFA
MODS
1.2 (1.9)
1.2 (1.6)
1.3 (1.6)
Kendall’s W (agreement)
0.005
NS
Maximum score= score of the highest values for neurologic dysfn
Petilla et al – CCM 2002; 30: 1705-1711
Because none of the scores are
sufficiently accurate

Calibration and discrimination properties varies
from excellent to very poor
–
Different according to






Definitions of variables
Percentage of death
Countries
Acute Diagnosis,case mix
Quality of care…
Inter-observer variability (Kappa<0.9)
Discrepancies between predicted probabilities
Lemeshow, Intensive care Med 1995; 21:770
Mean dif: 1%
SD: 17%
Predicted prob:
Dif: 10-20%: 19% pts
Dif>20%: 19.8% pts
Discrepancies between predicted probabilities
Arabi et al –Crit Care 2002; 6:166
969 Pts/ 1 center
Rsquare
MPM II0 MPM II 24
APACHE II
SAPS II
MPM II0
MPM II 24
APACHE II
SAPS II
1
0.48
0.56
1
0.52
0.62
0.66
1
0.67
1
Because no interaction are taken
into account

ODIN model: two-way interaction tested
–

MODS: Not reported
–

Le Gall et al – JAMA 1996; 276:802
APACHE II: Not reported
–

Vincent et al – CCM 1998; 26:1793
LOD: Not reported
–

Marshall et al CCM 1995; 23:1638
SOFA: Not reported (validation) but suggested
–

Fagon et al – ICM 1993; 19:137
Knaus et al – CCM 1985; 13:818
SAPS II: Not reported
–
Le Gall et al – JAMA 1993; 270: 2957
Because no interaction are taken
into account
SOFA score : Organ component > 3
Resp
hemato liver
Cardiov
Neuro Renal
20.7%
16.7%
14.3%
27.9%
24%
Hemato
60.3%
59%
55.4%
48.1%
57.4%
Liver
65.6%
69.2%
73.8%
72.3%
Cardio-v
71.2%
67.6%
73.8%
Neuro
64.7%
74.3%
Renal
66.7%
23%
No
interaction
43.7%
Vincent et al – CCM 1998; 26:1793
Because no interaction are taken
into account
In the SOFA and LOD scores organ components are
assuming to be independent:
 OUTCOMEREA database
Associations among involvements of the six organ
systems.(Kendall's b correlation coefficient )

–
–
SOFA: Strong positive correlations between Day0 and Day7
except for pulmonary versus liver dysfunction and for
pulmonary versus hematological dysfunction on days 1, 2, and
4.
LOD: Strong positive correlation between Day0 and Day 5
Timsit et al – CCM 2002: 30: 2003-2013
Combining Organ dysfonction and
severity scores




Is stupid (multicolinearity)
However the colinearity is not so important
Severity scores are mandatory (ressource
allocation) and organ dysfunction score is
useful
It works : TRIO score
Mortality (%)
SOFA score: initial and delta score
100
90
80
70
60
50
40
30
20
10
0
0-1
-2-7
-8 - 11
>11
Decrease
No decrease
Initial SOFA score
Vincent JL et al – JAMA 2001; 286:1754
Combining Organ dysfonction and
severity scores




Is stupid (multicolinearity)
However the colinearity is not so important
If daily scores are important initial value is
important
It works
Outcomerea database

Daily clinical and biological data (multicenter french
database)

Day 1 (n=1673 pts)
–
–

–
AUCROC:0.78, HLstat:=0.39
APACHE 2
SOFA
p<0.0001
p<0.0001
AUCROC:0.78, HLstat:=0.57
p<0.0001
p<0.0001
AUCROC:0.8, HLstat:=0.18
p<0.0001
p<0.0001
AUCROC:0.8, HLstat:=0.12
Day 3 (n=1336 pts)
–
–

p<0.0001
p=0.0003
Day 2 (n=1571 pts)
–

APACHE 2
SOFA
APÄCHE 2
SOFA
Day 7 (n=700 pts)
–
–
APACHE 2
SOFA
Outcomerea database

Daily clinical and biological data (multicenter french
database)

Day 1 (n=1673 pts)
–
–

–
AUCROC:0.77, HLstat:=0.26
SAPS 2
LOD
p<0.0001
p=0.0002
AUCROC:0.78, HLstat:=0.15
p<0.0001
p=0.0003
AUCROC:0.79, HLstat:=0.83
p<0.0001
p=0.01
AUCROC:0.77, HLstat:=0.76
Day 3 (n=1336 pts)
–
–

p<0.0001
p=0.001
Day 2 (n=1571 pts)
–

SAPS 2
LOD
SAPS 2
LOD
Day 7 (n=700 pts)
–
–
SAPS 2
LOD
…even when the scores have been built in the same database
European sepsis database


8353 patients, trained data collector + audit…
Correlation between LOD and SAPSII scores:
–

R2=0.711
Multiple logistic regression
Parameter
Intercept
SAPS
LOD
DF
Estimate
S. Error
Chi-Square
1
1
1
-3.7006
0.0534
0.0933
0.0888
0.00319
0.0152
1738.4827
280.1879
37.5703
Special thanks to Corinne Alberti
Pr > ChiSq
<.0001
<.0001
<.0001
TRIO score
Rationale:
To develop a score adapted to patients exposed to N.I.
(hospitalized more than 72 hours)
Easy to collect, reproducible
Parameter
estimate
Intercept
Transfer from ward (0/1)
LOD at admisson
SAPS II admission
Chronic illness (0/1)
SAPS2-SAPS3 alteration
LOD2-LOD3 alteration
-4.44
0.5543
0.1536
0.0388
0.8507
0.4161
0.6940
Odds ratio
95% CI
P value
(Wald)
1.74 [1.25-2.42]
1.16 [1.085-1.253]
1.04 [1.027-1.053]
2.34 [1.677-3.269]
1.516 [1.04-2.22]
2.00 [1.29-3.11]
0.0001
0.001
<0.0001
<0.0001
<0.0001
0.032
0.0002
Odds ratio
95% CI
(Bootstrap)
[1.253-2.453]
[1.093-1.276]
[1.026-1.053]
[1.622-3.296]
[1.055-2.373]
[1.292-3.019]
893 Pts (268 Hosp. deaths)
Outcomerea database, Timsit et al, ICM 2001;27:1012
Accuracy of SAPS, LOD MPM 72 and TRIO
score
AUC-ROC (95 % CI)
Hosmer-Lemeshow C statistic
(p value)
0.744 (0.714-0.773)
0.681 (0.646-0.714)
0.786 (0.757-0.812)
0.794 (0.766-0.820)
37.4 (0.001)
21.2 (0.01)
22.3 (0.01)
5.56 (0.70)
0.741 (0.688-0.789)
0.725 (0.668-0.777)
0.814 (0.774-0.854)
0.826 (0.780-0.867)
19.8 (0.02)
16.6 (0.05)
18.2 (0.03)
7.14 (0.5)
Training set
SAPS II
LOD at admission
MPM72
TRIO score
Validation set
SAPS II
LOD at admission
MPM72
TRIO score
External validation: 24 ICUs France
Outcomerea database, Timsit et al, ICM 2001;27:1012
Conclusion

Combining Organ dysfonction and severity
scores
–
–
–
If both scores are routinely measured
Even non logical
It could be considered as an alternative to a new
severity score (easiness of record, habits,
reproducibility)


In particular subsamples
Awaiting new scores….
Slides available on http://www.outcomerea.org