Transcript Key Performance Indicators, Centre Reports, and more

Key Performance Indicators,
Centre Reports, and more
Stephen McDonald
Barbecue talk
120
100
80
60
40
20
1970
1980
1990
Year
2000
Incident rate
Rate
Incident RRT, Australia only
95% CI
2010
More “good” news
Age-specific incident RRT rates
Australia
0-24
25-34
20
45-54
70
200
60
15
150
50
10
40
100
5
30
50
65-74
75-84
500
0
1980
1990
2000
2010
85+
600
300
400
200
200
100
0
0
1980
1990
2000
2010
Year
Rate
Graphs by age group
95% CI
1980
1990
2000
2010
Indigenous incidence rates
Aboriginal & TSI, Australia
500
400
300
200
100
0
1985
1990
1995
2000
Year
Rate
95% CI
2005
2010
Background
• A number of ongoing work themes exist
within ANZDATA for generating output
– Stock and flow figures
– Annual Report
– Contributor requests
• Responses to information needed for various
projects
– Research projects (internal and external
analyses)
– Outcomes reporting
Outcomes reporting
• Recent years have seen a growth of
interest in outcomes reporting
• Centre reports have been part of
ANZDATA for many years, with
increasing emphasis in recent years
– At “parent hospital level”
– Limited distribution historically
Why measure outcomes?
4
2
1
.5
.2
0
20
40
units
O/E
60
98% CI
All Australia & NZ Dialysis Units, 98% confidence intervals
80
Dialysis outcome
Adjusted relative risk
4
2
1
.5
.2
0
20
40
60
Units, ranked by RR
RR
95% CI
Mortality rate during dialysis treatment in Australia 2006-10, adjusted
for demographics and comorbidities
80
Variation in transplant
outcomes
50
RR graft failure
20
10
5
2
1
.5
.25
0
5
10
Units
RR
15
95% CI
Fully adjusted 1 year graft survival, by unit
All transplant units, Australia and New Zealand, patients transplanted
2005-2019, followup to 2010
20
What is happening to centre
reports?
• Greater reporting of demographics and
comorbidities
• Adjusted analyses in transplanting centre
and dialysis reports
– Details of models supplied
• Graphs
– Funnel plots
– CUSUM plots (transplant)
Centre reports – graph 1
Survival from 90th Day of Treatment
1.00
P atie nt S u rviva l
0.75
0.50
0.25
CNAR
Australia
New Zealand
0.00
0
1
2
3
Years
4
5
Centre reports – graph 2
Technique Survival - PD at 90 days
1.00
0.75
0.50
0.25
CNAR
Australia
New Zealand
0.00
0
1
2
3
Years
4
5
But....
100
80
60
40
20
0
Everywhere else
CNARTS
Adjusted graphs
Adjusted SMR (95% CI)
1.3
A dju sted S M R
1.2
1.1
1
.9
.8
.7
CNAR
Australia
New Zealand
Adjusted graphs
2.5
2
SMR
1.5
CNAR
1
.5
0
0
50
100
Expected Number of Deaths
150
200
How are reports derived?
You need a model
• Logistic regression model (transplant),
Poisson model (dialysis)
• Adjusted for demographics, comorbidities
(donor and XM variables)
• With this model, derive a probability of
“expected” failure for each person / graft
based on covariate matrix
• Compare this with actual outcomes
www.anzdata.org.au
Which predictors are important?
0.7
0.6
0.5
0.4
Harrell's C
Somer's D
0.3
0.2
0.1
0
Recipient age +comorbidities
gender & graft
number
+ HLA
matching
+ ischaemic
time
+ donor age + cause donor
death
Predictive power of multivariate Cox model predicting graft survival, all DD transplants 2001-2009,
with sequential addition of covariate groups
Don’t adjust for…
• Factors within the control of centre
– These may be why a particular centre gets
good or bad results
• Factors that occur as a result of treatment
decisions
• For example, don’t adjust for
– Choice of dialysis modality, HD access
– Use of immunosuppressives, rejection, 1
month graft function…
www.anzdata.org.au
Other graphical
demonstrations of output
• Funnel plots are a static measure and
summarise performance (relative to a
comparator) over a fixed period of time.
– Lack a dynamic element
– Weight recent and distant results equally
Adding time – CUSUM
Twoway CUSUM for a transplant centre
4
2
300
0
200
-2
100
-4
0
01jan2009
01jan2008
01jan2007
01jan2006
01jan2005
01jan2004
Tx date
Number of tx
Cumulative sum O-E
400
Removing credit for good
deeds
Oneway CUSUM for for a hospital
5
4
3
2
1
0
400
300
200
100
0
Tx number
Do we need to do more?
Why KPIs?
• Mortality is an insensitive and late
indicators of problems
– Hopefully rare
– Outcome of complex series of events
• Incompletely ascertained
– Important to monitor as best we can
• Key Process indicators
– Simpler to understand, easier to address
– Need to be valid and correctable (and related
to meaningful outcomes)
KPI Project
• Dialysis KPI project commenced 2011
– At instigation of DNT committee
• 2 markers chosen – Peritonitis and HD
access at first treatment
– Deliberately limited to existing data collection
• NO additional data collected
– Based on real time ANZDATA data collection
Variation in HD access
1
9
7
5
8
7
.8
10
55
18 28
.6
8
31
10
15
17
22
25
11
12
19
47
12
68
15
13
10
28 54
19
23
17
20
314433
44
64
29 34344144
29
434452
52
75
36
68
113
66
65
40
.4
.2
0
0
20
40
Centres
Proportion
ANZDATA, access at first HD where first dialysis
95% CI
60
Variation in peritonitis rate
Peritonitis rates by treating unit
2009 only
2.5
2
1.5
1
.5
27
4
Patient-months per episode
3
Confidence intervals not shown where upper limit >3
Units with <5 person-years PD over 2009 not shown
6
12
24
KPI reporting -- access
• Quarterly identified feedback to units
Peritonitis reporting
Where to from here?
•
•
•
•
COMMUNICATE
Improve data collection
Improve access to results
Enhance reporting
– Add peritonitis rates
– Access subdivided by late referral
– Graphs etc etc
• Or is it all just too hard?
How do we view quality?
Centre reports -- SMR