Transcript Advanced Improvement Practitioner Programme

Advanced Improvement
Practitioner Programme
Measurement
Mike Davidge
Issues and questions
2
WHAT TYPE OF PERSON ARE YOU?
3
Are you a
plan-doer ?
Are you
an
auditor?
6
Are you an
improver?
5 Measurement Sins
Having no baseline
(or having a
compromised one)
Measuring the
wrong thing
Only collecting data
at 2 points in time
Inappropriate or
mindless use of
statistics
Presenting results in
a misleading way
7
7 Steps to measurement
1 Decide aim
2 Choose measures
3 Define measures
6 Review
measures
7 Repeat
steps
4-6
4 Collect
data
5 Analyse &
present
8
The Measures Checklist
• Part One
•
•
•
•
Why important?
Who owns?
Definitions
Goals
• Part Two
• Collect
• Analyse
• Review
9
Step 1: Decide your aim





1 Decide aim
2 Choose measures
3 Define measures
6 Review
measures
7 Repeat
steps
4-6
5 Analyse &
present
4 Collect
data






Specific
Measurable
Achievable
Realistic
Time-bound
A worthwhile topic
Outcome focused
Measurable
Specific population
Clear timelines
Succinct but clear
10
Step 2: Choose measures
1 Decide aim
2 Choose measures
3 Define measures
6 Review
measures
7 Repeat
steps
4-6
4 Collect
data
5 Analyse &
present
11
Useful Tools when choosing measures
Process maps
Driver diagrams
12
Using the process map
Start ?
Decision
Point ?
Handover ?
End ?
The ABCD of managing a service
•
•
•
•
Activity: what we have actually done
Backlog: what we should have done but didn’t
Capacity: what we could have done
Demand: what we should have done
Listed in order of
frequency of
measurement
Step 3: Define measures
1 Decide aim
2 Choose measures
3 Define measures
6 Review
measures
7 Repeat
steps
4-6
5 Analyse &
present
4 Collect
data
An operational definition is a description,
in quantifiable terms, of what to measure
and the steps to follow to measure it
consistently
15
Definitions Exercise
 Take a piece of A4 paper and
follow my instructions to get an
identical shape to mine
16
Types of calculation
•
•
•
•
Counts
Ratios or rates
Percentages & Proportions
Time between or
cases between
Types of calculation:
When to use what
• Counts
– when the target population does not change very much
– Example: Number of falls on an elderly ward (always full)
• Percentages
– when the numerator is a subset of the denominator
– Example: Percentage of patients who fell
• Ratios or rates
– Numerator and denominator are measuring different things
– Example: Falls per 100 bed days
• Time between or cases between
– When you are tracking a ‘rare’ event, say one that occurs less than once a
week on average
– Example: Days since a patient last fell on this ward
Step 4: Collect your data
1 Decide aim
2 Choose measures
3 Define measures
6 Review
measures
7 Repeat
steps
4-6
4 Collect
data
5 Analyse &
present
19
Baselines
100
90
80
70
60
50
40
30
20
10
0
My Project - Linear trend
F
M
A
M
A
S
O
N
D
My Project - seasonal dip
J
F
M
A
M
100
90
80
J 70J
60
Months
50
40
30
20
10
0
Something Important
J
100
90
80
70
60
50
J 40J
30
Months
20
10
0
Something Important
Something Important
My Project - random fluctuation
A
S
J
F
O
M
N
A
D
M
J
J
Months
20
A
S
O
N
D
Analyse and present
1 Decide aim
“The type of
presentation you use
has a crucial effect
on how you react to
data”
2 Choose measures
3 Define measures
6 Review
measures
7 Repeat
steps
4-6
4 Collect
data
5 Analyse &
present
21
21
Delayed transfers of care
Indicator description
Community Health Delayed transfers of care (NHS and
Social Care)
2012 /13
Annual/Year end
Target
Dec
Jan
Feb
22
29
54
46
What you get presented with
What did you decide to do?
RAG
l
YTD Target
22
YTD Actual
YTD RAG &
12 month
Trend

How we assess performance #1:
2 point comparisons
Last
This
quarter quarter Change
80
96
+20%
Why has the number of complaints
gone up? Our service is getting
worse. We need to do something!
What decision are you going to
make?

UNDERSTANDING & DEALING WITH
VARIATION IN ANALYSIS
25
“If I could reduce my message to management to just a few
words, I'd say it all has to do with reducing variation”.
W Edwards Deming
26
What’s a person’s normal
body temperature?
27
“Data contains both signal
and noise. To be able to
extract information, one must
separate the signal from the
noise within the data.”
Walter Shewhart
RUN CHARTS
29
Run charts
• Plot data in time order
Run charts
• Calculate and display median as a line
• Analyse chart by studying how values fall
around the median
30
Summarise – the median
• Median defined
– The median is the middle value of a finite list of numbers where the
numbers are ordered from lowest value to highest value. If there is an
even number of observations, the median is not unique, so one takes
the mean of the two middle values.
• Why do we use it?
– Not affected by extremely large or small values
– Half of values will always be below/ above the median
• What do we need to beware of?
– It tells us little about the spread of values
– Tedious to calculate by hand if number of values is large
31
Creating your own run chart
Using the complaints handout, graph
paper, ruler and a pencil:
•
•
•
•
•
Draw and label the axes
Plot the dots (each monthly value)
Work out the median
Add a title (with dates)
Add a legend
32
Does yours look like this?
Complaints during 2011 to 2012
60
50
40
30
20
2011
2012
Number
33
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
0
Jan
10
Rule #1: a shift
Above centre
Below centre
Time 
Time 
At least 6 points above centre line
At least 6 points below centre line
34
Rule #2: a trend
Upward trend
Downward trend
Time 
Time 
At least 5 points all in
upward direction
At least 5 points all in
downward direction
35
Rule #3: astronomical data point
Time 
36
Rule #4: runs
1st run
2nd run
Time 
Time 
Too many or too few ‘runs’
Use table to determine
Interpreting your run chart
• Using the 4 rules listed below, see if
you have any special causes in your
data
Rule #1: A shift in the process, or too many data points in a run
(6 points above or below the median)
Rule #2: A trend (5 all increasing or decreasing)
Rule #3: An “astronomical” data point
Rule #4: An unusual number of runs (use the table to determine)
38
Apply the rules
Complaints during 2011 to 2012
60
50
40
30
20
2011
2012
Number
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
0
Jan
10
SPC CHARTS
40
Control charts
•
•
•
•
Control charts
Plot data in time order
Calculate and display mean as a line
Calculate and display control limits as lines
Analyse chart by studying how values fall
around mean and between control limits
Complaints during 2006 to 2007
70
60
50
40
30
20
10
2006
Number
Dec
Nov
Oct
Sep
Jul
2007
Aug
Jun
May
Apr
Mar
Feb
Jan
Dec
Nov
Oct
Sep
Jul
Aug
Jun
May
Apr
Mar
Feb
Jan
0
An example SPC chart
Complaints during 2006 to 2007
70
60
50
40
30
20
10
2006
2007
Number
42
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
0
Summarise – the mean
• Mean defined
– the arithmetic mean (or simply the mean) of a list of
numbers is the sum of all of the list divided by the number
of items in the list.
• Why do we use it?
– Provides a useful estimate of the typical value
– Easy to understand and calculate
• What do we need to beware of?
– A few extremely large values can inflate the mean
– It tells us little about the spread of values
Creating your own SPC chart
• Use the SPC handout, ruler and a
pencil.
• This time the basic chart is drawn
for you.
• You will be calculating and drawing
the mean and control or process
limits
• Then interpreting your chart
44
1. Plot individual values
% weekly compliance with hand hygience April - Sept 2010
100
90
80
Percentage
70
60
50
40
30
20
10
0
Week
2. Calculate Mean & plot it
% weekly compliance with hand hygience April - Sept 2010
Mean = 58
100
90
80
Percentage
70
60
50
40
30
20
10
0
Week
In Excel use the formula
=AVERAGE(range)
3: Derive moving range
These are required to calculate the control limits
The first row contains the chart data
Use the second row to record the difference between successive data values
4
8
9
Moving
0 as a positive
5 value 1
The difference is always recorded
Range
4
1
X Data
0
0
5
4: Calculate Average Moving Range
R1
R2
R3
R4
R5
R6
R7
R8
7
23
25
25
16
20
33
35
R1
Add up the individual
moving range values
= 7+23+25+25+…+18
Divide by number of
values
= 463
24
Average moving range
= 19.3
R23
R24
13
18
Now what
does this
mean?
5: Calculate the control limits
First derive one measure of variation
(referred to as 1 sigma)
1 sigma = Average moving range
1.128
= 19.3
1.128
1 sigma = 17.1
Calculate lower limit as Mean – 3 sigma
Lower limit =58 – 3 x 17.1
Lower limit = 6.7
Calculate upper limit as Mean + 3 sigma
Upper limit =58 + 3 x 17.1
Upper limit = 109.3
5. Plot limits
% weekly compliance with hand hygience April - Sept 2010
Upper limit = 109
100
90
80
Percentage
70
60
50
40
30
Lower limit = 7
20
10
0
Week
Constructing the chart: Summary
There are 5 steps to constructing your chart:
1. Plot the individual values
2. Calculate the mean and plot it
3. Derive the moving range values
4. Calculate the average moving range (MR)
5. Derive upper and lower limits from this and
plot them
Control charts are like yoghurt ...
They come in different flavours!
The XmR chart (or I-chart)
X – for individual values
mR – for moving range
Makes fewest assumptions about how the data values are distributed
around the mean
Will give you reliable results in almost all situations
Some software to help you create SPC
charts
Limited version is
FREE, full version
costs £50
INTERPRETING SPC CHARTS
55
Source: Lloyd Nelson, Technical Aids: The Shewhart Control Chart – Tests
for Special Causes
SPC CHART RULES
56
Rule # 1:
Any single point outside the control limits
Upper control limit
Mean
Lower control limit
Rule # 2: A shift
At least 7 points consecutively either above or
below the centre line
Upper control limit
Mean
Lower control limit
Rule #3: A drift
At least 7 points consecutively ascending or
descending
Upper control limit
Mean
Lower control limit
Note: the points can cross the centre line
Rule #4:
At least 14 points alternating up and down
Upper control limit
Mean
Lower control limit
Note: the points do not have to alternate
above and below the centre line
Other tests require you to plot the
intermediate variation values
Mean+3V
Upper control limit
Mean+2V
Mean+1V
Mean
Mean
Mean-1V
Mean-2V
Mean-3V
Lower control limit
Rule #5:
2 out of 3 points outside 2V on the same side of
the centre line
Mean+3V
Upper control limit
Mean+2V
Mean+1V
Mean
Mean
Mean-1V
Mean-2V
Mean-3V
Lower control limit
Rule #6:
4 out of 5 points outside 1V on the same side of
the centre line
Mean+3V
Upper control limit
Mean+2V
Mean+1V
Mean
Mean
Mean-1V
Mean-2V
Mean-3V
Lower control limit
Rule #7: Reduced variation
15 points within 1V either side of the centre
line
Mean+3V
Upper control limit
Mean+2V
Mean+1V
Mean
Mean
Mean-1V
Mean-2V
Mean-3V
Lower control limit
Rule #8: 2 processes
8 points in a row on both sides of the centre
line with none within 1V
Mean+3V
Upper control limit
Mean+2V
Mean+1V
Mean
Mean
Mean-1V
Mean-2V
Mean-3V
Lower control limit
Why 7 points in a row?
• Toss a coin: Chances of getting a ‘head’?
• Toss it twice: Chances of getting two ‘heads’?
66
How many options?
1 toss
2 tosses
3 tosses
1 in 2
1 in 4
1 in 8
4 tosses
1 in 16
67
Chances of ‘n’ heads in a row
Number of
tosses
1
2
3
4
5
6
7
8
9
Chance of all
heads
1 in 2 or 50%
1 in 4 or 25%
1 in 8 or 12.5%
1 in 16 or 6.25%
1 in 32 or 3.13%
1 in 64 or 1.56%
1 in 128 or 0.78%
1 in 256 or 0.39%
1 in 512 or 0.20%
68
SAFETY CROSSES AND RUN CHARTS
69
Your Safety Cross
Record the
dates
04/07/2013
11/07/2013
25/07/2013
70
Calculate the days between falls
Date
04/07/2013
11/07/2013
25/07/2013
Days between
11 – 4 = 7
25 – 11 = 14
What value do
we get if we
have 2 falls on
the same day?
71
Plot the days between on a run chart
The higher the
value, the
better it is
Use the 3 run
chart rules to
interpret
Plot X axis as
text not dates
72
RECOMMENDED CHARTS
73
The Golden Rule
when presenting data
One picture, one message
74
Types of chart and when to use them
Run & control charts
• Used to display performance over
time
Complaints during 2011 to 2012
60
50
40
• Shows whether common causes or
special causes (or both) are present
in our data
30
20
2011
75
Dec
Oct
Nov
Sep
Jul
2012
Number
Aug
Jun
Apr
May
Mar
Jan
Feb
Dec
Oct
Nov
Sep
Jul
Aug
Jun
Apr
May
Mar
Jan
0
Feb
10
Types of chart and when to use them
Pareto charts
• Used to display the number of
times distinct things happen such
as reason for cancellation
• To separate the ‘vital few’ from the
‘useful many’ (the 80/20 rule)
76
Types of chart and when to use them
Bar charts
• Used to display survey
results
• Display for each
question – which
questions look
different?
• Also display optionally
for each respondent –
are there types of
respondent?
I feel confident working on the Duty desk
7
6
5
4
3
2
1
0
Strongly
disagree
Disagree
Neutral
Agree
Source: Call of Duty
77
Strongly
agree
QUALITATIVE ANALYSIS
Surveys
Issues to consider
• Are all patients rating the same thing?
• Is there a ceiling effect?
• Do scales work in a linear fashion?
– A score of 6 is twice as good as 3
– Strongly agree is twice as good as agree
Collecting patient experience data
Interviews
Surveys
Patient tracking
Analysing qualitative data
Thematic analysis:
Look for the
common themes
Construct a story
around typical
findings
The power of a
good quote
The power of a quote
Staff Experience Quotes
Before changes:
“It felt unsafe, anxiety provoking, uncontained and
filled with dread”
“Scared, dreaded being on duty at times left
feeling isolated, unsafe, unsure with no clear
structures in place to support me”
“Unclear, unsafe, unsupported”
After Changes:
“ I now enjoy working on the duty desk, it feels
like you can make a quick difference to peoples
lives when they are in distress”
“I enjoy the challenges duty now presents, I feel
more confident and safe in my practice, I feel it is
more of a team approach.”
“Challenging at times but stimulating, anxiety
provoking but only at times and I feel supported”
Source:
Call of Duty
Step 6 – Review measures
1 Decide aim
2 Choose measures
3 Define measures
6 Review
measures
7 Repeat
steps
4-6
4 Collect
data
It is a waste of time
collecting and analysing
your data if you don't take
action on the results
5 Analyse &
present
84
DTOCs part 2
Number
Delayed transfers of care
80
70
60
50
40
30
Verdict
20
01 Feb 2013
01 Jan 2013
01 Dec 2012
01 Nov 2012
Months
01 Oct 2012
01 Sep 2012
01 Aug 2012
01 Jul 2012
01 Jun 2012
BaseLine
01 May 2012
Not Capable
of achieving
target
01 Apr 2012
Stable within
10
limits 0
(10 - 72)
What decision do you make?
Decision
Because
Do nothing
Performance ok
Contingency plans
Special cause variation
Process redesign
Common cause variation
Is your information presented in a way that allows
you to confidently make one of these decisions?
86
Team Review Meetings
Step 7: Keep going
1 Decide aim
2 Choose measures
3 Define measures
You may not get it right first
time! You may need several
attempts. Remember
PDSA
7 Steps now on YouTube
6 Review
measures
7 Repeat
steps
4-6
5 Analyse &
present
4 Collect
data
http://youtu.be/Za1o77jAnbw
Or put ‘Mike Davidge measurement’
into YouTube search box
88
References
• The run chart basic reference
– “The run chart: a simple analytical tool for learning from
variation in healthcare processes”; Perla R, Provost L,
Murray S; BMJ Qual Saf 2011;20:46e51.
doi:10.1136/bmjqs.2009.037895
• A great introduction to variation and SPC
– Understanding variation, Don Wheeler,
www.spcpress.com, 1986
• A couple of useful websites/blogs
– www.davisdatasanity.com
– www.kurtosis.co.uk
89
Application to management
• Fourth Generation Management
– Brian L Joiner, McGraw-Hill, 1994
• The Leader’s Handbook
– Peter R Scholtes, McGraw-Hill, 1998
• The New Economics (2nd edition)
– W Edwards Deming, MIT Press, 1994
Issues and questions
91