Measurement for Improvement

Why we look at data graphed over time

# immunizations to children

125 100 75 50 25 0 Mar Change to process made in June Sep 125 100 75 50 25 0

# immunizations to children

125 100 75 50 25 0

# immunizations to children

Change Made Change Made 125 100 75 50 25 0

# immunizations to children

125 100 75 50 25 0

# immunizations to children

Change Made Change Made

System of Feedback P B 1 P 2 O 1 P 1 O 2 P 2 B S 1 S 2 S 3 PDSA 4 PDSA 3 PDSA 2 PDSA 1 Improvement Science Consulting S 1 S 2 PDSA 4 PDSA 3 PDSA 2 PDSA 1 S 3 Measure Types O = Outcome Measure P = Process Measure B = Balance Measure S = Process Step Measure PDSA = Learning Cycle Measure © Improvement Science Consulting

Break out – developing a useful project level dashboard

• Spend some time developing the measures you would like to include in a project level dashboard • Which ones are outcome measures?

• Which ones are process measures?

• Do you have balance measures?

• Process step measures?

• Note any gaps in your measures – where would you like to add measures?

System of Feedback P B 1 P 2 O 1 P 1 O 2 P 2 B S 1 S 2 S 3 PDSA 4 PDSA 3 PDSA 2 PDSA 1 Improvement Science Consulting S 1 S 2 PDSA 4 PDSA 3 PDSA 2 PDSA 1 S 3 Measure Types O = Outcome Measure P = Process Measure B = Balance Measure S = Process Step Measure PDSA = Learning Cycle Measure © Improvement Science Consulting

As the scale of the test increases we move from qualitative to quantitative evidence

Sequence of learning and change

Evidence primarily Qualitative Very small scale test of a change idea Improvement Science Consulting Evidence primarily Quantitative with noticeable impact on process measures Large scale test of change idea or Implementation of a change idea

Run Charts Making and Interpreting

So what is a run chart?

Murray and Provost, Pg 3-4

Defining elements of a run chart

• contains at least 10 data points • must have a median • tells the story through careful use of annotation

What time is it?

• Stop what you are doing • Look at your watch • Write down the time

How do we prevent this?

How do we interpret variation?

• Distinguishing between random variation and non random variation • Four rules for discovering non-random variation • Shift • Trend • Too many or too few runs • Astronomical values

Shift Rule: Six or more consecutive data points either all above or all below the median

(skip values on the median and continue counting data points. Values on the median DO NOT make or break a shift.)

Rule 1

25 20 15 10 5 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 Murray and Provost, 3 (11-15)

Why do we need 6 data points?

What is the probability of a coin landing heads or tails?

.5 x .5 = .5

.25

.5 x .5 x .5 = .5 x .5 x .5 x .5 = .125

.0625

.5 x .5 x .5 x .5 x .5 = .03125

.5 x .5 x .5 x .5 x .5 x .5 = .015625

Trend Rule: Five or more consecutive data points either all going up or all going down.

(If the value of two or more consecutive points is the same, ignore one of the points when counting; like values do not make or break a trend.)

Rule 2

25 Median=11 20 15 10 5 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 Murray and Provost, 3 (11-15)

Run Rule: Too many or too few runs

(A run is a series of points in a row on one side of the median. Some points fall right on the median, which makes it hard to decide which run these points belong to. So, an easy way to determine the number of runs is to count the number of times the data line crosses the median and add one. Statistically significant change signaled by too few or too many runs).

Rule 3

10 Data points not on median Data line crosses once Too few runs: total 2 runs 25 20 15 10 5 Median 11.4

0 1 2 Murray and Provost, 3 (11-15) 3 4 5 6 7 8 9 10

Run Rule Reference Table

Table for Checking for Too Many or Too Few Runs on a Run Chart Total number of data points on the run chart

that do not fall on the median

10 11 12 13 14 15 16 17 18 19 20 21 22 Lower limit for the number of runs (< than this number of runs is “too few”) 3 3 3 4 4 5 5 5 6 6 6 7 7 Upper limit for the number of runs (> than this number of runs is “too many”) 23 24 7 8 25 8

Table is based on about a 5% risk of failing the run test for random patterns of data.

Adapted from Swed, Feda S. and Eisenhart, C. (1943). “Tables for Testing Randomness of Grouping in a Sequence of Alternatives. Annals of Mathematical Statistics. Vol. XIV, pp.66 and 87, Tables II and III.

9 10 11 11 12 12 13 13 14 15 16 16 17 17 18 18

Murray and Provost, 3 (11-15)

Astronomical Data Point

(For detecting unusually large or small numbers: Data that is a Blatantly Obvious different value. Everyone studying the chart agrees that it is unusual. Remember: Every data set will have a high and a low – this does not mean the high or low are astronomical).

Rule 4

25 20 15 10 5 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 Murray and Provost, 3 (11-15)

When should we transition from using a Run Chart to a Shewhart Control Chart?

Perla, Provost and Murray