Cumulative Sum (CUSUM) charts for medical student peripheral venous cannulation; development of a difficulty-adjusted CUSUM

Dr Harry Murgatroyd SpR Anaesthesia Leeds Teaching Hospitals Trust Sumaiyah Kola Medical Student Leeds University Medical School

QuickTi me™ a nd a TIFF (Uncompre ssed ) decomp resso r are need ed to se e th is p icture.

QuickTime™ and a TIFF (Uncompressed) decompressor are needed to see this picture.

QuickTime™ and a TIFF (Uncompressed) decompressor are needed to see this picture.

Runcie CJ. Assessing the performance of a consultant anaesthetist by control chart methodology. Anaesthesia. 2009; 64(3): 293-296

Developed initially to look at industrial processes

CUSUM Chart

Learning Curves Monitors performance Used to determine competency Graphical presentation over time Audit of quality clinical practice

Theory

Set: definition of success / failure acceptable failure rate unacceptable failure rate error Collect: binary data Algorithm: Score falls with success Score increases with failure Graph Boundary Lines QuickTime™ and a TIFF (Uncompressed) decompressor are needed to see this picture.

-10 -12

CUSUM Sequential cannulation attempts

2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 -2

Success is seen as a fall in the graph

-4 -6

Failure seen as a rise in the graph

-8 Series1

Each point represents a single cannulation attempt Score derived using the CUSUM formula

Problems

•Patient variability •Standard CUSUM •Constant failure and success rates •Risk adjusted CUSUM •Complicated •Not intuitive •Failure rates •Set by user •Can affect results considerably

-10 -12 -4 -6 -8

Comparing acceptable failure rate

6 4 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 -2 Acceptable failure rate 0.1

Acceptable failure rate 0.2

Acceptable failure rate 0.4

Medical student project

• Aims – Proficiency at intravenous cannulation – Plot individual CUSUM charts – Develop a ‘difficulty-adjusted’ CUSUM technique • Time Scale: – 5 weeks

How to insert an intravenous cannula 101

Methodology

• • Setting – Teaching hospital – Elective surgical lists Procedures – Verbal consent – Peripheral venous cannulation • Standard technique • Size of cannula appropriate to surgical procedure – Data collection • Success or failure • Appearance of vein • Size of cannula • Patient awake or anaesthetised

Conventional CUSUM

• Definition of ‘success and failure’ • Acceptable and unacceptable failure rates – Consultant consensus – Literature – 0.2 and 0.4 respectively de Oliveira. Anesth Analg 2002;95:411-6.

• Calculation – Published formulae – Error rates = 0.1

– MS Excel Williams et al. BMJ 1992;304:1359-61.

Conventional CUSUM

Failure rates Upper and lower boundries 0= failure,1= success Data is plotted sequentially Running total, CUSUM Example if “IF” formula in Excel

Difficulty Adjusted CUSUM

• Difficulty score – Appearance of vein – Cannula size – Awake or anaesthetised • Different failure rates – Two stages • Vein adjusted • All three variables – Intervention line • Average of all prior lines

Difficulty Adjusted CUSUM Vein adjusted

Vein appearance Acceptable failure rate Unacceptable failure rate Visible, palpable Just visible 0.15

0.3

0.3

0.6

Three variable methodology

Table shows the scoring of each of the recorded variables. These are then added up to give the total score for the cannulation attempt Shows the standard CUSUM formula, whilst incorporating different failure rates and scores dependent on the difficulty of the variables recorded.

Spreadsheet showing the final added up scores of the different variables. Hence including the vein, consciousness and cannula size. Using “IF” formulas the correct value of S is selected from the table above and the CUSUM then plotted in the same way as before.

Difficulty Adjusted CUSUM

• Successful difficult cannulation

– Large fall in score

• Failed difficult cannulation

– Small rise in score

• Successful easy cannulation

– Small fall in score

• Failed easy cannulation

– Large rise in score

-3 -4 -5 2 1 0 0 -1 -2 3

Conventional CUSUM vs Vein DA-CUSUM

An example CUSUM and difficulty-adjusted CUSUM chart for student A

10 20 30 Conventional CUSUM DA-CUSUM DA-CUSUM intervention line Conventional CUSUM intervention line 40

Patient number

50 60

Conventional CUSUM vs Three Variable DA-CUSUM

-3 -4 -5 3 2 Standard Including Difficulty 1 0 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 -1 -2

Three variable DA-CUSUM for two students

2.5

2 1.5

1 0.5

0 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 -0.5

-1 -1.5

-2 Sumi Dave

POSITIVES of CUSUM

•Objective •Simple Calculations •Shows improvement in learners •Early detection of poor performance •Allows comparisons between students Bolson S, Colon M. Int J Health Care Qual Assur 2000;12:433-438. Kestin IG. BJA 1995;75:805-809.

LIMITATIONS of CUSUM

•Only technical skills •Must have binary outcome •Relies on logbooks and honesty of user •Time consuming •Open to manipulation •Does not show improvements that do not change binary outcome

Difficulty adjusted CUSUM

•

POSITIVES

• Potentially corrects for patient variability • Easier and more intuitive than other methods of adjustment •

LIMITATIONS

• Failure rates set by the user • The more variables ‘corrected’ for, the more layers of estimation and inaccuracy • Loss of statistical element of conventional CUSUM

Summary

• Easy technique – Handheld devices – Electronic logbooks • Objective • Can be adjusted for patient variability • Allows – Charting of ‘learning curve’ – Comparison between practitioners – Identification of poor performance