Transcript THE EFFECT OF EXPANDED STANDARD DEVIATION ON WESTGARD

THE EFFECT OF SETTING QC
LIMITS GREATER THAN
ACTUAL SD ON WESTGARD
MULTI-RULES
Graham Jones
Department of Chemical Pathology,
St Vincent’s Hospital, Sydney
Background
• Westgard rules are commonly used to detect changes
in assay performances.
• The Power of Error Detection of rules can be
determined from available sources1.
• These sources assume that the SD set in the QC
protocol is the actual SD of the measurement system.
• Sometimes the SD in QC charts is set wider than the
actual instrument SD (figure 1).
• Here I investigate the effect of setting QC limits
wider than the actual SDs on the power of error
detection.
1 eg www.westgard.com
QC Limits
3
2
1
0
-1
-2
QC SD = Actual SD
•Spread of QC results across
Range.
•5% of results outside +/- 2SD
-3
3
2
1
0
-1
QC SD = 3 x Actual SD
•QC results clustered near
mean.
•No results outside +/- 2SD
-2
-3
Figure 1
Hypothesis
• Setting the SD limits in the Levy-Jennings QC
chart different to the actual instrument SD limits
will change the performance of the QC protocol.
Aim
• To investigate the utility of published Power
Function Charts to determine the Power of Error
Detection when the QCSD is larger than the Actual
SD of the method.
Setting QC Limits - illustration
A
QC SD = Actual SD
•Spread of QC results across range.
•5% of results outside +/- 2SD
+2
0
-2
+2
0
-2
B
QC SD = 3 x Actual SD
•Same ASD as graph A.
•QC results clustered near mean.
•No results outside +/- 2SD
Figure 1
Terminology
• ASD - The actual SD of the measurement system
• QCSD - The “SD” Limits set in the QC package or
drawn on Levy-Jennings chart.
• Power of error detection: The likelihood that a QC
rule will trigger for any given assay drift.
• 13s - 1 result outside 3 SD
• 12s - 1 result outside 2 SD
• 22s - 2 results outside 2 SD
• 41s - 4 consecutive results outside 1 SD
• 10x - 10 consecutive results on the same side of the
mean
Methodology
Power Function Charts were developed in Microsoft Excel.
– Random variation was simulated using the random number
generator with a normal distribution (n=1000).
– Bias was simulated by adding fixed amounts to the
randomly generated numbers.
– All rules were modelled on n=2 (2 levels of QC).
– Only change in bias (not change in precision) was
modelled (thus the R4s rule was not tested).
– the 41s and 10x rules were run on data from both materials.
– The 22s rule was run within the pair of simultaneous QCs
1
0.9
0.8
0.7
10x
4.1s
2.2s
1.3s
0.6
0.5
0.4
0.3
0.2
QCSD = ASD
Error Detection
0.1
0
1 0
0.9
1
2
3
4
5
6
0.8
0.7
10x
4.1s
2.2s
1.3s
0.6
0.5
0.4
0.3
Power Function
Charts
Effect of increasing
QCSD relative to ASD on
power of error detection
of individual rules.
QCSD = 1.5 x ASD
0.2
0.1
0
10
1
2
3
4
5
6
0.9
0.8
0.7
10x
4.1s
2.2s
1.3s
0.6
0.5
0.4
0.3
90% error detection for 10x
90% error detection for 41s
90% error detection for 22s
QCSD = 2 x ASD
0.2
0.1
0
0
1
2
3
4
5
Shift (multiples of SD)
6
Figure 2
Shift at 90% ED (xSD)
Effect of Increasing QCSD relative
to ASD
9
8
7
6
10x
4.1s
2.2s
1.3s
5
4
3
2
1
0
1
1.5
2
2.5
QCSD / ASD
Notes:
Lines are not straight due to non-smoothed data in Power Function Charts.
This graph shows the changes in error detection compared to the ASD.
Figure 3
Shift at 90% ED (xSD)
Effect of Decreasing ASD relative to
QCSD
4
3.5
3
10x
4.1s
2.2s
1.3s
2.5
2
1.5
1
0.5
0
1
1.5
2
2.5
QCSD / ASD
Notes:
Lines are not straight due to non-smoothed data in Power Function Charts.
This graph shows the changes in error detection compared to the QCSD.
Figure 4
Results and Discussion 1
• Increasing the QCSD relative to the ASD changes
the Power Function Charts for individual rules
(figure 2).
• Increasing the QCSD has different effects on the
different QC rules (figures 2 and 3).
– The error detection line for 10x does not change with
different settings of QCSD.
– The error detection lines for 41s, 22s and 13s all move
to the right, indicating reduced error detection.
• Thus the main effect is on the rules which offer
earliest error detection (the within-run rules).
Results and Discussion 2
• The effects on error detection have been expressed
relative to the actual instrument SD thus far.
• The effect of improving ASD relative to a set
QCSD is shown in figure 4.
• It can be seen that the power of error detection of
all rules change (improve) as the ASD improves.
• This demonstrates that published power function
charts cannot be used to predict error detection
when QCSD is not equal to ASD, even by using the
QCSD for the prediction.
Conclusions
• Setting the QCSD different to the ASD in QC
protocols affects the performance of Westgard
rules.
• If QCSD does not equal to ASD the published
Power Function Charts do not accurately represent
the power of error detection.
• Laboratories wishing to determine the power of
error detection using published data must set
QCSD = ASD, or develop alternate methods to
determine their own error detection power.