Transcript Westgard Multi-rules across Quality Control runs

AACB ASM 2003
THE POWER OF ERROR
DETECTION OF
WESTGARD MULTI-RULES:
A RE-EVALUATION
Graham Jones
Department of Chemical Pathology
St Vincent’s Hospital, Sydney
AACB ASM 2003
Background
• Westgard multi-rules are claimed to increase the
power of error detection of laboratory QC procedures.
• Power Function Charts can quantify the ability of
these rules to detect changes in assay performance.
• Examples of Power Function Charts are available on
the Westgard QC website (www.westgard.com)
• In this poster I re-evaluate the power of error
detection of QC rules which require data from more
than one QC run (Multi-run rules).
AACB ASM 2003
Hypothesis
• That the correct model for assessing the
Power of Error Detection for multi-run QC
rules should only show benefit for these
rules if the error has not already been
detected in QC runs required to gather data
for those rules.
• This hypothesis was modelled and
compared to data on the Westgard website.
AACB ASM 2003
Nomenclature
• Single-run rules
– All data is contained in a single QC run
• For n=2 includes 13s and 22s
• For n=4 includes 13s and 22s and 41s
• Multi-Run Rules
– Requires data from more than one QC run
• For n=2 includes 41s and 10x
• For n=4 includes 8x
AACB ASM 2003
Methods
• Power Function Charts were produced using a
Microsoft Excel spreadsheet.
• QC results were simulated using a random number
generator with a normal distribution.
• Changes in bias were modelled by adding various
constants to the output.
• QC rules were evaluated by the frequency with
which they were triggered at changes in bias.
• Westgard multi-rules with n=2 were evaluated for
bias detection: 13s/22s/41s/10x.
• Changes in random error were not modelled.
AACB ASM 2003
Hypothesis - Graphical Display
This display uses 10x as an
example of a multi-run rule
+3SD
+2SD
Mean
-2SD
-3SD
1
2
3
4
5
Change in assay bias
QC run - within-run rules evaluate performance (13s/22s)
QC run - within-run rules evaluate performance (13s/22s)
- multi-run rule evaluates performance (10x across both materials)
- Only adds benefit if shift NOT detected by QC events 1-4
AACB ASM 2003
A
Results
Probability for Rejection
Graph A
- Original data from Westgard
website
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
B
1.3s/2.2s/4.1s/10x
1.3s/2.2s/4.1s
1.3s/2.2s
1.3s
Graph B
- Model of data from Westgard website.
- Multi-run rules fire even if shift would
0.0
1.0
2.0
3.0
4.0
have been detected previously.
Shift in Bias (multiples of SD)
Probability for Rejection
1.3s/2.2s/4.1s/10x
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
AACB ASM 2003
1.3s/2.2s/4.1s
C
1.3s/2.2s
1.3s
0.0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
1.0
2.0
3.0
4.0
10x
4.1s
2.2s
1.3s
D
0.0
Graph C
- Westgard data adjusted for hypothesis.
- Multi-run Rules fire only if shift would
NOT have been detected previously.
1.0
2.0
3.0
4.0
Graph D
- Model of individual rules from Graph C
- Multi-run Rules fire only if shift would
NOT have been detected previously.
Shift in Bias (multiples of SD)
Shifts detected with 90% certainty from full multi-rules
Shifts detected with 90% certainty from within-run rules.
AACB ASM 2003
Results
• A power Function Chart from the Westgard website
showing multi-rules for bias detection with n=2 is
shown in graph A.
• The change in bias which Westgard claims full Multirules can detect with 90% certainty is about 2.0 times
the SD of the assay (Graph A).
• My model of the Westgard data, with multi-run rules
triggered even if the change in bias would been
previously detected, agrees well with the website data
(Graph B).
• In the Westgard model the multi-run rules (10x and 41s)
enhance the error detection over the within-run rules
(graphs A and B)
AACB ASM 2003
• The model excluding multi-run rules when a shift
would have been detected previously is shown in
Graph C.
• When these previously-detected shifts are
removed from the data, the assay bias which can
be detected with 90% certainty is reduced to about
3.3 times the assay SD (Graph C).
• With this model the multi-run rules do not add to
the within-run rules for confident error detection.
• When the individual rules are plotted it can been
seen that the multi-run rules never add to the error
detection with 90% certainty.
• The multi-run rules can be considered warning
rules.
AACB ASM 2003
Conclusion
• The multi-run rules, as described on the
Westgard website, give a falsely low
estimate of the change in bias which can be
detected with 90% certainty.
• The 10x and 41s rules add little to the overall
error detection at the 90% confidence level
with 2 QC samples per run.
• Multi-run rules are similarly noncontributory with 4 QC samples per run
(data not shown).