Transcript QC - ACB South Western and Wessex Region

IQC
Edward Kearney
Why is IQC/EQA important?
To ensure the test result is fit for purpose
EQA
Historic
Accuracy
Repeated samples may give precision
Who determines if a patient gets a statin?
We do !
Who determines if a patient gets a Dx of DM?
We do !
Who determines if a patient gets a prostrate
biopsy?
We do !
Who determines if a patient gets dialysis?
We do !
How do we know we are right?
Do we?
We
do !
Westgard .com “1575”
Diabetes
Fasting Plasma Glucose > 7.0 mmol/L
On two or more occasions can make the
diagnosis of Diabetes Mellitus
Fasting Glucose 6.8 mmol/L = IFG
Fasting Glucose 7.2 mmol/L = DM
Acute Coronary Syndrome
Troponin T
> 0.05 ug/L = Biochemical evidence of cardiac
damage
< 0.05 ug/L = No Biochemical evidence of
cardiac damage
How do we know we are right?
Standardisation
Calibrators
Reagents
Test conditions
Quality Control
External Quality Assurance (EQA)
Internal Quality Control (IQC)
IQC
Real time
Stable process
Bias
Precision
Statistical QC
Electronic QC
IQC Materials
Mimic Patients sample
Cover normal and pathological range
Ideal – decision points
Controlling the quality - IQC
Compare process performance with what is expected under stable
operation
Stable performance determined by assaying control material
over time : Calculate mean and SD
Measurements are made continually and compared with the original
distribution
Unexpected results are identified to alert the analyst to possible
changes in process performance
Multi-rule QC
Westgard rules
Use and Interpretation of Common Statistical Tests in Method-Comparison Studies - all 2 versions »
JO Westgard, MR Hunt - Clinical Chemistry, 1973
Performance characteristics of rules for internal quality control: probabilities for false rejection
JO Westgard, T Groth, T Aronsson, H Falk, CH de … - Clinical Chemistry, 1977
A multi-rule Shewhart chart for quality control in clinical chemistry –
JO Westgard, PL Barry, MR Hunt, T Groth - Clinical Chemistry, 1981
Assuring analytical quality through process planning and quality control.
JO Westgard - Arch Pathol Lab Med, 1992
European specifications for imprecision and inaccuracy compared with operating specifications
JO Westgard, JJ Seehafer, PL Barry - Clinical Chemistry, 1994
Internal quality control: planning and implementation strategies
JO Westgard - Annals of Clinical Biochemistry, 2003
The Quality of Laboratory Testing Today, An Assessment of σ Metrics for Analytic Quality Using Performance Data
From Proficiency Testing Surveys and the CLIA Criteria for Acceptable Performance
James O. Westgard, and Sten A. Westgard. Am J Clin Pathol 2006;125:343-354
Conceptual basis of the control chart
X
XX
XX
XX
XXX
XXXX
XXXX
X X245X X255X
235
240
250
260
•
•
•
X
265
•
•
Determine the expected distribution of control
values
Calculate mean and SD from control data to
establish control limits for control chart
Expect control values to fall with certain control
limits
– 95% within 2 SD
– 99.7% within 3 SD
Plot control values versus time to provide control
chart
Identify unexpected values
Very Unexpected
X
265
260
255
250
245
240
235
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
Somewhat Unexpected
1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 20 1 2 3 4
Run Number (or Time, Date)
James O Westgard Internal quality control: planning and implementation strategies Ann Clin Biochem 2003; 40: 593–611
Flowchart and logic for the multirule IQC procedure commonly
known as “Westgard Rules”
QC Data
Report Results
12s
13s
22s
R4s
41s
10x
Take Corrective Action
James O Westgard Internal quality control: planning and implementation strategies Ann Clin Biochem 2003; 40: 593–611
IQC
2 SD rule for 1 QC sample
Statistically 1 in 20 will be outside this range
This is not unexpected
5% of runs (false) rejected
IQC
2 SD rule for 3 QC samples
18% of runs (false) rejected
This is not unexpected
What actually happens?
• EM Kearney. The South Thames (East) Quality Assurance
Liasion Group (1998)
• James O Westgard. Internal quality control: planning and
implementation strategies Ann Clin Biochem 2003; 40: 593–
611
• David Housley. North Thames audit and QA group: Audit of
Internal Quality Control practice and processes. (2005)
5. NUMBER OF SAMPLES USED TO ASSIGN QC
VALUES
•
•
•
•
•
•
•
•
1 = provisionally 3, then re-assessed at 15
2 = 10 – 15
1 = 12
11 = 20 (one 20 days ?); one with minimum of 1
month run in (major variations discussed with
manufacturer); one re-assess after 1000; some
variation in number of batches / days etc
1 = 20 - 25
1 = > 30
3 = at least 50
1 = adjusted every month 9 ( no number stated)
David Housley. North Thames audit and QA group: Audit of Internal Quality Control practice and processes. (2005)
EP5-A
Clinical and Laboratory Standards Institute (NCCLS) EP5-A
Evaluation of precision performance of Clinical Chemistry Devices: Approved guidelines.
Precision Evaluation Experiment
Recommendations
Minimum of 20 working days
Batch analysers: two runs per day with two test samples at each of at least two levels.
Random access: four samples at each level should be analysed through out the day.
At the end of each 5 days the control limits are recalculated and all data checked for acceptability.
The cause of outliers should be determined.
Data may not be rejected without valid justification.
Learning curve data may be rejected and replaced by a equal number of points at the end.
Other methods:
Not recommended:
Single run of 20 samples for within run
Single observation each day for 10 – 20 days for day to day
Because:
Single runs do not reflect usual operating parameters, thus adversely affecting the estimate.
Single observations will be highly dependent on the number of days used.
EP5-A
Random access:
Four samples at each level should be analysed through
out the day.
At the end of each 5 days the control limits are
recalculated and all data checked for acceptability.
The cause of outliers should be determined.
Data may not be rejected without valid justification.
Learning curve data may be rejected and replaced by a
equal number of points at the end.
Criteria use to determine
acceptability
• Written procedure 1
• Professional judgement 6
• Combination of above 12
EM Kearney the South Thames (East) Quality Assurance Liasion Group (1998)
Is the procedure always
adhered to?
• Yes 2
• Mostly 1
• No 5
• Blank 11
EM Kearney the South Thames (East) Quality Assurance Liasion Group (1998)
QC rules used for calcium?
2SD
3SD
12s + Multi-rules
Westgard
6
1
1
3
Limits professional judgement, Vitros F2/F3
EM Kearney the South Thames (East) Quality Assurance Liasion Group (1998)
Calcium QC rule violated describe the
process for dealing with this
Check QC
3
Rerun
12
Recalibrate
4
Inspect QC history
1
EM Kearney the South Thames (East) Quality Assurance Liasion Group (1998)
Westgard Rules
1. Check control
1. Repeat
1. Repeat
2 Make up new
2 Make up new
2 Recalibrate
EM Kearney the South Thames (East) Quality Assurance Liasion Group (1998)
9. ARE WESTGARD RULES USED AS PRIMARY FORM
OF DATA EVALUATION?
• YES = 18 (62%)
• NO = 11 (38%)
– 1 = Mean and target range (< 2SD)
– 5 = single 2SD rule
– 1 = single 1.5 SD rule
– 1 = not formally but simplified Westgard used
– 1 = under consideration
– 1 = Instrument set flag points, but no rule stated
– 1 = 2SD in first instance, then Westgard for detail bias
profile
David Housley. North Thames audit and QA group: Audit of Internal Quality Control practice and processes. (2005)
REASONS FOR ACCEPTING FAILED QC
•
•
•
•
•
•
•
•
•
•
Only if not clinically significant – 7 sites
Lack of reagent prevents change
Lack of staff, pressure to reduce TAT, demotivated staff
Checked with EQA
Various. eg. very close to limit, one QC in/ one QC out, QC
deterioration before new QC is ready e.g. lot no. change until
new range established.
Only if all attempts to correct it have failed
Only after consultation with manufacturer
Only if calibrant returns result to normal
Only if no reason can be found and 1 QC out
eg. If LDH QC is out, the reagent is changed the next day
David Housley. North Thames audit and QA group: Audit of Internal Quality Control practice and processes. (2005)
Planning strategies (doing the right IQC)
Define the quality requirement for the test
Determine the method precision and bias
Identify candidate IQC procedures
Predict IQC performance
Select goals for IQC performance
Select an appropriate IQC procedure
A system of quality requirements and operating specifications
Medically
Important
Changes
Diagnostic
Classification
Goals
Individual
Biologic
Goals
Clinical Outcome
Criteria (DInt)
Performance Criteria
SDMax, BiasMax
State
of the
Art
Proficiency
Testing
Criteria
Total
Biologic
Goals
Analytical Outcome
Criteria (TEa)
Operating Specifications
(smeas, biasmeas, control rules, N)
Arbitrary
Control
James O Westgard Internal quality control: planning and implementation strategies Ann Clin Biochem 2003; 40: 593–611
Power curves for commonly used IQC procedures having two control measurements
per run. Pfr, or the probability for false rejection, is given by the y-intercept. Ped, the
probability for error detection, depends on the size error occurring, which is
illustrated by the vertical line, and estimated by reading the y-value at the point of
intersection with the power curve.
Power Function Graph (SE)
Probability for Rejection (P)
1.0
0.9
0.8
Ped depends on
size of
systematic error
0.7
Pfr given by
0.6y-intercept
0.5
0.4
0.3
Systematic Error (SE, multiples of s)
0.2
James O Westgard Internal quality control: planning and implementation strategies Ann Clin Biochem 2003; 40: 593–611
Example use of a power function graph to assess IQC performance for a cholesterol test where the medically
important systematic error is equivalent to 2.85 times the standard deviation of the method.
Power Function Graph (SE)
Ped is 1.00 to 0.40
1.0
Probability for Rejection (P)
0.9
0.8
Probability
0.7 of false
rejection, Pfr
given by
0.6y-intercepts
0.5
Pfr is
0.07
to
0.01
Probability of
error detection,
Ped , depends
on size of
systematic error
0.4
0.3
Systematic Error (SE, multiples of s)
0.2
James O Westgard Internal quality control: planning and implementation strategies Ann Clin Biochem 2003; 40: 593–611
Sizes of systematic errors that can be detected with 90% assurance by different IQC procedures
Power Function Graph (SE)
Probability for Rejection (P)
1.0
0.9
0.8
0.7
Lines show SE
detectable with
90% chance
by different
QC procedures
0.6
0.5
0.4
0.3
Systematic Error (SE, multiples of s)
0.2
James O Westgard
0.1Internal quality control: planning and implementation strategies Ann Clin Biochem 2003; 40: 593–611
Mathematical basis for a chart of operating specifications that shows the bias and
imprecision that are allowable for different IQC procedures, whose control rules and
Ns are given in the key at the right.
10.0
9.0
Allowable Inaccuracy (bias, %)
8.0
7.0
6.0
5.0
4.0
3.0
Allowable Imprecision (s, %)
James O Westgard Internal quality control: planning and implementation strategies Ann Clin Biochem 2003; 40: 593–611
Application of the OPSpecs tool by plotting the observed method bias as the y-coordinate and the
observed method precision as the x-coordinant to describe the “operating point” of an analytical
method.
OPSpecs Chart TEa 10% with 90%AQA(SE)
Allowable Inaccuracy (bias, %)
10.0
9.0
8.0
7.0
6.0
5.0
4.0
3.0
Allowable Imprecision (s, %)
2.0
James O Westgard Internal quality control: planning and implementation strategies Ann Clin Biochem 2003; 40: 593–611
A practical procedure for planning and selecting IQC procedures.
1. Define quality required
for diagnostic test, %TEa
5. Inspect normalized
OPSpecs charts
2. Assess method
%bias, %CV
6. Select control rules,
number of measurements
3. Calculate normalized
operating point
7. Adopt Total QC strategy
4. Plot on normalized
OPSpecs charts or EZ rules3
8. Reassess for changes
James O Westgard Internal quality control: planning and implementation strategies Ann Clin Biochem 2003; 40: 593–611
Total quality control strategies (TQC) that illustrate the proper combination of IQC
with instrument and function checks (Other QC) and need for quality improvement
(QI).
HI-Ped
Strategy
MOD-Ped
Strategy
LO-Ped
Strategy
IQC
IQC
IQC
Other QC
Other QC
Other QC
QI
QI
QI
James O Westgard Internal quality control: planning and implementation strategies Ann Clin Biochem 2003; 40: 593–611
A detailed flowchart to guide the development of TQC strategies.
Apply QP process
with OPSpecs
charts
HI-Ped (90%)
LO-Ped (<50%)
AQA or
Ped
Minimize cost of
statistical QC
Minimize cost of
non-statistical QC
Maximize
error detection
Maximize
error detection
Maximize
non-statistical QC
Improve method
performance
Maximize
non-statistical QC
Optimize QC for
process stability
Improve method
performance
Deploy skilled
analysts
MOD-Ped
(50%)
Add patient
data QC
Document TQC
system
James O Westgard Internal quality control: planning and implementation strategies Ann Clin Biochem 2003; 40: 593–611
Results of an example application of IQC design for a multitest chemistry analyzer.
Critical-Error Graph (SE)
1.0
CO2
0.9
Probability for Rejection (P)
Albumin
Chloride
OTHER TESTS
Crit SE > 4.0
Creatinine, T Bil, UA
Potassium, GGT,
ALP, Phosphorous
BUN, Cholesterol
AST, LD, Glucose
Total Protein
0.8
Calcium
0.7
0.6
0.5
0.4
0.3
Systematic Error (SE, multiples of s)
0.2
James O Westgard Internal quality control: planning and implementation strategies Ann Clin Biochem 2003; 40: 593–611
Implementation of a multistage IQC strategy.
Select
QC design
from
drop-down
list
Select
material
from
drop-down
list
Enter
control
result
Program
calculates
z-value
Program
displays
z-value
on scale
-4 to +4
Multi-chart display
shows multiple materials
and multiple designs with
multi-rule or single-rule
QC procedures
STARTUP
QC design uses
multirule
MONITOR
QC design uses
2.5s limits
Out-of-control
identified by
red bar
James O Westgard Internal quality control: planning and implementation strategies Ann Clin Biochem 2003; 40: 593–611
A Total Quality Control Support System that integrate IQC and EQA
EQA
Data Analysis
IQC Planning
And Design
Interfaces or
Manual data entry
Internal QC
Accept/Reject
EQA data transfer
Documentation
Validation Reports, Peer Reports
Toubleshooting
Laboratory
testing
processes
Inservice training
Validation tools
Internet
Support
Services
Inventory, ordering
Production, shipping
Supplies & materials
James O Westgard Internal quality control: planning and implementation strategies Ann Clin Biochem 2003; 40: 593–611
What to do?
• Do define the quality required for the test
• Do select QC procedures that minimise false rejections
• Do select QC procedures that detect medically important errors
• Do adopt modern QC planning tools and techniques
• Do standardise QC operations
• Do calculate control limits from your own laboratory data
• Do provide computer support to analyse and interpret QC data
• Do reject out-of-control runs, identify the problem, and
eliminate the cause
• Do adopt a Total QC strategy to maximise the costeffectiveness of QC
• Do calculate daily/weekly/monthly patient means
What not to do?
• Don’t use 2SD control limits
• Don’t just repeat the controls
• Don’t use the same control rules for all tests
• Don’t use bottle values to calculate control limits
• Don’t use medical decision limits as control limits
• Don’t rely on electronic QC alone
• Don’t eliminate SQC in POC applications
Additional methods of IQC
Average of Normals (AoN, Daily patient means)
Standard signals
Experience
Expertise
IQM
Monitors system performance real time
Automatically detects potential failures that affect
the analytical performance
Automatically performs corrective action
Automatically documents failures and actions taken
Provides quality control reports
IL GEM Premier 4000