Transcript Wisconsin AST Training 2012- Presntation_T.Novicki, 2012

Validation, Verification
and Quality Control
Thomas Novicki Ph.D. D(ABMM)
Marshfield Clinic
Marshfield WI
Today’s Topics
1.
Validation and Verification of ID and AST
systems
2.
Quality Control (QC) of Identification (ID)
and Antimicrobial susceptibility test (AST)
systems
(We will not be discussing proficiency testing
due to time constraints)
2
Disclosures

I have nothing relevant to this
presentation to disclose
3
VALIDATION AND
VERIFICATION
4
Validation and Verification

Validation: Documenting in vitro
diagnostic device performance in your lab

Verification: An ongoing demonstration of
the vendor’s performance claims using a
variety of tools
5
Validation and Verification
(or is it Verification and Validation…?)

Note that some entities (e.g. CLSI, CLIA)
reverse the definitions


Initially Verify
Then periodically Validate
The definitions used don’t matter, but their
consistent use in your lab is important.
What does your accrediting body use?
6
Who Guides Us?



Vendors of commercial systems
The Clinical and Laboratory Standards
Institute (CLSI)
The Feds



CLIA – Clinical labs
FDA – Instruments
Your accrediting body (COLA, CAP, JC)
(Did I miss anything…?)
7
Validation and Verification
The Centers for Medicare and Medicaid Services
(CMS), (a part of the US Department of Health and
Human Services) regulates clinical laboratory testing
through the Clinical Laboratory Improvements
Amendments (CLIA) that became effective in 2003.
(Recall that it all began in 1988)
8
Validation

For FDA cleared and approved tests, the lab
demonstrates performance claims by the
device manufacturer, typically found in the
product label (‘package insert’)

For lab-developed tests (LDTs), a more
rigorous set of studies demonstrating
performance is necessary
9
Validation of LDTs

Will not discuss further except to say that any
modification of an FDA IVD makes it an LDT

Be wary of modifying the more critical FDAapproved test (e.g. TB, HIV IVD devices )

Qualitative tests easier to modify than quantitative

Simple cleared tests (e.g. MRSA chromo-agars for
additional anatomical sites) are reasonably modified
10
iClick? 1

CLIA requires a lab to validate which of
the following for every new diagnostic test
1.
2.
3.
4.
5.
Accuracy
Precision
Reportable and Reference ranges
All of the above
1 and 2
11
iClick? 1 - Discussion

CLIA specifies that the user of an FDA IVD device of
Moderate/High complexity will




Validate accuracy,
Precision,
Reportable range, and
Reference range

You must satisfactorily complete validation of the device
before beginning its use.

You must document all validation study data and maintain it
for at least 2 years after test retirement.
12
Validation – AST

Two methods for an unmodified FDA-cleared
instrument
A.
B.

Compare to a reference method
Compare to an existing system
Method B probably the best fit for most labs with
an existing system For method A, see Cumitech
31A
(Cumitech 31A, Verification and validation of procedures in the clinical microbiology laboratory,
ASM Press 2009)
13
Disclaimer
What I’m about to say on validation…
‘… is more what you’d call guidelines than actual
rules’
14
Validation – AST

Source of samples/isolates to test

> 30 well-characterized clinical isolates per panel that
reflects the mix common to your population and also
some with unusual AST patterns

Can include proficiency or commercial validation panels,
but be wary:




Reflective of your population?
Incorrect sample matrix?
If from the device manufacturer, a self-fulfilling prophecy?
Note that with care a single panel can be
used for both AST and ID validations
15
Validation – AST
How to deal with discrepancies when two
non-reference systems are compared?
 No reference method, so grade results, then
apply predetermined limits on the number and
types of discrepancies
16
Validation – AST
For each bug/drug result, calculate %s for…

Categorical (S/I/R) Errors




Minor Error I vs S or I vs R
Major Error S vs R or R vs S
No Very Major Error category: ‘True’ result is not known
Essential Errors (> + 1 MIC dil. difference)

Can only score when both methods generate discrete
numbers (i.e. do not score < or > values)
17
Validation – AST
Using the total number of evaluable bug/drug data
points, an acceptable validation has:




< 5% Major Errors AND
< 10% Major + Minor Errors AND
< 10% Categorical Errors AND
< 10% Essential Errors
Also look for trends in a particular drug or bug which may
point towards a problem area
Consult your vendor rep with any validation problems
18
Validation – AST

Precision (reproducibility)

Test five isolates X3 for three to five days

Calculate intra-and inter-run categorical and essential %
agreements

Isolates should have known susceptibility profiles, such
as ATCC strains

Use both Gram positive and Gram negative strains if
validating GP and GN panels

Acceptable precision > 95%agreement
19
Validation – ID
Test a range of Gram positive and Gram
negative organisms ID’d by reference or current
method(s)
Minimum of 20 patient isolates; larger labs test
more. Additionally, test QC strains

< 10% discrepancies between reference and
new IDs
20
Validation – ID

Level of agreement may need to be adjusted
e.g.



Old ID of ‘Coag-negative Staphylococcus sp’
New instrument ID’s to species level
Precision



Test 2-4 isolates X3 for three to five days
Calculate % intra- and inter-run agreements
Accept > 95% agreements
21
Validation

What to do if validation fails? Do not use
the new test and either…

Withdraw test from further consideration OR

Develop and implement a corrective plan
with the manufacturer. Then, re-validate
22
Verification – Component of the QA Process
An ongoing, multi-faceted process that assures the
validated test continues to perform at the expected level
Personnel competency
Proficiency challenges
QC organisms or analytes
Comparison with other labs
Instrument PM/calibration
Trend analysis
Discrepant resolution
Pre- and post-analytic controls
Written procedures documenting the process
23
Quality Control
I think, therefore I control quality
24
iClick? 2
You have an instrument that uses micro-broth
dilution MIC plates in your lab. This system
incubates and reads plates automatically but also
has a manual plate reader. Your tech rep tells you to
read a plate manually if QC fails on the instrument. If
QC then passes, your QC is acceptable for the entire
system and no follow-up action is needed.
1.
2.
True
False
25
iClick? 2: discussion
QC is performed to assure the accuracy of a
test result by assessing all analytical
components





Reagents
Instrument (where applicable)
Test precision (i.e. repeatability)
Operator technique
Data interpretation
26
CAP on AST QC
‘For AST of either disk or dilution type,
control organisms are tested with each new
lot or batch of antimicrobials or media and
each day the test is performed thereafter.’
(Note that >3 failures per month per
drug/bug combo is unacceptable)
27
CAP on AST QC
‘However, the frequency of test monitoring
may be reduced to weekly …if the laboratory
can document satisfactory performance with
daily control tests as suggested by CLSI
guidelines.’
The CAP checklist then goes on to briefly
describe how to do so
28
AST QC Resources
COLA, JC have similar verbiage, but is the
checklist guidance enough?
IMHO*, No!
You need CLSI AST documents in your lab.
(*In my humble opinion.)
29
CLSI AST Documents

M2 (disk diffusion) and M7 (MIC), 3 year cycle

Test performance
 QC (including daily-to-weekly QC) 

M100, annual cycle

Recommended drugs to test and report, interpretive breakpoints
 QC strains, QC troubleshooting, when to re-validate QC 



Others documents for other classes of microorganisms
(Explore these and more at http://www.clsi.org/; choose Microbiology from the Shop drop down
menu)
30
iClick? 3
How many of you have these CLSI
documents in your lab:
1.
2.
3.
4.
5.
6.
2012 M2 (disk diffusion), M7 (MIC) and
M100 (Breakpoints, QC)
2012 M7 and M100
2012 M2 and M100
2012 M2 and M7
Any older versions of these documents
None of the above
31
Conversion to Weekly AST QC
When day-of-use QC is being successfully
performed at least once a week, a lab may
consider weekly AST QC by a validation
process:

Test QC strains 20 or 30 consecutive days

If 0-1 failures per drug-bug combo in 20 days
OR 2-3 failures in 30 days

THEN Weekly QC may be implemented
32
Weekly AST QC


Perform QC testing once per week and
with any new reagent
If weekly AST QC fails

If readily identifiable error (e.g. wrong QC
strain; see M2 or M7 for more examples)
correct error and retest once. If QC passes,
document findings and continue weekly QC

If no error is found…
33
Weekly AST QC Re-validation
Perform 5 consecutive days of QC on the failed drug-bug combo
(We automatically take out a fresh QC strain from the freezer at this point)

If ALL 5 days pass, document and resume weekly testing

If a failure occurs again, STOP TESTING and thoroughly evaluate the
process for error
New QC strain
New lot of reagent
Correct process errors
Consult the manufacturer
34
Weekly AST QC Re-validation

Once a likely error has found, re-validate


Do not include data from the prior 5-day sequence
Must again pass these criteria



0-1 failures per drug-bug combo/20 days OR
2-3 failures per drug-bug combo/30 days
Remember that, during re-validation the lab is
on a daily QC schedule-a failure means that
that drug’s results are not released
35
AST QC References

Disk diffusion



M02-A11
M100-S22
Section 15
Tables 3C, 3D
MIC


M07-A9
M100-S22
Section 16
Tables 4F, 4G
36
AST QC
Special Considerations
M100-S22 Tables 3C & 4F, ‘Reference
Guides to QC Frequency’

When and how much re-validation to perform
when a system change occurs (e.g. AST
instrument repair, Abx formula change & Etc)
Release of patient results when QC fails?

CLSI says OK with many qualifiers, but our
lab generally just says No
37
iClick? 4
You are performing a weekly AST QC re-validation for
drug X on your favorite automated ID/AST instrument
when you incur a failure on the same AST panel for drug
Y. What do you do?
1.
Ignore the failure of drug Y and press on with the 5-day
testing of drug X
2.
Now treat the failure of drug Y as a second failure and
start a 5 day test of drug Y
3.
Stop testing, evaluate the whole process, then perform a
new 20/30 day validation on the whole AST panel
38
iClick? 4 discussion
The correct answer isn’t clear, but Marshfield
Labs treats the failure of the second drug as a
failure and starts a 5 day test on that drug also
Why?  Subsequent failures may reflect a
larger problem
39
ID QC
CLIA requires a lab to show positive and
negative reactions for each new lot and/or
shipment of biochemical.
 This includes kit and automated
instrument ID methods!
40
RapID ANA II Identification System Quality Control
Each new lot number and/or shipment is to be tested with the following
organisms to provide at least one positive and one negative reaction
for each biochemical within the ID system.
Lot#/Exp Date ________________________________
Set-up Date/Time/Init. __________________________
Read Date/Time/Init. ___________________________
Results Acceptable? ___________________________
Expected Results:
Before Reagent Addition
Organism
Clostridium
sordellii
E
ATCC 9714
A
Bacteroides
distasonis
E
ATCC 8503
A
Bacteroides
uniformis
E
ATCC 8492
A
After reagent addition
URE
BLTS
ARA
ONPG
GLU
GLU
GAL
FUC
NAG
PO4
LGY
+
-
-
-
-
-
-
V
-
-
-
+
+
+
+
+
-
+
-
+
+
+
+
+
+
+
V
+
GLY
PRO
PAL
ARG
-
V
+
V
V
+
+
+
-
+
+
+
+
-
-
-
-
SER
PYR
IND
V
+
+
+
-
-
-
+
V
+ positive, - negative, V variable, (-) usually negative, (+) usually positive
E = Expected Reactions
A = Actual R
41
ID QC
There is a QC solution for automated ID
instrument users:
Streamlined QC
Stay Tuned for the Next Talk!
42
The End
Thank you…
Questions?
43