Transcript Method Validation Studies

r
Method Validation Studies
It’s a Jungle out There!
r
Well Duh...
r
Purposes of Method Validation
Studies:
• To quantifiably characterize system performance
• To assess potential for error
• To identify method-to-method differences
• To meet regulatory guidelines
r
When to Do Method Validation
Studies:
• When considering purchasing a new system
• When placing a new system into service
• At regular intervals to assess on-going system
performance
• When troubleshooting questionable system performance
Basic Components of Method
Validation Studies:
Any or All of the Following...
• Precision Studies
• Accuracy Studies
• Range Validation/Verification Studies
• Reportable Range
• Critical Limits
• Analytical Measurement Range
• Clinically Reportable Range
• Calibration Verification / Linearity
• Reference Range Studies
r
How to Do Method Validation
Studies:
Don’t wing it!!! Consult appropriate
reference guides.
• NCCLS evaluation document series
• Manufacturer’s package inserts, evaluation protocol
manuals and supplemental protocols.
r
r
Precision Studies
• Characterize the reproducibility of a test system
• Performed by repetitive testing of the same sample
• Intra-run
• Inter-run
• Inter-instrument
• Statistical parameters
• Mean
• Standard Deviation (SD), mg/dL
• Coefficient of Variation (CV), %
• Acceptability Expectations…the lower the better
r
Accuracy Studies
• Method-to-method studies comparing a POC
instrument to a properly calibrated primary Laboratory
analyzer.
• For the purpose of assessing accuracy, the same
samples should be tested on both the Laboratory
analyzer and the Point-of-Care instrument.
Accuracy Studies, cont’d:
r
• A typical Accuracy Study includes:
• all, or a representative sub-set, of Point-of-Care
instruments
• optimally at least 40 different samples (minimum of
20 for statistical validity)
• good distribution of sample values ranging from low
to high
• same samples tested on Point-of-Care instruments
and Laboratory analyzer. If capillary samples are
compared to venous glucose results, use only
samples from fasting patients (Capillary glucose
levels can be 20-70 mg/dL higher than venous
levels at the same time in the non-fasting state).
Accuracy Studies, cont’d
• Statistical analysis of accuracy study
results typically includes:
• Regression analysis
• Bias analysis
r
r
Accuracy Studies, Cont’d
Raw Data Table
Lab
Glucose
mg/dL
19
29
34
35
37
37
38
45
47
49
55
55
59
62
74
80
82
87
105
Accu-Chek
Glucose,
mg/dL
12
25
30
36
38
30
40
42
52
45
55
55
57
57
78
81
85
83
112
Lab
Glucose
mg/dL
108
146
189
200
217
232
237
254
266
302
321
325
330
355
365
403
405
458
552
Accu-Chek
Glucose,
mg/dL
118
158
184
184
189
222
214
244
243
283
296
309
292
341
344
342
387
484
551
Regression Analysis
Plotting the Comparative Data...
Accu-Chek Vs Lab Regression Analysis
(Y - Axis) Accu-Chek Glucose, mg/dL
500
450
400
350
300
250
200
150
100
50
0
0
50
100 150 200 250 300 350 400 450 500
(X - Axis) Lab Glucose, mg/dL
r
Regression Analysis
Plotting the Comparative Data...
Accu-Chek Vs Lab Regression Analysis
(Y - Axis) Accu-Chek Glucose, mg/dL
500
450
400
350
300
250
200
150
100
50
0
0
50
100 150 200 250 300 350 400 450 500
(X - Axis) Lab Glucose, mg/dL
r
Regression Analysis
Plotting the Comparative Data...
Accu-Chek Vs Lab Regression Analysis
(Y - Axis) Accu-Chek Glucose, mg/dL
500
450
400
350
300
250
200
150
100
50
0
0
50
100 150 200 250 300 350 400 450 500
(X - Axis) Lab Glucose, mg/dL
r
Accuracy Studies, cont’d
r
• Regression Statistics Review:
• Correlation Coefficient (r) - characterizes the
dispersion of results around the line of best fit.
• Slope - The “lean” of the line of best fit (proportional
bias)
• Y-Intercept - the point at which the line of best fit
intersects the Y axis. (constant bias)
• Acceptability Criteria:
• Correlation Coefficient (r) - the closer to 1.0 the
better
• Slope - The closer to 1.0 the better
• Y-Intercept - the closer to zero the better
r
Accuracy Studies, Cont’d
The Equation of the Line of Best
Fit:
y = mx + b
m = Slope
b = Y Intercept
Accuracy Studies, Cont’d
r
The Equation of the Line of Best Fit
y = 0.991x + 8.4
Slope
Y-intercept
Accuracy Studies, Cont’d
The Equation of the Line of Best
Fit
y = 0.991x + 8.4
Slope
Y-intercept
r
Accuracy Studies, cont’d:
r
• Bias Analysis:
• Calculate mg/dL or % difference between Point-ofCare instrument and Laboratory analyzer for each
set of paired data-points.
• Acceptability Criteria (Glucose):
• 95% of meter results within +/- 15 mg/dL of
Laboratory analyzer results below 75 mg/dL or
within +/-20% of Laboratory analyzer results equal
to or greater than 75 mg/dL (NCCLS document
C30A-2)
r
Bias Analysis, Cont’d
Bias Chart
Accu-Chek vs Lab Bias Chart
50
45
40
35
30
25
20
Accu-Chek Bias*
15
10
5
0
-5 0
50
100
150
200
250
300
-10
-15
-20
-25
-30
-35
-40
*Bias is calculated as mg/dL difference from lab results
<75 mg/dL and as % difference from lab results >75 mg/dL.
-45
-50
Lab Glucose, mg/dL
350
400
450
500
r
Range Validation Studies
• Definitions
• Reportable Range - The range of results for which a
system has been proven to yield numerically accurate
results
• Analytical Measurement Range (AMR) - The range of
results through which a method yields numerical
values
• Clinically reportable range - (CRR) An extension of the
AMR supplemented by dilution or concentration
protocols
• Calibration Verification - The process of verifying that
a system is properly calibrated
• Critical Limits - Low and high result limits which, when
exceeded, require follow up action
r
Range Validation Studies
• Definitions
• Analytical Measurement Range (AMR) - The range of
results through which a method yields numerical
values
• Clinically reportable range - (CRR) An extension
of the AMR supplemented by dilution or
concentration protocols
• Calibration Verification - The process of verifying that
a system is properly calibrated
• Reportable Range - The range of results for which a
system has been proven to yield numerically accurate
results
• Critical Limits - Low and high result limits which, when
exceeded, require follow up action
Range Validation Studies, Cont’d
r
• Validation of Reportable Range,
Analytical Measurement Range,
Reference Range, Critical Limits and
Calibration Verification (CAP POC July 2003
Checklist, Page 34 Commentary)
• Can all be accomplished through the same protocol
if the range of testing spans the broadest range
(AMR)
• Materials used must be “matrix appropriate” such
as:
• Split blood samples
• linearity/calibration verification materials
supplied by manufacturer or other vendor
r
CAP POC.08700 -
If the laboratory has more than
one method-system for performing tests for a given
analyte, are they checked against each other at least
twice a year for correlation of patient results?
NOTE: This includes same or different instrument
makes/models. This comparison must include all
instruments. If multiple methods are used for a particular
analyte, comparing a subset of each method to one another is
acceptable only after all like instruments have been compared to
one another to ensure correlation and establish that a
representative subset is used in the method-to-method
correlation.
CAP POC.08700, Cont’d: Significant
changes to the commentary!!!
r
•
“POCT glucose devices are a special case in which a large number
of devices may be in use in an institution. In this case alternate
approaches are acceptable for documenting correlation of patient
results among the devices and with the main laboratory. One
approach is to document for a lot of reagent strips the agreement
between patient results for a laboratory method and representative
POCT device(s) and simultaneously collect results for QC materials
run on the POCT devices. “
•
“Other correlation schemes may be suitable. The critical factor is to
have a mechanism that documents the agreement between patient
results performed by the POCT glucose devices and the main
laboratory.”
r
Where to Find the Method Validation
Regulations
CLIA
CAP
JCAHO
Regulatory Guidelines for Method
Validation:
• CLIA 88 Final Rule - (January 24, 2003)
• CAP General Checklist (July 2003)
• CAP Point-of-Care Checklist (July 2003)
• JCAHO - WT and QC Section (2004 Pre-Publication
with Crosswalks from 2003 Guidelines)
• NCCLS - C30A-2 - (Item 6.3.2).
r
r
CLIA Guidelines for Method Validation:
• CLIA 88 Final Rule - Subpart K (Quality Systems for
Non-waived Tests) 493.1253
• Revised January 24, 2003
• Outlines requirements for unmodified “non-waived”,
FDA cleared tests
• By definition, waived tests are not subject to items
in Subpart K
CAP Guidelines for Method Validation:
r
• CAP Laboratory General Checklist
• Revised July 2003 (GEN.42020 - GEN.42163)
• Does not distinguish based on CLIA designated test
complexity
• Requires validation of precision (intra- and inter-run)
and accuracy
• Requires verification of analytic sensitivity (Low end)
• Requires assessment of analytical interference
• Requires verification of reportable range (AMR)
• Requires verification of reference range
CAP Guidelines for Method Validation:
r
• CAP Laboratory Point-of-Care Checklist
• Revised July 2003 (POC.08200 - POC.08700)
• Does not distinguish based on CLIA designated test
complexity
• Requires calibration verification
• Requires validation of Analytical Measurement
Range
• Requires method-to-method and instrument-toinstrument correlation
JCAHO Guidelines for Method
Validation:
r
• WT Section (2004 Pre-Publication with Crosswalk from
2003 Guidelines)
• Item WT.1.40 - Non-specific reference to requiring
“Equipment performance evaluation”
• QC Section (2004 Pre-Publication with Crosswalk from
2003 Guidelines)
• Items QC.1.70 - Method Validation requirements for
precision, accuracy and reportable range apply to
FDA cleared, unmodified moderate complexity
tests.
• Requires reference range verification for all new
methods
JCAHO Guidelines for Method
Validation:
r
• QC Section (2004 Pre-Publication with Crosswalk from
2003 Guidelines)
• Item QC.1.70
• Method Validation requirements for precision,
accuracy and reportable range apply to FDA
cleared, unmodified moderate complexity tests.
• Requires reference range verification for all
new methods
• Item QC.1.80
• Method-to-method and instrument-to-instrument
requirement specifies that waived tests are
exempt.
r
Informative Regulatory Links
• http://www.phppo.cdc.gov/clia/regs2/toc.asp (CLIA Regs
with 1/24/03 updates)
• http://www.aacc.org/meetings/qc/ (Oct. 29th
Teleconference
• http://www.jcaho.org/accredited+organizations/laborator
y+services/standards/new+standards/2004+standards.h
tm (JCAHO Pre-publication 2004 Standards with
Crosswalk)
• http://www.cap.org/html/ftpdirectory/checklistftp.html
(CAP Checklist download page)
• http://www.dgrhoads.com/ee5/ee5.sht (EP Evaluator
Release 5)
r
Links to Informative Articles
• http://www.cap.org/captoday/archive/2003/0303/langua
ge_barrier.html (CAP Today on AMR/CRR and
Calibration Verification)
• http://www.westgard.com/cliafinalrule5.htm (CLIA
1/24/03 method validation changes by Sharon
Ehrmyeyer, Ph.D)
• http://www.westgard.com/essay50.htm (CLIA Key
1/24/03 Changes by Dr Westgard)
• http://www.westgard.com/essay15.htm (Method
Validations: The Inner, Hidden, Deeper, Secret Meaning
by Dr Westgard)
r
Method Validation Studies
Questions and Discussion
r