Transcript Preventing OOS Deficiencies

Preventing OOS Deficiencies
Lynn Torbeck
7/18/2015 1:58 AM
Preventing OOS Deficiencies
1
List of Topics
Briefly review:




Barr Case
FDA OOS Guidance
Able Laboratories Story
PDA Scientific Advisory Board Committees
Troublesome fundamentals
Unresolved issues
Preventing OOS deficiencies
Final recommendations
7/18/2015 1:58 AM
Preventing OOS Deficiencies
2
Barr Case
Audited in 1989, 1991 and 1992.
Refused to accept a consent decree.
FDA was forced to go to court.
Civil action taken June 1992.
Decision in favor of the FDA on
February 4, 1993.
7/18/2015 1:58 AM
Preventing OOS Deficiencies
3
Barr and Statistical Issues
Initial investigations
Full investigations
Testing
Retesting
Averaging
Outliers techniques
7/18/2015 1:58 AM
Preventing OOS Deficiencies
4
OOS Logic
by Lynn Torbeck
Tes t
D oc um ent
Out of Spec ?
R eport out
res ults
No
R es am ple as
needed
Y es
Inv alidate dat a,
D oc um ent,
Start Ov er
Y esKnown Phy s ic al
R eas on?
No
No
Y es
Original s am ple
repres ent ativ e?
Marc h 1999
Y es
U SP allows
res am ples ?
2 nd St age
(i. e. C ont ent
U nif orm it y )
Y es
Pas s es ?
No
H is t oric al D at a
Lab or
Analy s t
error?
Begin Lab Init ial
Inv es tigation
Y es
No
U SP <111> Outlier ID
R et es t
Pas s es ?
Y es
Copyright 1999 by
L. Torbeck
Ot her
reas on?
F inis h
inv es t igation
No
D oc um ent
jus t if ic ation
Y es
F inis h
Inv es tigation
No
D oc um ent
and report
No
R ev iew is
inc onc lus iv e
F ull
Inv es tigation
Y es
All pas s ?
R et es t n tim es
Y es
R et es t
jus t if ied?
No
No
7/18/2015 1:58 AM
Preventing OOS Deficiencies
5
Barr: Lessons Learned
FDA takes OOS issues very seriously.
OOS SOP’s, laboratory logs and
documented investigations will be part
of any Quality System review.
Companies are still getting Form 483
observations for not having an
adequate SOP or for not following the
SOP.
7/18/2015 1:58 AM
Preventing OOS Deficiencies
6
Barr: OOS Prevention
Analysts, supervisors and managers
should read and discuss the Barr case
and understand the OOS issues in
context.
7/18/2015 1:58 AM
Preventing OOS Deficiencies
7
FDA Guidance
“Investigating Out of Specification
(OOS) Test Results for Pharmaceutical
Production.”
Issued as a draft in September 1998.
Still in draft as of today.
FDA has sent it to the attorneys.
Final version could be out this year.
7/18/2015 1:58 AM
Preventing OOS Deficiencies
8
Draft: OOS Prevention
All laboratory personnel, analysts,
supervisors and managers should read,
study and discuss in-depth, sentence by
sentence if necessary, the draft OOS
guidance.
Then do it again when the final
guidance is released.
7/18/2015 1:58 AM
Preventing OOS Deficiencies
9
Able Labs – Cranbury, NJ
Massive number of OOS errors
Recall of all 46 products
3,184 lots recalled
Five ANDA’s withdrawn
Hundreds of staff laid off
Sold to Sun Pharm in December 2005
www.ablelabs.com
7/18/2015 1:58 AM
Preventing OOS Deficiencies
10
Able Labs: Lessons Learned
It is still possible to have wide spread
misunderstanding of the Barr case, the
OOS guidance and OOS SOPs.
Apparently the analysts felt they could
not give an “incorrect result.”
Management needs to instill and
cultivate a “GMP Culture” in the
analytical laboratory.
7/18/2015 1:58 AM
Preventing OOS Deficiencies
11
Able Labs: OOS Prevention
Review the Able Labs web site.
Discuss the Able Labs story with
laboratory analysts, supervisors and
managers.
Discuss what a “GMP Culture” means in
the analytical laboratory and how to
develop and reward it.
7/18/2015 1:58 AM
Preventing OOS Deficiencies
12
PDA OOS Committees
Chemical OOS:




Lynn Torbeck, Chair
Eight members
Draft technical report reviewed by the FDA
Planning a PDA/FDA conference
Microbial Data Deviations:


Jeanne Moldenhauer, Chair
Draft in revision
7/18/2015 1:58 AM
Preventing OOS Deficiencies
13
Troublesome Fundamentals:
Outliers
Reportable Values
Averaging
Testing into compliance
Full consideration
7/18/2015 1:58 AM
Preventing OOS Deficiencies
14
Outliers - Defined
Extreme values vs outliers:
7/18/2015 1:58 AM
Preventing OOS Deficiencies
15
Outliers – Judge Wolin
"The USP expressly allows firms to
apply this test (outlier) to biological and
antibiotic assays, ..., but is silent on its
use with chemical tests.”
"In the Court's view the silence of the
USP with respect to chemical testing
and outliers is prohibitory."
7/18/2015 1:58 AM
Preventing OOS Deficiencies
16
Outliers - Investigation
"In chemical procedures, where method
accuracy variation is small, an outlier test
may be appropriate as part of an OOS
investigation, provided the sample and test
procedure assumes homogeneity ... as in the
composite strength assays. Our current
thinking is that outlier tests are never
appropriate where the purpose of the sample
is to measure uniformity"
Paul Vogel,
September 10, 1993.
7/18/2015 1:58 AM
Preventing OOS Deficiencies
17
Outliers - Tests
Dixon's criteria, the test in USP<111>, is
general in nature and not specific to
biological issues. It can be used anywhere
the statistical assumptions can be met.
In general, statisticians agree with the
philosophy that outlier tests should be used
infrequently and with great caution.
7/18/2015 1:58 AM
Preventing OOS Deficiencies
18
Outliers - Recommendations
Don't use any outlier rejection test for
rejection of chemical test results. But it can
be used as supporting information in an OOS
investigation to consider retesting.
Keep all data, especially suspect data, for
future review. Unusual data when seen in
context and with other historical data often is
not unusual at all, but in fact forms a known
and well-behaved statistical distribution.
7/18/2015 1:58 AM
Preventing OOS Deficiencies
19
Reportable Values
“Reportable Values for Out of
Specification Test Results”




Lynn Torbeck
Pharmaceutical Technology
Vol. 23, No. 2, February 1999
Special Supplement
7/18/2015 1:58 AM
Preventing OOS Deficiencies
20
FDA R.V. Definition
“It should be noted that a test might
consist of replicates to arrive at a result.
For instance, an HPLC assay result may
be determined by averaging the peak
responses from a number of
consecutive, replicate injections from
the same preparation. The assay result
would be calculated using the peak
response average.”
7/18/2015 1:58 AM
Preventing OOS Deficiencies
21
FDA R.V. Definition
“This determination
is considered one
test and one result.”
7/18/2015 1:58 AM
Preventing OOS Deficiencies
22
Implications of FDA Definition
A reportable value is the end result of
the complete measurement method as
documented.
It is the value compared to the
specifications.
It is the value used for official reports.
It is usually the value used for statistical
analysis.
7/18/2015 1:58 AM
Preventing OOS Deficiencies
23
Figure 1
Batch
Sample
Preparation
Inj
Reportable
Value, RV
Figure 1
7/18/2015 1:58 AM
Preventing OOS Deficiencies
24
Figure 2
Batch
Sample
Preparation 1
Inj1
Inj 2
Inj 3
Preparation 2
Inj 4
Inj 5
Inj 6
Preparation 3
Inj 7
Inj 8
Inj 9
Reportable
Value, RV
Figure 2
7/18/2015 1:58 AM
Preventing OOS Deficiencies
25
Figure 3
Batch
Sam ple
Retes t
RV
Res am ple
RV
Preparation 1A
Repreparation
2B
Preparation 1B
Preparation 1C
Inj1
Inj 2
Reportable
Value, RV
Inj 3
Repreparation
2C
Reinjection
Inj 4
Inj 5
Inj 6
Rem eas ure
Inj 7
Inj8
Inj 9
Rem eas ure
Figure 3
7/18/2015 1:58 AM
Preventing OOS Deficiencies
26
Interpretation
The individual determinations do not have to
meet the specification.
Individual determinations are not reported
out of the lab.
However the variability of the determinations
is a system suitability issue.
Set a limit on the standard deviation or
%RSD.
7/18/2015 1:58 AM
Preventing OOS Deficiencies
27
R.V.: OOS Prevention
Record in writing the operational
definition of the Reportable Value for
each test method in the method
documentation, any protocols and any
reports.
Add “Only this reportable value can be
compared to the specification criteria.”
7/18/2015 1:58 AM
Preventing OOS Deficiencies
28
Averaging
Specifically, the arithmetic mean; the sum of
all of the numbers divided by the count of the
numbers.
More generally, it is a value that represents
the central point of a data set. (In this sense,
it can include the arithmetic average, the
median, the mode, the geometric mean or
the harmonic mean.)
7/18/2015 1:58 AM
Preventing OOS Deficiencies
29
Averaging
"... as a general rule, firms should avoid this
practice, because averages hide the
variability among individual test results.“
"[Averaging] is particularly troubling if
testing generates both out-of-specification
and passing individual results which when
averaged are within specification.
"Here, relying on the average figure without
examining and explaining the individual
out-of-specification results is highly
misleading and unacceptable."
7/18/2015 1:58 AM
Preventing OOS Deficiencies
30
Averaging
"Averaging the results of tests intended to
measure the uniformity of the test article is
not current good manufacturing practice ...
because it may hide the variability of the
sample the test procedure is intended to
detect. For this reason, all individual test
results must be reported and evaluated on an
independent basis" Paul Vogel, September
10, 1993.
7/18/2015 1:58 AM
Preventing OOS Deficiencies
31
Averaging: OOS Prevention
Do not average out of specification
reportable values within specification
reportable values to get an in
specification result.
Do not average reportable values for
QA to make a decision. QA must see all
individual reportable values, OOS and
retests.
7/18/2015 1:58 AM
Preventing OOS Deficiencies
32
Testing Into Compliance
Torbeck, L., “Preventing the Practice of
Testing into Compliance”, Pharmaceutical
Technology, Oct 2002.
Testing into compliance is the practice of
ignoring valid information that should be used
to make decisions.
Such a practice is at best not scientific and at
worst is fraudulent, illegal, and immoral.
Such practices if found must be stopped.
7/18/2015 1:58 AM
Preventing OOS Deficiencies
33
Testing Into Compliance
Averaging OOS results with in specification
results to get an in specification result.
Physically averaging powers, granulations and
liquids to get in specifications results.

If not part of the validated process.
Discarding data or not recording data until is
known to be in specification.
Missing samples and rejected cans.
Overwriting HPLC chromatograms.
7/18/2015 1:58 AM
Preventing OOS Deficiencies
34
Not Testing Into Compliance
Large initial sample sizes are acceptable
if all data generated is reported.
Large number of retests are acceptable
if all data generated is reported.
Failing system suitability is not an OOS.
Out of limits for an in-process
adjustment is not an OOS.
7/18/2015 1:58 AM
Preventing OOS Deficiencies
35
Compliance: OOS Prevention
Train all laboratory personnel, analysts,
supervisors and managers to be able to
identify specific situations of testing into
compliance.
Train to be able to defend situations
that are not testing into compliance
during an audit.
7/18/2015 1:58 AM
Preventing OOS Deficiencies
36
Full Consideration
“For inconclusive investigations …. The
OOS result should be retained in the
record and given full consideration in
the batch or lot disposition decision.
This statement has caused some
discussion as it is considered to be
vague and undefined. It can, I think, be
defined in a simple way.
7/18/2015 1:58 AM
Preventing OOS Deficiencies
37
Full Consideration
First, all QA decisions are made with the
Reportable Values, both OOS and
retests.
Second, QA looks at the magnitude of
the retest values compared to the
specifications.
7/18/2015 1:58 AM
Preventing OOS Deficiencies
38
Full Consideration
If the retest values are close to the
target, the lot can be released.
If the retest values are close to the limit
that the OOS exceeded, technically the
lot can be released, but QA should
consider further investigation to
determine why the retests are not at
target.
7/18/2015 1:58 AM
Preventing OOS Deficiencies
39
Consideration: OOS Prevention
QA should detail and document the
logic and rational for decisions based on
retesting results after a OOS result is
found.
7/18/2015 1:58 AM
Preventing OOS Deficiencies
40
Unresolved Issues
Specification Limits for OOS?
What size the retest sample?
Second analyst?
Statistical treatment of data?
7/18/2015 1:58 AM
Preventing OOS Deficiencies
41
Specification Limits for OOS?
Regulatory Limits
Release: accept/reject
Action limits, Cpk=1.33
Alert, Cpk=1.0



Warning limits
Trend
Validation limits
7/18/2015 1:58 AM
Preventing OOS Deficiencies
42
Specification Limits
Cpk=1.0
Cpk=1.33
Accept/Reject
Regulatory
7/18/2015 1:58 AM
Preventing OOS Deficiencies
43
Specification: OOS Prevention
Define in writing the levels of
specification criteria.
Justify in writing which specifications
are considered applicable to OOS and
why or why not.
7/18/2015 1:58 AM
Preventing OOS Deficiencies
44
What Size the Retest Sample?
“… a matter of scientific judgment,”
“… retesting cannot continue ad infinitum.”
“Such a conclusion cannot be based on on 3
of 4 or 5 of 6 passing results, but possibly 7
of 8.”
“… will vary on a case by case basis … “
“… an inflexible retesting rule … is
inappropriate.”
7/18/2015 1:58 AM
Preventing OOS Deficiencies
45
What Size the Retest Sample?
“The number of retests … should be specified
in advance …”
“The number of tests should not be adjusted
‘on-the-fly’, as results are being generated.”
“… a firm’s predetermined testing procedure
should contain a point at which testing ends
and the product is evaluated.”
7/18/2015 1:58 AM
Preventing OOS Deficiencies
46
What Size the Retest Sample?
This is an unresolved issue and the
statisticians are still publishing journal
articles and discussing it.
Barr case n=7.
Could be too much or not enough.
Currently n= 3 to n=9.
PDA OOS committee will recommend.
7/18/2015 1:58 AM
Preventing OOS Deficiencies
47
Retest References
Hofer, J., Considerations when
determining routine sample size for a
retest procedure, Pharmaceutical
Technology, Nov. 2003.
Anderson, S., An alternative to the ESD
approach, Pharmaceutical Technology,
May 2004.
7/18/2015 1:58 AM
Preventing OOS Deficiencies
48
Retest: OOS Prevention
Define in writing the sample size for
retests or define the procedure to be
used to determine the sample size.
Provide scientific justification.
7/18/2015 1:58 AM
Preventing OOS Deficiencies
49
Second Analyst
Guidance suggests a second analyst.
Issues:




Added complication and variation
May not have a second analyst
May not find the root cause
Second analyst may not be as proficient
Recommend that the manager decide
and justify decision in writing.
7/18/2015 1:58 AM
Preventing OOS Deficiencies
50
Statistical Treatment of Data
“Statistical treatments of data should
not be used to invalidate a discrete
chemical test result.”
“ … a statistical analysis may be
valuable as one assessment of the
probability of the OOS result …
Another way to say outlier rejection.
7/18/2015 1:58 AM
Preventing OOS Deficiencies
51
Preventing OOS Deficiencies:
Setting specification criteria
Statistical Thinking
Sources of variation
Common cause vs. special cause
Variation reduction
Training
Education
7/18/2015 1:58 AM
Preventing OOS Deficiencies
52
Setting Specification Criteria:
Two sides to the OOS issue.
Incorrect limits are the major source of
OOS.
Many specifications were set early in
the development process and may not
be appropriate for the current process.
Many specification were set using
wishful thinking or incorrect approach.
7/18/2015 1:58 AM
Preventing OOS Deficiencies
53
Setting Specification Criteria:
Use historical data
Use distribution analysis

Normal, log-normal, exponential
Don’t use X bar  3S
Use Statistical Tolerance Intervals


X bar  K S for the alert limits
where K is based on confidence and
percent of future values
7/18/2015 1:58 AM
Preventing OOS Deficiencies
54
Setting Specification Criteria
For action limits, permit the average to
vary and widen the Tolerance Limits
For accept/reject limits, add a further
allowance for stability.
Consider the clinical results as part of
the justification for limits.
7/18/2015 1:58 AM
Preventing OOS Deficiencies
55
Statistical Thinking
1. All work occurs in a system of
interconnected processes.
2. All processes have variability.
3. Process understanding and variability
reduction is the key to success.
Variation is the enemy.
7/18/2015 1:58 AM
Preventing OOS Deficiencies
56
Sources of Variation:
Common cause variation:






People
Materials
Methods
Measurement
Machines
Environment
Special cause variation:

One single factor changed
7/18/2015 1:58 AM
Preventing OOS Deficiencies
57
Common vs. Special Causes
I Chart for Yield%
106
1
105
104
Individual Value
A plot of the data
with natural limits
illustrates common
cause variation.
A value that is larger
than would be
expected by chance
alone is assumed to
be due to a special
cause. Use CAPA to
find it.
103
UCL=103
102
101
100
Mean=100
99
98
97
LCL=97
96
0
50
100
Observation Number
7/18/2015 1:58 AM
Preventing OOS Deficiencies
58
Variability Reduction:
Display boards
Operational definitions
Work to target, Target ( Low, High )
Flexible consistency
Hold constant
Mistake proofing
High tech equipment
7/18/2015 1:58 AM
Preventing OOS Deficiencies
59
Training
Training is for a specific task or SOP.
The goal is consistency.
Freelancing causes problems.
Little background is provided.
An in-depth understanding is not
needed to be in compliance if the SOP
is followed.
7/18/2015 1:58 AM
Preventing OOS Deficiencies
60
Education
Someone needs to:



Learn and understand the basic philosophy
and principles.
Know the background as it relates to the
topic.
Understand the material well enough to be
able to make difficult decisions with
confidence and be able to defend them.
7/18/2015 1:58 AM
Preventing OOS Deficiencies
61
Need for Understanding
Why was Able Labs out of compliance?
Defend Reportable Values.
Defend specifications applicable to OOS
Defend not testing into compliance.
Defend retest sample size.
Why variability reduction is needed.
7/18/2015 1:58 AM
Preventing OOS Deficiencies
62
Final Recommendations
Read and understand the Barr Case.
Read and study in-depth the OOS
Guidance. Once is not enough.
Audit the company SOP against the
Guidance line by line.
Have an active program to reduce OOS
results.
Keep management informed.
7/18/2015 1:58 AM
Preventing OOS Deficiencies
63
Thank You
That ends my presentation.
We are now ready for questions and
answers.
7/18/2015 1:58 AM
Preventing OOS Deficiencies
64
References
1.
2.
3.
4.
5.
6.
7.
USA vs. Barr Laboratories, Inc. Civil Action No. 92-1744, US District
Court for the district of New Jersey, February 4, 1993.
FDA, CDER, “Guidance for Industry, Investigating Out of
Specification (OOS) Test Results for Pharmaceutical Production,”
September 1998.
WWW.AbleLabs.com
Torbeck, L., “Reportable Values for Out-of-Specification Test
Results,” Pharmaceutical Technology, February 1999.
Torbeck, L., “Preventing the Practice of Testing into compliance,”
Pharmaceutical Technology, October 2002.
Hahn, G and Meeker, W., Statistical Intervals, John Wiley & Sons,
1991.
Torbeck, L., “Statistical Thinking,” Pharmaceutical Technology, July
2001.
7/18/2015 1:58 AM
Preventing OOS Deficiencies
65