Transcript Statistical Considerations for Defining Cut Points

Statistical Considerations for
Defining Cut Points and Titers in
Anti-Drug Antibody (ADA) Assays
Ken Goldberg, Non-Clinical Statistics
Johnson & Johnson Pharmaceutical Research &
Development, LLC, Chesterbrook, PA
Midwest Biopharmaceutical
Statistics Workshop
Muncie IN, May 24-26, 2010
Outline
• Introduction
– Why are ADA and IR assays important?
• Two case studies
1. RIA: How to define %binding?
2. ECL: How to define titer cut point?
3. Both use a Huber 3-parameter
nonlinear logistic regression
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 2
Immune Response (IR) Assay
•
•
•
•
Primary question: ADA, Yes or No?
Every biologic must be evaluated.
Safety and Efficacy concerns.
Too much IR can kill a compound.
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 3
Biological Drug Products are
Different than Traditional
Small Molecule Drugs
• Made by cells not chemists
• Complicated manufacturing process
• Small & simple vs large & complex
chemical structures
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 4
Reference: Genentech, Inc. http://www.gene.com/gene/about/views/followon-biologics.html
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 5
Adverse Clinical Sequelae
• Hypersensitivity & autoimmunity
• Altered PK
– Drug neutralization
– Abnormal biodistribution
– Enhanced clearance rate
 Regulatory bodies require ADA
evaluation for all biologics
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 6
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 7
Immune Response (IR) Assay
Challenges
• Cut Point for confidence that screening
bioassay response (eg, ECL, OD, RLU,
CPM) reflects immunogenicity
• Statistical issues of variance
components, distributions, outliers, …
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 8
Screening Cut Point Flags 5% of
Naïve Samples as False
Positive
• Use Mean + 1.645 x SD with caution
– Only for normally independently
distributed data without outliers
– Usually requires at least a
transformation like logs
• Nonparametric often easier
– Simply use 95th percentile
– Caution if unbalanced design
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 9
ELISA Activity
Positive Negative Patient
Control Control
A
1.689
0.153
0.055
Patient
B
0.412
Patient Assay
C
Control
1.999
0.123
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 10
ELISA Cut Point Example
Histogram of -1/OD^.75
Normal Distribution Overlaid
-1.61
25
Mean
StDev
N
Frequency
20
15
10
5
0
-24.5
-21.0
-17.5
-14.0
-10.5
-1/OD^.75
-7.0
-3.5
0.0
Mean and Standard Deviation based on mixed effects analysis of 117 non-outliers.
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 11
-3.872
1.381
118
Analysis of an RIA Cut Point
Assay Validation Experiment
•
•
•
•
•
•
6 Assay controls
2 Analysts with 3 assays each
2 Populations (Normal and Diabetes)
75 Naïve Human Serum samples
Nonnormal data
Unequal variances
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 12
RIA Histogram of 450 Naïve Sample Results
Transformed %Binding = ln(35+%Binding). Parametric Cut Point = 10.757 ± 3.524.
Transformed Cut Point = 3.823 based on adjusted mean = 3.402 and total standard deviation = 0.256.
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 13
RIA Normal Probability Plot of 450 Naïve Sample Results
Transformed %Binding = ln(35+%Binding). Parametric Cut Point = 10.757 ± 3.524.
Transformed Cut Point = 3.823 based on adjusted mean = 3.402 and total standard deviation = 0.256.
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 14
SAS Code
proc mixed; * For Cut Point;
class sample run analyst;
model t35Pct0_100= / ddfm=sat;
random sample;
random sample / type=sp(exp)(tube) subject=analyst*run;
repeated / group=analyst*run;
proc mixed; * For Example Hypothesis Test;
class sample run analyst;
model t35Pct0_100 = Analyst Tube / ddfm=sat;
random sample;
random intercept tube / type=fa0(2) subject=analyst*run;
repeated / group=analyst*run;
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 15
My RIA Notation
MinCPM = Minimum of the 2 Sample CPMs
MaxCPM= Maximum of the 2 Sample CPMs
AvgCPM = Average of the 2 Sample CPMs
CV
= Coefficient of Variation of the 2 Sample CPMs
B0
B100
B250
B1000
NSB
TC
= Average of all 6 “Validation sample 0 ng/mL” CPMs
= Average of all 6 “Validation sample 100 ng/mL” CPMs
= Average of all 6 “Validation sample 250 ng/mL” CPMs
= Average of all 6 “Validation sample 1000 ng/mL” CPMs
= Average of all 2-6 “NSB” (Non-Specific Binding) CPMs
= Average of all 2-6 “TC” (Total Count) CPMs
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 16
Some RIA %Binding Definitions
Response
%CV Sample Sample Sample
Sample
1
Limit
N
Mean
SD Addend
%CV1
(MinCPM-B0)/(B100-B0)*100
450
-3.490
7.968
65
13.0
420
-1.173
8.373
85
10.0
(MinCPM-NSB)/(TC-NSB)*100
450
1.249
0.841
4.4
14.9
MinCPM/NSB
450
1.321
0.218
-0.7
35.0
403
5.459
1.119
3
13.2
MinCPM-B0
450 -59.339 151.356
1000
16.1
MinCPM/sqrt(B100*B0)
450
0
15.3
(AvgCPM-B0)/(B100-B0)*100
AvgCPM/(TC-NSB)*100
25
20
0.549
1CV
0.084
of (Response + Addend) = Standard Deviation / (Mean + Addend) x 100%.
Addend chosen so that CV is not related to control concentration.
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 17
How to Choose the RIA
%Binding Definition?
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 18
New versus Old RIA
%Binding Definitions
• New: (MinCPM – B0) / (B100 – B0)
– Repeat if CV > 25% and (MaxCPM – B0) /
(B100 – B0) > 12.0% (the Cut Point)
• Old: (AvgCPM – NSB) / TC
– Repeat if CV > 20%
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 19
Attributes of Selected RIA
%Binding Definitions
%CV Cut LOD 0 ng/mL
Response
Limit Point (ng/mL) %Pos. N
(MinCPM-B0)/(B100-B0)
.120
23.5
0.04 450
(AvgCPM-B0)/(B100-B0)
25
.149
25.5
1.29
420
(AvgCPM-B0)/(B100-B0)
20
.153
25.0
0.10
403
(AvgCPM-NSB)/TC
20
3.380
31.7
0.112
403
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 20
RIA Validation Control Curve with Lower 1-sided 95% Prediction Limit
65 + %Binding = A+B·ConcentrationC
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 21
A Logistic Curve with an Infinite

Plateau is Linear wrt X
C + R XH / ( MH + XH) =
Substitute α = C,  = H, and R/β = MH
α + R X
/ (R/β + X) =
Multiply second term by β/β
α + β R X / ( R + βX)
Apply L’Hopital’s rule
Lim[ α + R β X / (R + β X) ] = α + β X
(R)
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 22
RIA Naïve Sample %Binding vs Test Tube Order by Population
Scatterplot of MinPct0_100 vs Tubepair
40
Population
Diabetes
Normal
30
MinPct0_100
20
10
0
-10
-20
-30
0
20
40
60
80
100
Tubepair
120
140
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 23
160
180
RIA Naïve Sample %Binding vs Test Tube Order by Analyst
Scatterplot of MinPct0_100 vs Tubepair
40
A naly st
1
2
30
MinPct0_100
20
12.05
10
0
-10
-20
-30
0
20
40
60
80
100
Tubepair
120
140
160
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 24
180
RIA Naïve Sample %Binding vs Test Tube Order by Analyst and Run
Scatterplot of ln(35+Pct0_100)*100 vs Tubepair
50
1, 1
100
150
1, 2
1, 3
400
ln(35+Pct0_100)*100
385.1
300
2, 1
2, 2
200
2, 3
400
385.1
300
200
50
100
150
Tubepair
Panel variables: Analyst, Run
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 25
50
100
150
RIA Naïve Sample Means vs Test Tube Order by Population, Analyst and Run
Scatterplot of ln(MeanPct+35)*100 vs Tubepair
50
1, 1
100
150
1, 2
1, 3
450
ln(MeanPct+35)*100
400
350
300
250
2, 1
450
2, 2
2, 3
400
350
300
250
50
100
150
50
100
Tubepair
Panel variables: Analyst, Run
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 26
150
Population
Diabetes
Normal
RIA Naïve Sample Mean %Binding vs CV by Analyst and Run
Scatterplot of ln(MeanPct+35)*100 vs lnCV
-4
1, 1
0
4
1, 2
1, 3
450
ln(MeanPct+35)*100
400
350
300
250
2, 1
450
2, 2
2, 3
400
350
300
250
-4
0
4
lnCV
-4
Panel variables: Analyst, Run
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 27
0
4
RIA Naïve Sample Minimum %Binding vs CV by Analyst and Run
Scatterplot of ln(35+Pct0_100)*100 vs lnCV
-4
1, 1
0
4
1, 2
1, 3
400
ln(35+Pct0_100)*100
385.1
300
2, 1
2, 2
200
2, 3
400
385.1
300
200
-4
0
4
lnCV
-4
Panel variables: Analyst, Run
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 28
0
4
RIA Naïve Sample CPM CV vs Mean by Analyst
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 29
RIA Naïve Sample CPM CV vs Mean by Population and Control
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 30
ProbabilityPlots
Plotofofln(35+%Binding)•100
ln(35+Pct0_100)*100
RIA Probability
by Analyst
385.1
99.9
Analyst
1
2
99
Mean StDev
N
AD
P
344.0 24.90 225 2.052 <0.005
339.9 24.84 225 3.863 <0.005
95
Percent
90
80
70
60
50
40
30
20
10
5
1
0.1
250
275
300 325 350 375
ln(35+Pct0_100)*100
400
425
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 31
Probability
Plot
of ln(35+Pct0_100)*100
RIA Probability
Plots
of ln(35+%Binding)•100
by Population
Normal
99.9
Population
Diabetes
Normal
99
Mean StDev
N
AD
P
341.8 19.87 150 0.542
0.162
342.0 27.14 300 1.573 <0.005
95
Percent
90
80
70
60
50
40
30
20
10
5
1
0.1
250
300
350
400
ln(35+Pct0_100)*100
450
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 32
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 33
Electrochemiluminescence
(ECL) BioVeris Assay
• New way to determine screening cut
point (Data = naïve samples)
• New way to determine titer cut point
(not equal to screening cut point)
(Data = positive samples’ Titration series)
• Estimator of Titer within-assay CV
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 34
Screening Cut Point Determination
ECL of Naïve Sample vs Diluent Alone with Cutoffs by Diluent ECL
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 35
Titer Definition
• Smallest distinct dilution in a titration
series with a negative response
– Response is Sample ECL mean / Diluent
Control ECL mean in this case study
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 36
Plot where Sample/Diluent Control ECL Ratio < 4 for 1 Selected Plate out of 24
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 37
Potential Problems with a
Common Screening and
Titer Cut Point
• Highly diluted samples tend to be
positive!
– The opposite would not be a problem
• Titration curve too flat at cut point
– Makes the titer highly variable
– Common
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 38
Titer Cut Point Defined
• The continuous titer inverse predicted
from it has CV ≤ 30.0% with 95%
confidence
– 30.0% makes best case CV = worst
case CV in ideal assay
– Continuous titer is exact dilution giving
cut point (only as a theoretical concept)
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 39
Asymptotic CV
• CV  Standard deviation of natural log
ratio or titer
• CV of dilution@ratio  CV of ratio /
slope of titration curve@ratio
• CV of dilution decreases as ratio and
slope increase
• These CVs are within-plate CVs
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 40
Four Theoretical Titer Distributions
CV = 34.7% = ln(F)/2
30% CV of Continuous Titer
=> Discrete Titer CV = 37.5%
50
Percent
Percent
50% at X and 50% at X*F. CV=ln(F)/2
50
50
50
25
0
2
49
25
0
4
49
Discrete Titer
1
1
1
2
4
8
Discrete Titer
CV = 34.7% = ln(F)/2
30% CV of Continuous Titer
75% at X, 12.5% each at X/F and X*F
=> Discrete Titer CV = 34.7%
75
50
25
0
12.5
2
12.5
4
Discrete Titer
75.16
75
8
Percent
Percent
75
50
25
0
0.03
1
12.39
2
12.39
4
8
Discrete Titer
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 41
0.03
16
Titer Cut Point Defined
• A continuous (interpolated) titer
inverse predicted from it has
CV<30.0% with 95% confidence
– Exact dilution giving cut point (eg,
1.357 ratio) is the continuous titer
– Continuous titer used here only as
a theoretical concept
– Our cut-point 5 SD above diluent
mean so false-positives of
noncensored titers unlikely
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 42
Summary
• All biologics need ADA evaluation
• Use controls to adjust for plate-toplate variance and minimize the LOD
• Define titer cut point so best case CV
= worst case CV in ideal assay
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 43
Acknowledgements:
• Sheng Dai
• Allen Schantz
• Pam Cawood
• Gopi Shankar
• Bill Pikounis
Reference:
Shankar, G. et al, (2008). Recommendations for the
validation of immunoassays used for detection of
host antibodies against biotechnology products.
Journal of Pharmaceutical and Biomedical Analysis.
48:1267–1281.
Statistical Considerations for Defining Cut Points and Titers in ADA Assays.
Ken Goldberg. Midwest Biopharmaceutical Statistics Workshop, May 24-26, 2010. Slide 44