Transcript Stan Letovsky Senior Director, Computational Sciences Costs and Benefits of Biomarkers in Clinical Trials Washington D.C. September 29, 2006 © 2006 Millennium Pharmaceuticals Inc.

Stan Letovsky
Senior Director, Computational Sciences
Costs and Benefits of
Biomarkers in Clinical Trials
Washington D.C.
September 29, 2006
© 2006 Millennium Pharmaceuticals Inc.
Drug Response/Toxicity Biomarkers
• Biomarker is a measurement or test on a
patient that can predict (with some probability)
– Efficacy of a treatment
– Toxicity of a treatment
– Disease severity (independent of drug)
• E.g. Gleevec/BCR-ABL, Iressa/EGFRmut
• Drug-specific biomarkers need to be validated
in clinical trials to affect approvals.
©2006 Millennium Pharmaceuticals, Inc.
2
©2006 Millennium Pharmaceuticals, Inc.
3
Question
Under what circumstances does it make
sense to include a biomarker efficacy
hypothesis as part of the main study
objectives of a clinical trial?
• What are the costs?
– Assays, logistics
– P-value / sample-size adjustments
• What are the benefits?
– Increased probability of drug approval
©2006 Millennium Pharmaceuticals, Inc.
4
Possible Trial Designs
• Traditional – efficacy only, no biomarker component
• Biomarker Discovery – hitchhike on phase 2-3 trial, resulting
biomarkers not validated.
• Static Biomarker trial – specific biomarker hypotheses tested as
part of trial design, could yield validated biomarkers and
stratified market. Patient population not biased by biomarker.
• Adaptive Validation – a form of adaptive trial in which a
biomarker hypothesis is formulated at an interim point. May
yield a validated biomarker. No biased sampling.
• Adaptive Sampling – a form of adaptive trial in which a
biomarker hypothesis is evaluated at an interim point, and
subsequent patient selection may be biased by the biomarker.
– for Response: Sampling biased towards responding subset / away from
adversely-responding subset
– for Speed: Sampling biased towards severest disease for faster trial.
– for Power: Sampling is biased to allocate more sample to the
hypothesis that is most likely to benefit.
©2006 Millennium Pharmaceuticals, Inc.
5
Multiple Comparison Corrections
• Study Design#1:
– Hypothesis H: “drug not efficacious”
• Significance threshold a=.05
• Study Design#2:
– Hypothesis H0: “drug not efficacious”
• Significance threshold a=.04
– Hypothesis H1: “drug not efficacious in
biomarker positive population”
• Significance threshold a=.05-.04=.01
©2006 Millennium Pharmaceuticals, Inc.
6
Power Curves for Static Design (schematic)
$$
For a given choice of a
(significance) and b (power)
get curve of N vs F.
$
6.8% for a1=.01!!
n=Max affordable study size or duration
N=
Sample
Size
H0 powered at a0 <a : (f0>f , N0)
H powered at a: (f,,n)
Hi powered at ai : (fi>f0 , Ni=N0*pi)
Adding biomarker hypothesis imposes a
multiple comparisons “tax” that must be paid in
dollars (by increasing sample size), sensitivity
(increasing F) or risk (decreasing power).
or Min
clinically
acceptable
effect
f
f0
f1
F = Effect size (e.g. TTP for new drug + SOC / SOC alone)
©2006 Millennium Pharmaceuticals, Inc.
7
F0>=F1*p1
Parameter Space for
Static Design
Biomarker Win:
Reject H1 only:
biomarker pays off;
stratified market
better than none.
Payoff=p1 vs. 0
F1 = mean
effect size
f
in biomarker 1
population
Must have
f1*p1 < f for
biomarker
strategy to be
viable. The
steeper the
line, the
smaller
market.
F0<F1
Impossible to be left of blue line
Drug Failure:
Possible Partial
Backfire:
Possible
Partial
Backfire#2:
Apparent
success of
H0
explained
by H1
Reject H1 only,
would have
rejected H0 w/o
biomarker.
Market may be
Trial outcome is
stratified;
a point1 in
payoff=p
or the
1
vs.
1. 1 plane
F0,F
Biomarker
Backfire:
Reject none,
drug is no
good,
biomarker
didn’t help.
Fail to reject H0
Slope
line have
is if
butof
would
biomarker
you hadn’t used
enrichment
B1
the biomarker.
Total loss of
market.
Payoff = 0 vs. 0
Redundant:
Reject both: didn’t
need biomarker.
Biomarker not
predictive on
green line,
antipredictive
below
Payoff = 1 vs. 1
Biomarker Failure:
Reject H0 only; biomarker
useless, no harm done.
Payoff=1 vs. 1
Payoff = 0 vs. 1
f1*p1 f
f0
F0 = mean effect size in entire study population
©2006 Millennium Pharmaceuticals, Inc.
8
Likelihood: Outcomes Are Not Equally
Probable
Joint Distribution of F0 X F1
Implied Distribution of F 1
Given prior pdf for F0
(e.g. from phase II
results, literature)
and B1, (made up),
can infer (assuming
independence) joint
distribution of F0 X F1
and pdf of F1.
1.5
1.50
1
1.00
0.5
0.50
NB: F=T/C.
0
Biomarker Enrichment
1
Pr(F1=x)
0
0.00
0
Pr(F0=x)
Pr(B i=x)
4
2
0
0.5
1
1.5
Prior on B 1
2
2.5
©2006 Millennium Pharmaceuticals, Inc.
0.5
1
1.5
4
2
0
0
0.5
1
1.5
Prior on F0 from phase II
9
Utility: Outcomes Are Not Equally Valuable
Joint Distribution of F 0 X F1
NPV1
50
1.50
e2
1.00
e1
0.50
0.00
0
e0
0.5
1
0.8
30
0.6
20
0.4
10
0.2
1.5
10
EPV1
10
40
8
30
20
4
20
10
2
10
30 40
50
©2006 Millennium Pharmaceuticals, Inc.
40
50
-3
x 10
5
40
6
20
30
50
30
10
20
EPV1 - EPV 0
-3
x 10
50
=
X
40
0
-5
10
20
30
40
50
10
A Biomarker-Favorable Scenario
Joint Distribution of F 0 X F1
NPV1
50
2.00
40
0.8
30
0.6
20
0.4
10
0.2
1.50
1.00
0.50
0.00
0
0.5
1
1.5
EPV1
2
10
30
40
50
EPV1 - EPV 0:  =28%
-3
x 10
50
20
-3
x 10
50
40
4
30
3
30
20
2
20
10
1
10
10
20
30 40
50
5
40
0
-5
10
20
30
40
50
If unlikely to succeed in main trial, but likely in
biomarker subpopulation. Better redesign trial?
©2006 Millennium Pharmaceuticals, Inc.
11
Parsing the Parameter Space
• Simply by assuming reasonable values of f,f0,f1 and
looking at different plausible priors one can learn a lot:
– If the F0 prior makes it likely that F0 > f1, there is no need to
bother with a biomarker.
– If it is likely that F0 > f but it may not be > f1, you may be
better off not risking the multiple comparison “tax”.
– If there is substantial risk that F0 < f and you have a
biomarker with substantial likelihood of significant
enrichment, the biomarker strategy may have higher EPV.
©2006 Millennium Pharmaceuticals, Inc.
12
Multiple Comparison Tax Relief
• Suppose regulator wants to encourage
biomarker validation…
• What is consequence of ignoring a=.01 worth
of multiple comparison correction to main
efficacy hypothesis?
– No change to drug approvals in main study
population – false positive rate of 5% already
deemed societally acceptable.
– 1% Probability of false positive “biomarker wins”
already deemed acceptable in 4%/1% split.
– Assuming something like 10% of biomarkers
tested really are predictive, precision=91%,
FDR=9%.
– Social cost of biomarker backfire avoided
©2006 Millennium Pharmaceuticals, Inc.
13
Adaptive Biomarker Validation
good
Initial
Unbiased
Recruiting
Interim
Evaluation
Of
Biomarker
Add
Biomarker
Hypothesis
To Trial Design
No good
Continue
As
Before
Advantages:
•Can validate biomarker during phase III
Disadvantages:
•Never been done, breaking new regulatory ground
•Some complex statistical issues – bias, multiple comparisons…
©2006 Millennium Pharmaceuticals, Inc.
14
E.g. Freidlin and Simon Adaptive Signature Design, Clinical Cancer Research Vol. 11, 7872-7878, Nov 2005
Biomarker-driven Adaptive Sampling
good
Initial
Unbiased
Recruiting
Interim
Evaluation
Of
Biomarker
Recruit
Biomarker
Positive
Population
No good
Continue
Normal
Recruiting
Advantages:
•Can validate biomarker during phase III
•If biomarker works, save money and/or improve chances of approval
Disadvantages:
•Never been done, breaking new regulatory ground
•Some complex statistical issues – bias, multiple comparisons…
©2006 Millennium Pharmaceuticals, Inc.
15
Parameter Space View of Adaptive Validation
Interim outcome
gives estimate of
final outcome
Uncertainty radius
varies inversely with
interim sample size
Interim outcome
is a point in the
F0,F1 plane
f1
Adaptive strategy is triggered if
interim point falls in a predefined
region. Decision analysis
optimizes shape of region. Want
final point in same (or better)
region as interim point.
f
©2006 Millennium Pharmaceuticals, Inc.
f0
16
Conclusions
• The requirement of correcting for multiple
comparisons has a significant impact on the
incentives for including biomarkers in clinical
trial designs.
• The circumstances under which a cost/benefit
analysis favors inclusion of a biomarker
hypothesis in the main study objectives may
be surprisingly rare.
• Adaptive designs combining biomarker
discovery, validation and use warrant further
investigation.
©2006 Millennium Pharmaceuticals, Inc.
17
Acknowledgements
Millennium
• Mark Chang
• Barb Bryant
• Chris Hurff
• Bill Trepicchio
• Andy Boral
FDA (CDER)
• Gene Pennello
©2006 Millennium Pharmaceuticals, Inc.
18
SM
Breakthrough science. Breakthrough medicine.
©2006 Millennium Pharmaceuticals, Inc.