Transcript kraemer 6 18

Clinically Significant Treatment Effects:
Effect Sizes and Moderators
Helena Chmura Kraemer, Ph.D.
Stanford University (Emerita)
University of Pittsburgh
Randomized Clinical Trails
(RCTs)

RCT: the “gold standard” method to
evaluate the efficacy or effectiveness
of a treatment in a population.
RCT: Background

A patient’s response following
treatment combines:
• Statistical Regression to the mean
• Expectation Effects
• Spontaneous changes
• Secular trends
• PLUS: the actual effect of treatment

Thus the effect of treatment cannot
be assessed in pre-post studies.
RCT: The Effect of Treatment




Definition of “Effect of Treatment”:
• Response(T) compared to Response(not-T)
Requires some control/comparison condition:
“not-T”=C
Requires measurement of response that is not
influenced by T or C: “blindness”
Problem: Cannot observe any individual patient
both with T and not-T…. Thus cannot assess the
effect of treatment on an individual patient.
RCT: Randomization


Can assess the typical effect of treatment in a
population by:
• Sampling the population of interest
• Randomize patients to T or to C.
• Compare T vs C
 Treatment effect on the “typical” patient on.
 Analysis “by intention to treat”.
There is no assumption that the effect of
treatment is the same for all individuals in the
population.
RCT: Comparisons? (Classic)


“Statistical Significance” means that the
design (sample size) was good enough to
detect a non-random difference.
• Comment on design, not on effectiveness
of treatment
 A treatment effect may be statistically
significant and clinically trivial.
• Absence of “statistical significance” means
 Flawed conceptualization, inadequate
design, measurement, sample size
 NOT absence of “clinical significance”.
NEEDED: A measure of effect size that allows
assessment of clinical significance.
Powering up a RCT
T<C
1.000
T=C
T>C
Prob significant result
0.900
0.800
0.700
0.600
0.500
0.400
0.300
0.200
0.100
0.000
0.0
0.2
0.4
0.6
Effect Size, AUC
0.8
1.0
The ”critical value” or “threshold
of clinical significance”?



Below this effect size, clinicians and
patients would consider the effect
clinical trivial.
The more above this value, the
greater the preference for T over C.
Obtained from previous research,
clinical experience, opinion of
experts.
Effect Sizes—Recommended

AUC=Prob(T>C)+.5Prob(T=C)
• Null value: .5; Extremes: 0,1.

SRD=Prob(T>C)-Prob(T<C)=2AUC-1
• Null value: 0; Extremes: -1, +1.

Number Needed to Treat: NNT=1/SRD.
How many patients would you have to
treat with T to get one more success than
if you had treated them with C?
• Null value: infinity: Extremes: -1, +1.
Effect Sizes—More Common

Cohen’s d= Standardized mean
difference between T and C.
• Meant to be used when the responses
have a normal distribution in both T and
C. Almost never exact!
• However, when reasonable:
AUC=normsdist(d/√2)

Odds Ratio—not recommended.
• NNT> (√OR+1)/(√OR-1)
Effect Size—Standards
d
AUC
SRD
NNT
0 Null
.50
0
Infinity
.2 Small
.56
.11
8.9
.5 Med
.64
.28
3.6
.8 Large
.71
.43
2.3
1.0
.76
.52
1.9
Possible RCT Outcomes T1 vs T2: c the critical
effect size: 95% Confidence Intervals for the SRD
*T1 is clinically superior to T2
*T1 is non-inferior to T2
*T1 and T2 clinically equivalent.
T1 and T2 clinically equivalent.
A failed RCT
-1
-c
T1<T2
0
T1=T2
+c
Effect Size (SRD)
+1
T1>T2
Moderators and Mediators of
Treatment in a RCT

Moderator: M moderates the effect of T in
a RCT on O if
• M is a baseline variable (hence precedes T and
O and is uncorrelated with T)
• The effect size of T on O differs depending on
what M is.

Mediator: M mediates the effect of T in a
RCT on O if
• M is an event or change that happens during
treatment (hence follows T but precedes O).
• M is correlated with T
• The effect size of T on O is explained wholly or
in part by the effect of T on M.
Examples: Moderators



Gene (5HTT) moderates the effect of
drug treatment on outcome for
depressed patients
(pharmacognetics).
Baseline depression moderates the
effect of psychotherapy for treatment
of anorexia .
Level of addiction moderates the
effect of smoking cessation treatment
on abstinence.
Example 2: Exploration after a RCT.
How???
Exploration
Secondary
Data analysis
RCT Hypothesis
RCT Design
Publication
Validation
Pilot Study
RCT