Transcript Document

Design Strategies for Behavioral
Intervention Research:
A New Direction
Bibhas Chakraborty
Department of Statistics, University of Michigan
Joint Work with
Linda Collins, Susan Murphy and Vijay Nair
7/17/2015
1
Outline of Talk




Introduction
Underlying Model + Simulation Design
Tutorial on Fractional Factorials
Experimental Approach
7/17/2015
2
Motivational Example I





Fast Track.
Response: a measure of behavior problems.
Psychosocial factors that possibly affect the response
are Peer-pairing, Saturday-sessions, Home-visits,
Tutoring etc.
Goal is to evaluate the multi-component program that
maximizes children’s social skills.
Our method could be used to find the optimal multicomponent program that maximizes the response.
7/17/2015
3
Motivational Example II




Smoking Cessation.
Response: a measure of smoking cessation.
Psychosocial or communication factors that possibly
affect the response are Outcome expectations,
Message-framing, Testimonials, Dose-schedules etc.
Goal is to find the multi-component program that
maximizes smoking cessation.
This is an actual project being studied by the Center
for Health Communications Research, University of
Michigan…
7/17/2015
4
The General Framework




A behavioral intervention program with continuous
response (outcome).
The response possibly depends on many clinical or
psychosocial factors. A factor is a potential component
of a multi-component program… So for example, Peerpairing is a factor in Fast Track.
Most factors are binary (high vs. low), some of them
may have more than two levels though.
The goal is to find the optimal multi-component
program that maximizes the response.
7/17/2015
5
Classical Approach
Classical approach based on Randomized
control trials, followed by Post hoc analysis
for refinement.
 Every subject in treatment group receives
(higher levels of) all program components…
Kitchen sink approach…
This is what actually happened in Fast Track…

7/17/2015
6
Experimental Approach

Multi-phase experimental approach, based
mainly on Balanced Fractional Factorial
Designs. This approach is new to the field…
I will focus on experimental approach in the
rest of this talk…
7/17/2015
7
Why Something New?


The classical approach is essentially a Black
Box. Opening the black box is necessary for
understanding the science better.
Philosophical reason…
It is important to understand the individual
effects of components, to develop a more
efficient and economical package.
Practical reason…
7/17/2015
8
Pros of the New Approach





Opens the Black Box.
Precise notion of effects/interactions.
Individual effects of the program components
and their interactions are estimated and tested.
Based on a principled approach.
Fractional Factorials have been a huge success
in Industrial Engineering.
Why don’t we give it a try?...
7/17/2015
9
Cons of the New Approach


A little bit complicated at the first sight.
May not work well in a setting where each
component in itself is quite weak, but when
put together, they produce a strong effect on
the response.
But this is often not the case due to burden…
More is not always better…
7/17/2015
10
Our Research Goal



Compare the two approaches through a
comprehensive simulation study.
Identify settings where one method will
work better than the other and vice versa.
Operationalize our proposed method so that
a behavioral scientist can use it without the
help of a statistician.
So we are digging our own graves!!
7/17/2015
11
Outline of Talk




Introduction
Underlying Model + Simulation Design
Tutorial on Fractional Factorials
Experimental Approach
7/17/2015
12
The Set-up
The underlying model is a total mediation
model.
 6 factors, A1 … A6: A1 has 3 levels, rest binary.
 6 adherence variables, Ad1 … Ad6.
 6 mediators, M1 … M6.
 An unknown variable ‘Type’.
For example, child’s genetic pre-dispositions…
 The response, Y.

7/17/2015
13
The Model Schematic

Specified in 3 stages, through structural
equations.
η
A
α
Ad
7/17/2015
β
M
Y
14
Structural Model: A Detailed View
Example: Depression
Treatment: Cognitive
Behavioral Therapy (CBT)
A4: Individual counseling
A5: Group therapy…
Ad4, Ad5: Percentage of
sessions attended…
M4: Strategic analysis skills
M5: Social competency…
7/17/2015
This is a hypothetical example…
15
Simulation Design




Each scientist can choose any number of factors (up
to 6) according to prior belief. Not always all…
Priors are not only scientific theory, but often based
on past studies…
Priors are true on average (across scientists). This,
in effect, gives an advantage to classical approach,
though it might not be true in general.
Each scientist uses a total of 2500 subjects in this
study: 1600 in screening phase, 900 in refining
phase.
7/17/2015
16
Why care about Factorials?



Once the scientist decides on which factors to
study, how to design an experiment for that?
Factorial design is an efficient way…
successfully used in agricultural and industrial
experiments for decades!
Cost constraint: Suppose we have money to
construct at most 16 treatment groups!
Now let’s dive into the tutorial on factorials…
7/17/2015
17
Outline of Talk




Introduction
Underlying Model + Simulation Design
Tutorial on Fractional Factorials
Experimental Approach
7/17/2015
18
Design of Experiments

Let us consider the usual linear model:
Y = Xβ + ε



In observational study (say, regression of Y on X),
we observe data on X,Y as they come.
Instead, manipulate X, run the study, observe Y.
Experimental Study.
Design of an experiment means construction of the
X-matrix. For us, this means assigning different
levels of the factors to the subjects…
7/17/2015
19
Factorial Experiments




Studies effects of several factors, each at a
number of levels, simultaneously.
Considers all possible combinations…
Two-level designs: 2 x 2, 2 x 2 x 2, …
Usual coding: +1 (high), -1 (low)
7/17/2015
20
Will focus on
two-level designs
OK in screening phase
i.e., identifying
important factors
7/17/2015
21
4 treatment
groups
8 treatment
groups
Each row represents a treatment group…
7/17/2015
22
Estimated
Averaged over all levels of other factors…
7/17/2015
23
Interaction of X1X2 = (Average response when X1X2 is +)
- (Average response when X1X2 is -)
7/17/2015
24
Design Matrix of Full Factorial
8 rows, i.e., 8 treatment groups! We are fine.
7/17/2015
25
Why Fractional Factorials?



What if the scientist wants to study 4 factors?
Exactly 16 treatment groups will be needed.
Full factorial design is still OK…
How about 5 or all 6 factors? Full factorials
require 32 and 64 groups respectively…But
we have money for 16 groups only…
We can’t afford a full factorial… What to do?
Fractional factorial is a solution…
7/17/2015
26
How?
Consider full factorials…
Order of interactions
Why waste your
resources in
estimating these
interactions that are
often small in size
and can be
ignored?
There is an excess no. of higher-order interactions
that can be estimated in full factorials…But do we
really believe that a 3 or higher order interaction is
meaningful or informative?...
Fractional factorials exploit this redundancy…
7/17/2015
27
First consider a simple toy example…
Suppose we have money to construct only 4 groups.
How to choose a fractional design then?
Each row represents
a group.
We never varied X3!!
Full factorial
with 3 factors
X1 is set at lower level 75% of the
time. The Design is not balanced!!
7/17/2015
We need a principled approach…
28
Here is the solution…
Balanced design
All factors occur at low and high levels
same number of times; same for interactions.
Columns are orthogonal …
 Good statistical properties
7/17/2015
What is the principled approach?...
29
What is the principled approach?
Exploit redundancy in interactions:
Set X3 column equal to
the X1X2 interaction column
You have reduced the no. of groups by 50%...
Is it for free? What is the catch?
7/17/2015
30
Aliasing – A price you must pay
No Free Lunch!!
X1=X2X3
X2=X1X3
X3=X1X2
Each main effect is aliased (mixed) with some twofactor interaction. Consequence? …You can’t estimate
them separately!!
Normally we would not want to alias main effects with
2-way interactions… but again this is just a toy example
for the sake of simplicity!
This is a Resolution III design (1+2=3).
7/17/2015
31
More on Resolution



Resolution III: (1+2)
Main effects aliased with 2-way interactions.
Resolution IV: (1+3 or 2+2)
Main effect aliased with 3-way interactions and 2way interactions aliased with other 2-ways.
Resolution V: (1+4 or 2+3)
Main effect aliased with 4-way interactions and 2way interactions aliased with 3-way interactions.
Higher the resolution, better the design!!
7/17/2015
32
Now let’s address the original question…
We have 6 factors, but just enough money
for only 16 groups. How to choose the design?
X5 = X1*X2*X3; X6 = X2*X3*X4  X5*X6 = X1*X4
Software will construct designs for you!
7/17/2015
33
What is the resolution?
Aliasing Pattern
X5 = X1*X2*X3; X6 = X2*X3*X4  X5*X6 = X1*X4
5 = 123; 6 = 234; 56 = 14 
Main-effects:
1=235=456=12346
2=135=346=12456
3=125=246=13456 …
7/17/2015
Resolution IV Design
15 possible 2-factor interactions:
12=35
13=25
14=56
15=23=46
16=45
24=36
26=34
34
Outline of Talk




Introduction
Underlying Model + Simulation Design
Tutorial on Fractional Factorials
Experimental Approach
7/17/2015
35
Screening Phase: Priors



Scientist chooses those factors that he feels
have positive main effects.
Scientist has prior about interactions as well,
and can choose up to 3 interactions that he
thinks are important and wants to safeguard.
Scientist studies only the two extreme levels
(out of the 3) of the first factor in this phase.
7/17/2015
36
Screening Phase: Designs




We are restricted to 16-treatment groups.
If chosen # factors <=4,
- Full factorial.
If chosen # factors =5,
- Resolution V fractional factorial.
If chosen # factors =6,
- Resolution IV fractional factorial.
These are the optimal designs in these settings…
7/17/2015
37
Role of interactions





Useful when scientist chooses all 6 factors to
study.
Suppose anticipated interactions are:
13, 26, 56
Construct the design so that they are not
aliased among each other. Safeguard them!!
One possible aliasing: 13=25, 26=34, 56=14
There are many other choices!!
7/17/2015
38
Analysis



Usual Linear Model analysis.
Main effects and anticipated interactions are
tested at 10% level (to increase power).
Level of significance for unanticipated
interactions: 10%, Bonferroni corrected.
7/17/2015
39
Refining Phase: Purpose

Two goals:
- Find correct level (dose) for factor 1, if it is
significant. We considered only two extreme
levels of factor 1 in screening phase!
- Settle confusion about significant unanticipated
aliased interactions, if any.

On the basis of above results, figure out the best
treatment combination.
7/17/2015
40
Refining Phase: Designs


In some cases, we can use the signs (and/or
sizes) of the significant effects to decide.
Can avoid complicated designs.
An Example:
Unanticipated
interactions
Significant effects: 1(+), 2(+), 3(+), 4(+), 5(-), 26=45(-)
Signs of effects are consistent. Can avoid
expensive de-aliasing experiment.
7/17/2015
41
Refining Phase: Designs


In some other cases, we have to run dealiasing or confirmatory experiments. Again
some factorial design!
Unanticipated
An Example:
interactions
Significant effects: 1(+), 2(+), 3(+), 5(-), 23=45(-)
We have to run confirmatory study for 23.
7/17/2015
42
Summary



We proposed a two-phase experimental approach to
find the optimal multi-component program, based on
Fractional Factorial Designs.
This opens the Black Box. Understands the effects of
individual components and their interactions. It also
provides systematic experimental tests of different
components.
Even though it might seem complicated to construct
the designs, it has led to effective but economical
treatments in other domains and can be expected to
do the same for preventive interventions as well.
7/17/2015
43
End Notes



Our proposed method have been developed in
terms of operationalized algorithms coded in
MATLAB.
Preliminary results, based on simulated data,
shows that our method works better than the
classical one in the current setting.
We will have a technical report soon, detailing
the method and the results obtained…
7/17/2015
44
References



Box, G.E.P., Hunter, W.G., and Hunter, J.S. (1978).
Statistics for Experimenters: An introduction to
Design, Data Analysis, and Model Building.
Wu, C.F.J., and Hamada, M. (2000).
Experiments: Planning, Analysis, and Parameter
Design Optimization.
Collins, L.M., Murphy, S.A., Nair, V., and Strecher, V.
(2004). A Strategy for Optimizing and Evaluating
Behavioral Interventions. To appear in Annals of
Behavioral Medicine.
7/17/2015
45