Transcript Slide 1

Class 11 -12
Chapters 5 & Elkins (1989)
Elkin et al: Purpose
 Test feasibility of the collaborative
clinical trial model
 Examine relative efficacy of CBT,
IPT, and Medication for Depression
2
NIMH Treatment of Depression
Collaborative Research Program
 U. of Pittsburg
 George Washington U.
 U. of Oklahoma
 250 Patients: Major depressive disorder
 28 therapists: years experience 2 -27;
 10 psychologists
 18 psychiatrists
71% male
3
Experimental Between-Group Designs
1. Post-Test Only Control
2. Pre-Test -- Post-Test Control
3. Solomon Four Group
(combination of 1 and 2 above)
 Factorial Design

more than one independent variable; interactions
treatment X therapist
or patient characteristic
 Dependent Sample Design (Matching)
4
Experimental Between-Group Designs
1. Post-Test Only Control
2. Pre-Test -- Post-Test Control
3. Solomon Four Group
(combination of 1 and 2 above)
 Factorial Design - Post Hoc

more than one independent variable; interactions

treatment X patient characteristic (depression level at intake)
 Dependent Sample Design (Matching)
5
IVs: Experimental Groups:
 Cognitive
Behavioral Therapy
 Interpersonal Therapy

16 individual sessions/ 50 min.
 Medication
+ Clinical Management*
 Pill-Placebo + Clinical Management*

1st session 55 min.; then 20 to 25 min.
* Minimal supportive therapy condition
6
Dependent Variables
Clinical Evaluator
Self Report
7
Dependent Variables
Clinical Evaluator
• Hamilton Rating Scale
Depression (HRSD)
• Global Assessment
Scale (GAS)
Self Report
• Beck Depression
Inventory (BDI)
• Hopkins Symptom
Checklist (HSCL-90)
8
Outcome Research Strategies
 Primary Analyses
 Secondary Analyses (Post-Hoc)
9
Outcome Research Strategies
 Primary Analyses
 Treatment package
 Comparative
 Secondary Analyses
 Client
Variation -moderation effect?
10
Outcome Research Strategies
 Secondary Analyses
 Client
Variation -moderation effect
depression level at intake as moderator
of relation of treatment groups to outcome
Were outcomes across treatment types different
for patients with higher versus lower levels of
depression at pre-test?
11
Control Groups
 CBT
 IPT
 Medication
+ Clinical Management*
 Pill- Placebo + Clinical Management*
* Minimal supportive therapy condition
12
Treatments & Therapists
Cognitive Behavioral
Therapy
Different group of
experienced therapists
Interpersonal Therapy
Medication + Clinical
Mngmnt
Pill-Placebo + Clinical
Mngmnt
Same therapists psychiatrists
13
Ensure Valid Treatments
 Specify the treatment(s)
 Therapist training/monitoring
 Fidelity Checks
14
Ensure Valid Treatments
 Specify the treatment(s)
 Manuals
 Therapist training/monitoring
 Fidelity Checks- therapy tapes
Collaborative Study Psychotherapy Rating
Scale (CSPRS)
 Treatments could be discriminated 95% of
the time

15
Attrition (>15 sessions or 12 weeks)
Total: 77/239
32%
CBT
IPT
Meds/CM
Placebo/CM
32%
23%
33%
40%
16
Which group to use in outcome analysis??
Total N = 239
Completers
N = 155
15 weeks or
12 sessions
N = 204
At least 3.5 weeks or
4 sessions
End Point
Intent to Treat Group
N = 239
(last assessment or pre-test)
End-Point
17
Assessment Times
Pre treatment
Post Treatment
 4,
8, 12 weeks
 Termination – 15 weeks
 Follow up: 6, 12, 18 months
18
Analyses of Pre-test/Post-test (1)
 Paired T-Test to examine
differences between pre-test
and post-test scores (p. 974)
 How Many ??
19
Table 1 Completer Group:
At least 12 sessions; n=155 (page 975)
20
Analyses of Pre-test/Post-test (1)

Paired T-Test to examine differences between
pre-test and post-test scores (p. 974)
 How Many ??
4 Treatment groups X 4 Outcome measures
CBT
IPT
IMI-CM
Pla-CM
HRSD
GAS
BDI
HSCL-90
X 3 Samples – Completers; End Point 204; 239
21
Findings – T-Tests
22
Analyses of Post-test scores

Use pre-test as a covariate in analyses of
co-variance to compare mean post-test
scores across the 4 treatment groups


Calculate a residualized change scores – amount
of variability in the post-test that is not associated
with the pre-test score
Used a P<.10 in ANCOVAS and
p = .10/.06 = .017 for pair-wise comparisons
(p.974)
23
ANCOVAS: Post test scores
 Statistically significant differences
between groups in scales at post-test

Four 3 X 4 ANCOVAS: differences across
treatments in Post-treatment scores in:
HRSD, GAS --- BDI, HSCL90
 3 (sites) X 4 (treatment groups)
 Analyses reported only for treatment groups
combining them across sites
24
Co-Variates
 Pre-test scores
 Marriage Status (1,2)
 Why
not MANCOVAS? P.973
25
Table 1 Completer Group:
At least 12 sessions; n=155
(page 975) p<.10
26
Table 1 End Point 239 Group
CBT
IPT
IMI-CM
PLA-CM p<.10
27
Findings Pair-wise Comparisons
Sample
Clinical Evaluator
Self-Report
BDI
Pairwise NS
HSCL-90-T p=.006
Completer
N = 155
IMI-CM<PLA-CM
EP-204
GAS
IMI-CM<PLA-CM
(trend p=.020)
EP-239
HRSDep
IPT, IMI-CM<PLA-CM (trend p=.017,.018)
GAS p =.010
IMI-CM<PLA-CM
28
Measuring Change Elkin et al. 1989
 Statistical significance
 Clinical significance
29
Measuring Change Elkin et al. 1989
 Statistical significance
 Differences between groups in scales at
post-test controlling for pre-test scores
 Clinical significance
 Percentage of participants that changed
from dysfunctional to functional level (using
cut-off scores)
30
Clinical Significance
 Recovery Analysis

Proportion of patients who improved vs. not
improved
 Cut Off Scores


Not Depressed HRSD < 6 and BDI < 9
Depressed
HRSD > 6 or BDI > 9
 Statistical Analyses

Chi square: Proportion of depressed and nondepressed patients across treatment groups at
termination.
32
33
End Point 239 HRSD
p = .04
CBT
IPT
IMI-CM
P-CM
Proportion of
cases that met
recovery
criteria
36%(ns)
Proportion of
cases that met
recovery
criteria
43%
Proportion of
cases that
met recovery
criteria
Proportion of
cases that met
recovery criteria
42%
21%
 Chi Square (Χ2) tests to what extent the proportion in each
group is what may be expected by chance or if it is larger or
smaller than expected…….
 IPT = IMI-CM>Placebo-CM
 CBT - % comparison was not sig. for any group
34
Secondary Analyses
 To examine effect of pre-treatment
severity (HRSD/GAS) on outcome by
treatment group
 DVs:
Post-treatment scores
 Severity Criteria
HRSD>20
 GAS<50

 Covariate
44% of sample
41%
Marital Status
35
2X4 ANCOVA (severity x treatment)
DVs- Post Test HRSD, GAS, BDI, HSCL-90
 Main Effect for
 Main Effect for
 (Interaction term)***
36
2X4 ANCOVA (severity x treatment)
DVs- Post Test HRSD, GAS, BDI, HSCL-90
 Main Effect for Severity


More Severe Pre-Test HRSD>20; GAS<50
Less Severe Pre-Test
 Main Effect for Treatment




CBT
IPT
IMI-CM
P-CM
 Severity X Treatment (interaction term)*******
37
Interaction Effect HRSD Severity x TG
Dependent Variables: HRSD* GAS, BDI, HSCL-90 (p.976)
Completer*
Completer S
CBT
BDI
IPT
IPT
IMI-CM
IMI-CM P-CM
P-CM
CBT
IPT
IMI-CM
CBT
IPT
High
HRSD
High
Depression
Low
LowHRSD
Depression
End Point 204*
P-CM
High HRSD
Low HRSD
End Point 239^
IMI-CM P-CM
High HRSD
Low HRSD
4 sets of 3 2X4 Ancovas: 4DVs, 3 sample subgroups *p<.10; ^p<.11
38
Interaction Effect GAS Severity x TG:
Dependent Variables: HRSD GAS, BDI, HSCL-90
Completer S
Completer**
BDI
CBT
IPT
IPT
IMI-CM
IMI-CM P-CM
P-CM
High
HighGAS
Depression
Low
LowGAS
Depression
End Point 204****
CBT
IPT
IMI-CM
P-CM
High GAS
Low GAS
End Point 239*
CBT
IPT
IMI-CM P-CM
High GAS
Low GAS
39
Treatment by Severity Interaction/end-point 204 sample
Higher score Negative Outcome
Higher Score Positive Outcome
40
Summary All Pairwise analyses
following interaction effects p.976
 Less severe groups: no differences across
treatment groups
 More severe groups

IPT more effective than PLA-CM in 3
instances all in the HRSD measure in the END
Point Sample 204 (3 out of 4 comparisons)

IMI-CM more effective than PLA-CM across a
number of measures (8 out 10 comparisons)
41
Figure 2
Recovery Rates (%) endpoint /204 sample
42
Figure 2 Recovery Rates (%) endpoint /204
sample for severity groups (p.977)
 Less severe subgroups: NS differences among
treatments for all samples with HRSD or GAS.
 More severe subgroups for HRSD and GAS:
 Consistent findings across the three samples

IPT>PLA-CM 5/6 and IMI-CM>PLA-CM 6/6
43
Threats to Statistical Conclusion Validity
Are the observed relations among variables accurate?
Power
Unreliability of Measures
Unreliability of Treatment
Implementation
Extraneous Variance in the
Experimental Setting
Heterogeneity of
Participants
44
Threats to Statistical Conclusion Validity
Are the observed relations among variables accurate?
•
•
•
•
Large N by group range 34-62 +
Outcome measures are well-known +
Power analyses 81-95% for medium effects +
p<.10 for Mancovas and .10/6 for pairwise comp
Unreliability of Treatment
Implementation
•
•
•
•
Experienced Therapists – 2-27yrs Mean = 11 +
Manuals, training per treatment group +
Closely monitored +
Taped sessions – 95% correctly classified +
Extraneous Variance in the
Experimental Setting
•
•
•
•
Not known for the most part 28 therapists from 3 – 11 patients each no way to control for therapist effects P. 980 one site CBT another site IPT similar to
Meds/CM
Heterogeneity of
Participants
•
•
•
•
Random assignment to groups +
Only included 45% of those screened. +
Mostly women 70% female +
89% white participants +
Power
Unreliability of Measures
45
Threats to Internal Validity
Can we conclude that there is a causal relation between the IV and
the DV? Did treatment cause differences in DV across groups?
Selection
Who gets assigned to which group?
History
Attrition
What do we know about drop-outs?
Repeated Testing
Effects
Reaction to
Control Group
Assignment
46
Threats to Internal Validity
Can we conclude that there is a causal relation between the IV
and the DV?
Selection
Used RandomizationSee factors under Heterogeneity of Participants
History
Time frame of study not reported
Did therapy happen at about the same time for
everyone?
Attrition
Relatively high attrition rates - 32% -- about 25%
was for negative reasons related to treatment- (-)
Early terminators were more depressed at intake (-)
Repeated Testing
Effects
Tested at frequent intervals –’ pre-test, 4, 8, 12,
weeks, termination 6 12 and 18 months follow-up
Reaction to
Control Group
Assignment
Not known – but could be the case.
Placebo/CM experienced the highest attrition –
32% CBT—23% IPT – 33% Meds/CM -- 40%
Placebo/CM
47
Threats to Construct Validity
To what extent variables capture desired constructs
Mono-Operation Bias
(Instruments)
Mono-Method Bias
Experimenter
Expectancies
48
Threats to Construct Validity
To what extent variables capture desired constructs
Mono-Operation Bias
(Instruments)
• Used 4 different outcome measures
HIRSD, BDI, GAS, HCSL-90 +
• Measures of well-known psychometric
properties +
Mono-Method Bias
• Used both patient self report and clinician
completed measures +
• Measures of well-known psychometric
properties +
Experimenter
Expectancies
• Clinicians not blind to therapy modality• Psychiatrist blind to Med condition +
49
Threats to External Validity
Can we generalize observed relations across persons,
settings and times
Person-Units
Outcome Measures
Settings
50
Threats to External Validity
Can we generalize observed relations across persons,
settings and times
Person-Units
Outcome Measures
Settings
• Highly selected sample (-)
• Only 45% screened were selected (-)
• Generalizable to white (89%) women (70%)
highly educated (75% coll degree or some coll)
who were less severely depressed (p.974)
• Interview and self –report measures +
• Clinical significance recovery rates +
• Statistically significant findings were not
consistent across measures – HRSD detected
more differences in depression that BDI Empirical Question ????
51
Results: Summary 1/3
 Paired T test showed stat. sig. differences
(p<.001) in Pre- Post scores in all measures
for all three groups of participants (even
placebo pill/CM)


Intent-to treat
Completers Minimum


3.5<Sessions
Completers of all or most sessions

At least 12<sessions > 15 (n=155)
52
Results: Summary 2/3
 ANCOVAS showed
 no stat sig differences in pre-test scores in
any measure for any treatment group
 Stat
sig differences in post-test
BDI/HSCL90
 HSRD/GAS

Completers
Total Group (239)
Results: Summary 3/3
 Pairwise Follow-up ANCOVA

HSCL-90 IMI-CM> PLA-CM (Completer)

GAS --
IMI-CM>PLA-CM (Total 239 group)

HRSD
IPT, IMI-CM>trend PLA-CM (Total 239)
 Recovery Findings (Clinical Significance)


IPT, IMI-CM > PLA-CM ( End-Point 239)
43% 42%
21%
Post-test HRSD<6
CBT = 36% NS
54