Transcript Document
RESEARCH DESIGN
PHC 6700/RCS 6740
March 7, 2006
1
RESEARCH DESIGN
Experimental Designs
The specific research designs available to investigators
can be divided into two basic types:
group designs, and
single-subject designs.
2
RESEARCH DESIGN
A Typical Experimental Design
Pretest-Posttest Control Group Design
R O1
R O3
X
O2
O4
3
RESEARCH DESIGN
Group Designs
The group (multi-subject) designs all include one or more
groups of subjects and are classified as either:
between-groups,
within-subjects,
or mixed.
4
RESEARCH DESIGN
Between-Groups Design
Between-groups design is used to assess the effects of
different levels of an independent variable by
administering each level to a different group of subjects
and then comparing the status or performance of the
group on the dependent variable.
The simplest between-groups designs include a single
independent variable with two levels. When using the design, the
study includes two groups that each receives a different level of
the IV.
5
RESEARCH DESIGN
Example:
A psychologist assesses the effects of a “self-control”
procedure by comparing the achievement of children who
have been trained in the procedure (experimental group)
with that of children who have not been trained in the
procedure (control group).
6
RESEARCH DESIGN
The simple two-group design can be expanded in two ways. One way
is to include more than two levels of a single independent variable.
The psychologist in this study could compare three levels of the
control procedure
a procedure that includes self-instruction only,
a procedure that includes self-instruction, self-monitoring, and selfreinforcement, and
no procedure.
In this situation, the study would involve comparing the average
academic achievement test scores of subjects in the three groups.
7
RESEARCH DESIGN
Another way to expand a two-group design is to include
two or more independent variables.
Whenever a study includes two or more independent
variables, it is called a factorial design.
The major advantage of a factorial design is that it
provides more thorough information about the
relationships among variables by allowing an investigator
to analyze the main effects of each independent variable
as well as the interaction between independent variables.
8
RESEARCH DESIGN
If the psychologist in the self-control study included initial
symptom severity (mild, moderate, and severe) as a
second independent variable, s/he would be able to
determine if there are:
MAIN effects of the self-control procedure,
MAIN effects of initial symptom severity, and/or
an INTERACTION between self-control procedure and initial
symptom severity.
9
RESEARCH DESIGN
Main Effect
A main effect is the effect of one independent variable on
the dependent variable, disregarding the effects of all
other independent variables.
An interaction refers to the effects of two or more
independent variables considered together.
An interaction occurs when the effects of an independent variable
differ at different levels of another independent variable.
10
RESEARCH DESIGN
Illustration:
Assume the psychologist in the self-control study obtains a
sample of 60 children and divides them into three groups
on the basis of their initial symptom severity (mild,
moderate, or severe). S/he then randomly assigns
subjects in each group to either the experimental (selfcontrol procedure) or control (no procedure) group so that
there are 10 children in each of the study's now six groups
(see table in next slide)
Although the data collected by the psychologist would have to be
analyzed with an inferential statistical test to determine if there are
significant main and/or interaction effects, tentative conclusions
can be drawn by examining the data.
11
RESEARCH DESIGN
As an example, assume that the psychologist obtains the following
mean achievement test scores for the six groups of children:
Self-Control
Procedure
No Procedure
Mild Symptoms
52
36
Moderate Symptoms
40
30
Severe Symptoms
34
36
12
RESEARCH DESIGN
To determine if there are main effects of each IV, it is
necessary to calculate the marginal means. For selfcontrol procedure, the marginal means are 42 and 34.
These means were obtained by adding the means in each
column and dividing by 3 (the number of means): (52 + 40
+ 34)/3 = 42 and (36 + 30 + 36)/3 = 34.
Because the marginal means are different, it is possible to
tentatively conclude that there are main effects for the self-control
procedure.
Overall, the self-control procedure seems to have beneficial effects
on academic achievement test scores.
13
RESEARCH DESIGN
For initial symptom severity the marginal means are 44, 35,
and 35. These means were obtained by adding the means
in each row and dividing by 2 (the number of means): (52 +
36)/2 = 44; (40 + 30)/2 = 35; and (34 + 36)/2 = 35.
The marginal means indicate that there are also main effects for
initial symptom severity.
Although children with moderate and severe symptoms obtained
the same mean achievement test score (35), children with mild
symptoms obtained a higher mean score (44). (If all three means
were identical, there would be no main effects of symptom
severity.)
This indicates that, overall, mild symptoms are associated with the
highest achievement test scores.
14
RESEARCH DESIGN
Self-Control
Procedure
No Procedure
Mild Symptoms
52
36
44
Moderate Symptoms
40
30
35
Severe Symptoms
34
36
35
42
34
15
RESEARCH DESIGN
To determine if there is an interaction, the cell means are inspected. If
there is an interaction, the effects of the self-control procedure will
differ at different levels of symptom severity.
As can be seen in the previous table, for children with mild symptoms, the
self-control procedure had a very positive effect:
For children with moderate symptoms, the self-control procedure also had
positive effects, but the gap between the two groups is somewhat smaller:
Children who were trained in the procedure obtained a mean of 52,
while children who received no training obtained a mean of 36.
Children who received training obtained a mean of 40,
while children who received no training obtained a mean of 30.
Finally, for children with severe symptoms, the self-control procedure did
not have a beneficial effect.
Children with severe symptoms obtained a higher achievement test score if
they received no training (34 with training versus 36 with no training).
16
RESEARCH DESIGN
These results suggest that there is an interaction between
the two IVs. The effects of the self-control procedure differ
for different levels of initial symptom severity. (If the effects
of the self-control procedure were the identical for all
levels of symptom severity, there would be no interaction.)
17
RESEARCH DESIGN
An inspection of the data suggests that there are main
effects for both variables as well as an interaction effect.
Note that the presence of the interaction invalidates the conclusion
that was drawn on the basis of the main effects alone:
Overall, the self-control procedure seems to be beneficial (main
effect), but a closer inspection of the data shows that this is true only
for children with mild and moderate symptoms (interaction).
This illustrates the importance of interpreting the main effects (or
lack of main effects) with caution whenever there is an interaction:
An interaction is likely to modify or invalidate the conclusions made on
the basis of the main effects alone.
Finally, note that a study can have any combination of main and
interaction effects. It is possible to have an interaction, for example,
without any main effects (or vice versa).
18
RESEARCH DESIGN
Study Tip: Be sure you understand what main and
interaction effects are on a conceptual level and be
able to determine, from a table of data like the one
given, whether it looks like there are main and/or
interaction effects. Keep in mind that there can't be an
interaction unless the study has at least two IVs and
that, when there is an interaction, the main effects
must be interpreted in light of the interaction.
19
RESEARCH DESIGN
Within-Subjects Designs
When using a within-subjects (AKA: repeated measures)
design, all levels of the independent variable are
administered sequentially to all subjects.
Consequently, comparisons of different levels of the independent
variable are made within subjects rather than between groups of
subjects.
Like between-groups designs, within-subjects designs can include
only two levels of a single independent variable or can be
expanded to include three or more levels of a single IV and/or two
or more IVs.
20
RESEARCH DESIGN
The single-group time-series design (AKA the simple
interrupted time-series design) is one type of withinsubjects design.
When using this design, the effects of a treatment are evaluated
by measuring the dependent variable several times at regular
intervals both before and after the treatment is applied.
This procedure allows subjects to act as their own no-treatment
controls.
21
RESEARCH DESIGN
Example: A psychologist in a study assesses the
effects of a low dose of phenothiazines on Brief
Psychiatric Rating Scale (BPRS) scores by
administering the BPRS to a single group of
patients at one week intervals for two months
before and two months after patients begin
receiving a low dose of the drug.
22
RESEARCH DESIGN
A shortcoming of the single-group time-series design is
that its internal validity can be threatened by history:
It's possible that an external event could occur at about the same
time the independent variable is applied and account for any
observed difference in pre- and posttest scores.
Note, however, that this design does help control maturation since
maturational effects tend to occur gradually over time and can
usually be detected in the overall pattern of pre- and posttest
scores.
23
RESEARCH DESIGN
In another type of within-subjects design, two or more
levels of an independent variable are applied sequentially
to each subject, and the dependent variable is measured
after each level has been applied.
Example: A psychologist compares the effects of a low and
high dose of phenothiazines on BPRS scores by giving the
low dose of the drug to a group of patients for two months
and administering the BPRS; and then giving the same
patients the high dose of phenothiazines for two months and
administering the BPRS again. The effects of the
phenothiazines will be analyzed by comparing the BPRS
scores obtained by the patients after each dose was
administered.
24
RESEARCH DESIGN
A problem with this design is that it is susceptible to
carryover effects (multiple treatment interference):
If the psychologist this study finds that the high dose of
phenothiazines is more effective than the low dose for improving
symptoms, this could be because the high dose followed a period
of time during which patients received the low dose.
Counterbalancing can be used to control carryover effects.
Carryover effects could be controlled in the above study by dividing
the patients into two groups and administering the two levels of
phenothiazines in a different order to each group (i.e., low dose, high
dose to patients in Group # l and high dose, low dose to patients in
Group #2). If the high dose is more effective for both groups of
patients, its effects would not be attributable to carryover effects.
25
RESEARCH DESIGN
Within-subjects designs have at least two advantages over
between groups-designs.
First, they require a fewer number of subjects and, consequently,
are more "economical."
Second, these designs help control individual differences that can
threaten a study's internal validity since subjects are compared
with themselves rather than with other subjects.
As a result, within-subjects designs can actually be more
powerful than between groups-designs (i.e., better able to
detect the actual effects of the IV).
26
RESEARCH DESIGN
A disadvantage of the time-series and other
within-subjects designs is that the analysis of the
data can be confounded by autocorrelation (also
known as serial dependency).
In other words, subjects' performance on the post-tests
is likely to correlate with their performance on the
pretests.
Autocorrelation can inflate the value of the inferential
statistic (e.g., the t or F), thereby resulting in an
increased probability of a Type I error.
For this reason, a number of experts recommend that
special statistical techniques be used to analyze data
collected in a study using this type of design.
27
RESEARCH DESIGN
Study Tip: Be sure you have this design linked with
the concept of autocorrelation and know that
autocorrelation can decrease power and inflate the
chance of making a Type I error.
28
RESEARCH DESIGN
Mixed Designs
A mixed ("split-plot") design combines betweengroups and within-subjects methodologies.
Counterbalanced designs can be considered a type of
mixed design because they permit comparisons both
between groups and within subjects.
A design is also a mixed design when it includes two or
more independent variables and at least one variable is
a between-groups variable and another is a withinsubjects variable.
29
RESEARCH DESIGN
Example: In the example study, the psychologist
would be using a mixed design if therapy
approach is treated as a between-groups variable
(patients receive only one type of therapy), while
phenothiazines is treated as a within-subjects
variable (the placebo, low dose, and high dose
are administered sequentially to each patient).
30
RESEARCH DESIGN
Mixed designs are common in research studies that
involve measuring the dependent variable over time or
across trials.
In this type of study, time or trials is an additional IV and is
considered a within-subjects variable because
comparisons on the dependent variable will be made
within subjects across time or across trials.
31
RESEARCH DESIGN
Example: In our example study, the psychologist
decides to compare the effects of four levels of
therapy (family therapy, individual therapy, a
combination of the two, and no therapy) by
assigning patients to one of the levels and
measuring the short- and long-term effects of
therapy by administering the BPRS at two-month
intervals for 24 months after therapy begins.
Because the study includes a between-groups
variable (therapy) and a within-subjects variable
(time), it is utilizing a mixed design.
32
RESEARCH DESIGN
Single-Subject Designs
The single-subject designs were derived primarily from the
work of behavioral psychologists, especially those
engaged in applied behavioral analysis, which combines
behavioral principles with the techniques of experimental
psychology to solve socially-relevant problems.
While the single-subject designs are often used to
investigate the effects of an independent variable on the
behavior of one subject or a small number of subjects,
they can also be used with groups of subjects.
33
RESEARCH DESIGN
1.
2.
Several characteristics distinguish the single-subject
designs from the group designs:
Each single-subject design includes at least one baseline
(no treatment) phase and one treatment phase. As a
result, each subject acts as his/her own no-treatment
control.
The dependent variable is measured repeatedly at
regular intervals throughout the baseline and treatment
phases. Repeated measurement of the DV helps control
any maturational effects that might otherwise threaten
the study's internal validity by enabling an investigator to
detect those effects in the pattern of performance on the
DV measure.
34
RESEARCH DESIGN
There are a large number of single-subject designs,
but the most commonly used are the AB design (and
its extensions) and the multiple baseline design.
AB Design
Reversal (Withdrawal) Designs
Multiple Baseline Designs
35
RESEARCH DESIGN
AB Design
The simplest single-subject design is the AB design,
which includes a single baseline (A) phase and a
single treatment (B) phase. As in all single-subject
designs, the dependent variable is measured at
regular intervals during both phases.
36
RESEARCH DESIGN
Example: A psychologist decides to assess the effects of a
low dose of phenothiazines on the BPRS scores of one patient
using the AB design. She administers the BPRS to the patient
at two-week intervals for three months during the baseline (A)
phase of the study and for three months during the treatment
(B) phase. The design of this study and its possible results are
illustrated below:
A
B
H
L
Time
37
RESEARCH DESIGN
Because the patient's symptoms decreased only
after he began receiving the phenothiazines,
these results suggest that a low dose of
phenothiazines has a positive effect on
symptoms.
38
RESEARCH DESIGN
Reversal Designs (ABA, ABAB, Etc.)
The AB design can be expanded to include more than
one baseline phase or more than one baseline and
more than one treatment phase. Because any
expansion requires the withdrawal of the treatment
during the second and subsequent baseline phases,
the extensions of the AB design are called reversal
(withdrawal) designs.
39
RESEARCH DESIGN
•
Advantage of the reversal designs over the simple AB
design: they provide additional control over potential
threats to a study's internal validity.
•
When an ABAB design is used, if status on the DV
returns to the initial baseline (no treatment) level during
the second A phase and then to its previous treatment
levels during the second B phase, an investigator can be
more certain that any observed change in the dependent
variable is actually due to the IV rather than to an historical
event or other extraneous factor.
•
Repeating the study by adding another A and B phase
is referred to as intrasubject replication.
40
RESEARCH DESIGN
Example: To confirm that the observed improvement in symptoms
was due to the phenothiazines, the psychologist extends her study
to include an additional baseline and treatment phase. This ABAB
design and its possible results are illustrated below:
A
B
A
H
H
L
L
B
Time
41
RESEARCH DESIGN
•
Reversal designs are considered inappropriate when
withdrawal of a treatment during the course of a research study
would be unethical (e.g., when the treatment has successfully
eliminated a self-injurious behavior).
•
Reversal design does not provide conclusive information if
the effects of an independent variable persist rather than
"reverse" (return to baseline levels) when it is withdrawn.
•
When this occurs, an investigator can't be certain whether
an observed effect on the dependent variable is due to the
independent variable or to other factors.
•
If a reversal design is inappropriate for ethical or practical
reasons, an investigator might use a multiple baseline design.
42
RESEARCH DESIGN
Multiple Baseline Design
•
Multiple baseline design does not require withdrawal of a
treatment during the course of the study but, instead, involves
sequentially applying the treatment either to different behaviors
of the same subject (multiple baseline across behaviors); to the
same subject in different settings (multiple baseline across
settings); or to the same behavior of different subjects (multiple
baseline across subjects).
• Once the treatment has been applied to a "baseline" (behavior,
setting, or subject), it is not withdrawn from that baseline during
the course of the study.
43
RESEARCH DESIGN
Example: To test the effects of the self-instructional
component of the self-control procedure on attention
span, a psychologist uses a multiple baseline design.
S/he trains a child in self-instruction and then tells the
child to use the technique when working on arithmetic
homework in three different settings: first when working
alone in a quiet room; then when working in the library;
and then when working in the classroom. The
psychologist measures the child's attention span in all
three settings at regular intervals during the baseline
and treatment phases.
44
RESEARCH DESIGN
This design and its possible results are illustrated below
A
B
Quiet Room
Library
Time
45
RESEARCH DESIGN
Because the child's attention span in each setting did
not increase until self-instruction was applied to it,
these results confirm that self-instruction is useful for
improving attention span.
As can be seen in the above figure, sequentially
applying an intervention to different settings helps
determine if an intervention is actually responsible for
any observed changes in the target behavior:
46
RESEARCH DESIGN
•
If the behavior changes in a particular setting
only after the intervention has been applied in that
setting, an investigator can be more certain that
the change is due to the intervention rather than
to history or other factors.
•
To be effective, the setting, behaviors, or
subjects chosen for inclusion in the study must be
relatively independent. If they are not, it may not
be possible to evaluate the effects of the IV with
the multiple baseline design.
47
HAPPY SPRING BREAK!
48