Transcript Document

Chapter 22
Quasi-Experimental And N=1
Designs OF Research
• In earlier chapters we stated and emphasized that
one of the major goals of science is to find causal
relations. In the behavioral sciences, the true
experiment is the strongest approach used to
meet this goal.
• However, there are research problems in the
behavioral sciences and especially educational
research that cannot be studied using a true
experimental design. We will examine two
research designs where one or more of the
components of the true experiment have been
compromised. The first is called quasi
experimental designs and the second is called
single subject or N=1 designs.
Compromise Designs a.k.a. QuasiExperimental Designs
• Recall that true experimentation requires
at least two groups, one receiving an
experimental treatment and one not
receiving the treatment, or receiving it in
different form. The true experiment
requires the manipulation of at least one
independent variable, the random
assignment of participants to groups, and
the random assignment of treatments to
groups.
Compromise Designs a.k.a. QuasiExperimental Designs
• Cook and Campbell (1979) present two
major classifications of quasi-experimental
design.
• The first is called the ”nonequivalent
control group designs,” the second is the
“interrupted time series designs.”
Nonequivalent Control Group
Designs
•
•
•
•
•
No-treatment control group designs
Nonequivalent dependent variables designs
Removed treatment group designs
Repeated treatment designs
Reversed treatment nonequivalent control
group designs
• Posttest only designs
• Regression continuity designs
No-Treatment Control Group
Designs
• Design 22.1
• An effort should be made to at least use
samples from the same population, or samples
that are as alike as possible. The experimental
treatments should be assigned at random. Then
the similarity of the groups should be checked
using any information available (sex, age, social
class, and so on). The equivalence of the groups
could be verified using the means and standard
deviations of the pretests: t-test and F-test will
do.
No-Treatment Control Group
Designs
• There are still difficulties, all of which are
subordinate to one main difficulty—selection.
When participants are selected into groups on
bases extraneous to the research purposes, we
call this “selection” or alternatively, “selfselection”
• Note that if we had used only volunteers and
had assigned them to experimental and control
groups at random, the selection difficulty is
lessened. External validity or representativeness,
however, would be decreased.
No-Treatment Control Group
Designs
• Without the benefit of random assignment,
attempts should be made through other
means to eliminate rival hypotheses. We
consider only the design that uses the
pretest because the pretest could provide
useful information concerning the
effectiveness of the independent variable
on the dependent variable.
No-Treatment Control Group Designs
• Another more frequent example in educational
research is to take some school, classes for the
experimental group and others for the control group.
If a fairly large number of classes are selected and
assigned at random to experimental and control
groups, there is no great problem.
• But if they are not assigned at random, certain ones
may select themselves into the experimental
groups, and these classes may have characteristics
that predispose them to have higher mean Y scores
than the other classes.
No-Treatment Control Group
Designs
• In other words, something that influences
the selection process (e.g., volunteer
participants), also influences the
dependent variable measures. This occurs
even though the pretest may show the
groups to be the same (alike) on the
dependent variable. The X manipulation is
“effective” because of selection, or selfselection, but it is not effective in and of
itself.
No-Treatment Control Group
Designs
• Possible outcomes from this design are
given in Figure 22.1. There is the
possibility of a different interpretation on
causality depending on which outcome the
researcher obtains. In almost all of the
cases the most likely threat to internal
validity would be the selection-maturation
interaction.
No-Treatment Control Group
Designs
• You might recall that this interaction occurs
when (1) two groups are different to begin
with as measured by the pretest; then (2)
one of the groups experience greater
differential changes, such as getting more
experienced, more accurate, more tired,
and so on, than the other group. The aftertreatment difference, as observed in the
posttest, can not exactly be attributed to
the treatment itself.
No-Treatment Control Group
Designs
• There are four alternative explanations to
the outcome in Figure 22.1(a)
• The first is selection-maturation interaction.
Group E’s increase may be due to their
higher level of intelligence. With a higher
level of intelligence, these participants can
process more, or grow faster, than Group
C.
No-Treatment Control Group
Designs
• A second explanation is one of
instrumentation. The scale used to measure
the dependent variable may be more
sensitive at certain levels than others. In a
normal distribution, changes in raw scores
near the center of the distribution reflect
bigger percentile changes than at the tails.
No-Treatment Control Group
Designs
• The third explanation is statistical
regression. The increase in scores by
Group E would be due to their selection on
the basis of extreme scores. On the
posttest, their scores would go up because
they would be approaching the population
baseline.
• The fourth explanation centers on the
interaction between history and selection.
No-Treatment Control Group Designs
• All of the threats mentioned for Figure 22.1(a)
are also true for Figure 22.1(b). To determine
if selection-maturation is playing a main role
in the results, Cook and Campbell (1979)
recommend two methods. The first method
involves looking only at the data for the
experimental group (Group E). If the withingroup variance for the posttest is
considerably greater than the within-group
variance of the pretest, then there is evidence
of a selection-maturation explanation.
No-Treatment Control Group Designs
• The second method is to develop two plots
and the regression line associated with
each plot. Figure 22.2
• If the regression line slopes for each plot
differ from each other, then there is
evidence of a differential average growth
rate, meaning that there is the likelihood of
a selection-maturation interaction.
No-Treatment Control Group Designs
• The outcome shown in Figure 22.1(c) is
more commonly found in clinical
psychology studies. The treatment is
intended to lead to a decline of an
undesired behavior.
• This outcome is also susceptible to
selection-maturation interaction and three
others.
No-Treatment Control Group Designs
• The fourth outcome is shown in Figure
22.1(d). The selection-maturation threat can
be ruled out since this effect usually results in
a slower growth rate for low scores and a
faster growth rate for high scores.
• The final outcome is shown in Figure 22.1(e).
The four threats can be ruled out. Hence, the
outcome in Figure 22.1(e) seems to be the
strongest one and should enable the
researcher to make a causal statement
concerning treatment.
Research Examples
• Nelson, Hall, and Walsh-Bowers (1997):
Nonequivalent Control Group Design.
• They were unable to assign participants to
different housing settings randomly.
• They state that the difference they found
between these three groups on posttest
measures could have been due to the
selection problem, and not the type of care
facility.
Research Examples
• Chapman and McCauley (1993): QuasiExperiment
• Although one can perhaps think of this study as
a nonexperimental one, Chapman and
McCauley felt that it came under the
classification of quasi-experimental.
• Awards were given to approximately half of a
homogeneous group of applicants in a
procedure that Chapman and McCauley say
approximates random assignment to either
fellowship or honorable mention.
Time Designs
• Design 22.2: A longitudinal Time Design
(a.k.a. Interrupted Time Series Design)
• The reactive effect should show itself by
comparing Y3 to Y4; this can be contrasted
with Y5. If there is an increase at Y5 over
and above the increase at Y4 from Y3, it can
be attributed to X. A similar argument
applies for maturation and history.
Time Designs
• One difficulty with longitudinal or time
studies, especially with children, is the
growth or learning that occurs naturally
over time: Children do not stop growing
and learning for research convenience.
The longer the time period, the greater the
problem. In other words, time itself is a
variable.
Time Designs
• The most widely used statistical test is ARIMA
(autoregressive, integrated, moving average)
developed by Box and Jenkins (1970). The use
of such a statistical analysis requires the
availability of many data points.
• The usual tests of significance applied to time
measures can yield spurious results. One
reason is that such data tend to be highly
variable, and it is as easy to misinterpret
changes not due to X as due to X.
Multiple Time Series Design
• Design 22.3
• This design has the advantage of
eliminating the history effect by including a
control group comprised of an
equivalent—or at least comparable—
group of participants who do not receives
the treatment condition. Consequently, the
design offers a greater degree of control
over sources of alternative explanations or
rival hypotheses.
Single Subject Experimental
Designs
• The majority of today’s behavioral
research involves using groups of
participants. However, there are other
approaches.
• The single-subject designs are sometimes
referred to as the N=1 design. They are an
extension of the interrupted time series
design. Where the interrupted time series
generally looks at a group of individuals
over time.
Single Subject Experimental
Designs
• Common characteristics:
• Only one or a few participants are used in
the study.
• Each subject participants in a number of
trials (repeated measures).
• Randomization procedures are hardly ever
used.
Single Subject Experimental
Designs
• These design observe the organism’s
behavior before the experimental
treatment and use the observations as a
baseline measure. Observations taken
after the treatment are then compared to
the baseline observations. The participant
serves as his or her own control.
Single Subject Experimental Designs
• Behavioral scientists doing research before
the development of modern statistics
attempted to solve the problem of reliability
and validity by making extensive observations
and frequent replication of results. This is a
traditional procedure used by researchers
doing single-subject experiments.
• The assumption is that individual participants
are essentially equivalent and that one should
study additional participants only to make
certain that the original subject was within the
norm.
Single Subject Experimental
Designs
• The single-subject approach assumes that
the variance in the subject’s behavior is
dictated by the situation. As a result, this
variance can be removed through careful
experimental control.
• The group difference research attitude
assumes that the bulk of the variability is
inherent and can be controlled and
analyzed statistically.
Some Advantages of Doing SingleSubject Studies
• In Figure 22.3, if group-oriented research
is employed, two groups have the same
means and measures of variability. But
visual inspection for the data shows a
trend pattern vs. a random pattern. The
single-subject approach does not have this
problem, because a participant is studied
extensively over time. The cumulative
record for that participant shows the actual
performance of the participant.
Some Advantages of Doing Single-Subject
Studies
• Statistical significance and practical significance are
two different things. The experiment may have little
practical significance even if it has plenty of
statistical significance.
• Simon (1987) advocates using well-constructed
designs with the number of participants necessary
to find the strongest effects. Single-subject
researcher, on the other hand, favor increasing the
size of the effect rather than attempting to lower
error variance. They feel that this can be done
through tighter control over the experiment.
Some Advantages of Doing SingleSubject Studies
• With single-subject studies, the researcher
can avoid some of the ethical problems
that face group-oriented researchers. One
such ethical problem concerns the control
group, which does not receive any real
treatment.
• If there are not enough participants of a
certain characteristic available for study,
the researcher can consider single-subject
designs instead of abandoning the study.
Some Disadvantages of Doing
Single-Subject Studies
• One of the more general problems with the
single-subject paradigm is external validity.
Some find it difficult to believe that the
findings from one study using one subject
can be generalized to an entire population.
• With repeated trials on one participant,
one can question whether the treatment
would be equally effective for a participant
who has not experienced previous
treatments.
Some Disadvantages of Doing SingleSubject Studies
• Single-subject studies are perhaps even
more sensitive to aberrations on the part
of the experimenter and participant. These
studies are effective only if the researcher
can avoid biases and the participant is
motivated and cooperative.
• A researcher doing single-subject research
could be affected more so than the grouporiented researcher and needs to develop
a system of checks and balances to avoid
this pitfall.
Some Single-Subject Research
Paradigms
• The Stable Baseline: An Important Goal
• The behavior before the treatment
intervention must be measured over a long
enough time period so that a stable baseline
can be obtained. This baseline, or operant
level, is important because it is compared to
later behavior.
• If the baseline varies considerably, it could
be more difficult to assess any reliable
change in behavior following intervention.
Designs that Use the Withdrawal of
Treatment
• The ABA Design
• The ABA design involves three major steps.
The first step is to establish a stable baseline
(A). The experimental intervention is applied
to the participant in the second step (B). If
the treatment is effective, there will be a
response difference from the baseline. In
order to determine if the treatment
intervention caused the change in behavior,
the researcher exercises step three: a return
to baseline (A).
Designs that Use the Withdrawal of
Treatment
• There are also some ethical concerns
about reverting the organism back to the
original state if that state was an
undesirable behavior. Experiments in
behavior modification seldom return the
participant back to baseline. To benefit the
participant, the treatment is reintroduced.
The ABAB design does this.
Designs that Use the Withdrawal of
Treatment
• Repeating Treatments (ABAB Designs)
• There are two versions of the ABAB
design. The first was briefly described in
the above section. Repeating the
treatment also provides the experimenter
with additional information about the
strength of the treatment intervention.
• The ABAB design essentially produces the
experimental effect twice.
Designs that Use the Withdrawal of
Treatment
• The second variation of the ABAB design is called
the alternating treatments design. In this variation
there is no baseline taken. The A and B in this
design are two different treatments that are
alternated at random. The goal of this design is to
evaluate the relative effectiveness of the two
treatment interventions.
• The advantage this design has over the first ABAB
design is that there is no baseline to be taken, and
the participant is not subjected to withdrawal
procedures. Since this method involves comparing
two sets of series of data, some have called it the
between-series design.
Designs that Use the Withdrawal of
Treatment
• There are some other interesting
variations of the ABAB design where
withdrawal of the treatment is not done.
McGuigan (1996) calls it the ABCB design
where in the third phase, the organism is
given a “placebo” condition. This placebo
condition is essentially a different method.
A Research Example
• Powell and Nelson (1997): Example of an
ABAB Design
• The treatment intervention was letting
Evan choose the class assignment he
wanted to work on. There are two
conditions: choice and no-choice. Baseline
data were collected during the no-choice
phase.
Using Multiple Baselines
• There is a form of single-subject research
that uses more than one baseline. Several
different baselines are established before
treatment is given to the participant. These
types of studies are called multiple baseline
studies.
• There are three classes of multiple baseline
research designs: across behaviors, across
participants, and across environments.
Using Multiple Baselines
• There is a common pattern for
implementing all three classes of this
design. That pattern is shown in Figure
22.4.
Using Multiple Baselines
• With the multiple baselines across behaviors,
the treatment intervention for each different
behavior is introduced at different times. If
one of the behavior changes, while the other
behaviors remain constant or stable at the
baseline, the researcher could state that the
treatment was effective for specific behavior.
• After a certain period of time has passed, the
same treatment is applied to the second
undesirable behavior (Baseline 2).
Using Multiple Baselines
• An important consideration with this
particular class of multiple baseline design
is that one assumes the responses for
each behavior are independent of the
responses for other behaviors.
• The intervention can be considered
effective if this independence exists. If the
responses are in some way correlated,
then the interpretation of the results
becomes more difficult.
Using Multiple Baselines
• In the multiple baseline design across
participants, the same treatment is applied
in series to the same behavior of different
individuals in the same environment.
• In the multiple baseline design across
environments, the same treatment is given
to different participants who are in different
environment.