Transcript Lecture 4

EMR 6550:
Experimental and QuasiExperimental Designs
Dr. Chris L. S. Coryn
Kristin A. Hobson
Fall 2013
Agenda
• Basic design elements and notation
• Quasi-experimental designs that
either lack a control group or lack
pretest observations on the outcome
• Midterm examination
• Case study
Questions to Consider
• What are the limitations of designs
lacking either control groups and/or
pretest observations?
• What simple strategies can be used
to improve these types of designs?
• Why are such designs sometimes the
only ones that can be used?
Basic Design Elements and
Notation
Assignment
•
•
•
•
•
Random assignment
Cutoff-based assignment
Other nonrandom assignment
Matching and stratifying
Masking
Measurement
• Posttest observations
– Single posttests
– Nonequivalent dependent variables
– Multiple substantive posttests
• Pretest observations
–
–
–
–
–
Single pretest
Retrospective pretest
Proxy pretest
Repeated pretests over time
Pretests on independent samples
• Moderator variable with predicted interaction
• Measuring threats to validity
Comparison Groups
•
•
•
•
•
Single nonequivalent groups
Multiple nonequivalent groups
Cohorts
Internal versus external controls
Constructed contrasts
– Regression extrapolation contrasts
– Normed contrasts
– Secondary data contrasts
Treatments
•
•
•
•
Switching replications
Reversed treatments
Removed treatments
Repeated treatments
Notation
X
O
R
NR
X
X+
=
=
=
=
=
=
treatment
observation
random assignment
nonrandom assignment
removed treatment
treatment expected to produce an
effect in one direction
X= conceptually opposite treatment
expected to reverse an effect
C
= cutting score
- - - = non-randomly formed groups
…
= cohort
Logic of Quasi-Experimentation
Rationale
• Quasi-experiments are often a necessity given
practical and logistical constraints
– Greater emphasis on construct or external validity
rather than cause-effect associations (least
common)
– Funding, ethics, administration (somewhat
common)
– The intervention has already occurred (most
common)
• Sometimes they are the best alternative, even
if causal inferences are weaker than is possible
with other designs
• Even so, great care must be taken when
planning such studies as numerous threats that
cannot be controlled are often operating
Central Principles
• Identification and study of plausible threats
to internal validity
– Careful scrutiny of plausible alternative
explanations for treatment-outcome
covariation
• Primacy of control by design
– Use carefully planned and implemented design
elements rather than statistical controls for
anticipated confounds
• Coherent pattern matching
– Complex (a priori) causal hypotheses that
reduce the plausibility of alternative
explanations
Designs without Control
Groups
One-Group Posttest Only
Design
X
O1
• Absence of pretest makes it difficult
to know if change has occurred
• Absence of a control group makes it
difficult to know what would have
happened without treatment
One-Group Pretest-Posttest
Design
O1
X
O2
• Adding a pretest provides weak
information concerning what might
have happened to participants had
the treatment not occurred
One-Group Pretest-Posttest
Design with Double Pretest
O1
O2
X
O3
• Adding multiple pretests reduces the
plausibility of maturation and
regression effects
• Additional pretests can confirm
maturational trends
One-Group Pretest-Posttest
Design Using a Nonequivalent
Variable
{O1A , O1B}
X
{O2A , O2B}
• Measure A is expected to change
because of treatment, B is not
• Both A and B are expected to
respond to the same validity threats
in the same way
A = sale of lottery tickets
B = sale of alcohol
C = sale of tobacco
A
C
C
B
B
A
• Lottery ticket sales in convenience stores
after introduction of signs in store windows
reading “did you buy your ticket?”
Removed-Treatment Design
O1
X
O2
O3
X
O4
• Demonstrates that outcomes rise
and fall with the presence or absence
of treatment
Better
Outcome
Uninterpretable
outcome
Worse
Interpretable
outcome
O1
X
O2
O3
X
• Generally interpretable outcome
pattern
O4
Repeated-Treatment Design
O1
X
O2
X
O3
X
O4
• Few threats could explain a close
relationship between treatment
introductions and removals and
parallel outcome changes
70%
60%
50%
40%
30%
20%
10%
0%
off MM on MM #1 off MM on MM #2 off MM on MM #3 off MM on MM #4 off MM on MM #5
• Mean narcotics use over multiple
Methadone maintenance on/off
conditions
A-B Designs
• Multiple-baseline design (a class of singlesubject designs), or collection of A-B
designs, to assess the effects of an
intervention across separate baselines
A = baseline
B = treatment
• The intervention is introduced in a
staggered manner and the baseline
provides a predicted level of the dependent
variable in absence of the treatment
• A-B-A designs are sometimes called
removal designs (i.e., the treatment is
removed)
Treatment
Effect
Site 2
Baseline
Treatment
Effect
Baseline
Treatment
Site 3
Number of Accidents
Site 1
Baseline
Effect
Weeks
Designs that use a Control
Group but no Pretest
Posttest-Only Design with
Nonequivalent Control Group
NR
NR
X
O1
O1
• Unknown pretest group differences
make it extremely difficult to
separate treatment effects from
selection effects
Posttest-Only Design using an
Independent Sample Pretest
NR
O1
NR
O1
X
O2
O2
• Assumes overlapping group membership
• Useful when
– Pretest measurements may be reactive,
– Cannot follow same groups over time, or
– When interested in studying intact
communities whose members change over
time
Posttest-Only Design using
Proxy Pretest
NR
OA1
NR
OA1
X
OB2
OB2
• Proxy measures should be
conceptually related to and
correlated with the outcome
• Can be used for a variety of purposes
including indexing selection bias
and/or attrition
Case Control Studies
• Predominant method for many forms of
epidemiological research
• Used to identify factors that may
contribute to a condition by comparing
subjects who have that condition (i.e.,
'cases') with those who do not have the
condition but are otherwise similar (i.e.,
'controls')
• Famously, the association between
smoking and lung cancer
• Similar in many respects to Scriven’s
GEM and MOM
Midterm Examination
Midterm Examination
• The examination will consist of 50-75
multiple-choice items, scored as 0 or
1
• You will have 2½ hours to complete
the examination
• You may use one page of notes
(front and back) on 8½” X 11’’ paper
– You will be asked questions about
statistical power, but will not be required
to calculate power
Case Study
Case Study Activity
•
•
•
•
•
•
An aid agency implemented a project in Bangladesh with the objective
of improving the nutritional and physical health status of men and
women
The intervention consisted of a package of services including:
nutrition education, primary health care, and other activities
To determine whether the intervention might be effective, the project
was field-tested in a small rural community prior to large-scale
implementation throughout the country
A small monetary incentive was provided and slightly more than half
of the community’s men and women participated in the study
All men and women in the community were weighed and height
measurements taken prior to the intervention - body mass index
(BMI) was calculated and then again six months after the intervention
Those who did not participate were used as a control group and the
evaluators found significant improvements in nutritional and physical
health indicators for the treatment group contrasted with the control
Questions
• What is the design of the study?
• What internal validity threats are
most plausible?
• How might the design feasibly be
improved?