Transcript Establishing a Cause-Effect Relationship

Establishing a Cause-Effect
Relationship
Internal Validity
Is the relationship causal between...
• The “treatment” and the “outcomes”
• The independent and dependent variables.
Alternative
cause
Alternative
cause
Treatment
What you do
Observation
Outcomes
Alternative
cause
What you see
Alternative
cause
In this study
Establishing Cause and Effect
Temporal precedence
Establishing Cause and Effect
Temporal precedence
Cause
then
Effect
Time
It can get complicated through:
-sloppiness (campaign contributions
- Chicken and egg cyclical functions
(democracy and GDP)
Establishing Cause and Effect
Temporal precedence
Cause
then
Time
Covariation of cause and effect
Effect
Establishing Cause and Effect
Temporal precedence
Cause
then
Effect
Time
Covariation of cause and effect
if X, then Y
if not X, then not Y
if treatment given, then outcome observed (usually)
if program not given, then outcome not observed
Establishing Cause and Effect
Temporal precedence
Cause
then
Effect
Time
Covariation of cause and effect
if X, then Y
if not X, then not Y
if program given, then outcome observed
if program not given, then outcome not observed
Dosage effects or comparative statics:
If more of treatment, then more of outcome observed
if less of treatment given, then less of outcome observed
Establishing Cause and Effect
Temporal precedence
Cause
then
Effect
Time
Covariation of cause and effect
if X, then Y
if not X, then not Y
No alternative
explanations
Treatment Micromediation
Outcome
Establishing Cause and Effect
Temporal precedence
Cause
then
Effect
Time
Covariation of cause and effect
No alternative
explanations
if X, then Y
if not X, then not Y
Alternative cause
(substantive)
Alternative
cause
Treatment Micromediation
Alternative
cause
Outcome
Alternative cause
(nuisance)
In Lab or Field Experiments…
Temporal
precedence
Covariation of
cause and effect
No alternative
explanations



Is taken care of because you
intervene before you measure
outcome
Is measured by comparing
treated and untreated groups
Is the central issue of internal
validity -- usually taken care of
through random assignment
Single-Group Threats to Internal
Validity
The Single Group Case
Two designs:
The Single Group Case
Two designs:
Administer
program
X
Measure
outcomes
O
“Post-test only single-group design”
- X is the treatment
- O is the observation
The Single Group Case
Two designs:
“pre-test, post-test single-group design”
or
“interrupted time-series”
Measure
baseline
O
Administer
program
X
Measure
outcomes
O
The Single Group Case
Alternative
explanations
Two designs:
Administer
program
Measure
outcomes
X
Measure
baseline
O
O
Administer
program
X
Alternative
explanations
Measure
outcomes
O
Alternative
explanations
Example



After the 2003 recall election, did
Democrats in the California Assembly
move to the center?
California ran a full legislative “season”
before the October, 2003 election, then
ran another “season” afterward.
We can look at roll call vote behavior
Example: What Kind of Design?
History Threat
Pretest
O



Program
Posttest
X
O
Any other event that occurs between pretest
and posttest
Perhaps the nation was just shifting to the
center at this time.
How might we rule it out?
Maturation Threat
Pretest
O


Program
Posttest
X
O
Normal growth between pretest and
posttest.
Coming into an election year, state
legislators always shift to the center.
Ruling Out a Maturation Threat
Testing Threat
Pretest
O


Program
Posttest
X
O
The effect on the posttest of taking the
pretest
Legislators may have learned that the state
was watching them. When real tests are
given, this is a big problem.
Instrumentation Threat
Pretest
O


Program
Posttest
X
O
Any change in the test from pretest and
posttest
A different test may have been used if a
different roll call estimation technology used.
Mortality Threat
Pretest
O


Program
Posttest
X
O
Nonrandom dropout between pretest and
posttest
If some legislators had been recalled along
with Gray Davis, this would be a problem.
Regression Threat
Pretest
O


Program
Posttest
X
O
Group is a nonrandom subgroup of
population.
The 2003 session was particularly extreme,
any other session would look more centrist.
Multiple-Group Threats to Internal
Validity
The Central Issue



When you move from single to multiple
group research the big concern is
whether the groups are comparable.
Usually this has to do with how you
assign units (for example, persons) to
the groups (or select them into groups).
If you are not careful, may mistake a
selection effect for a treatment effect.
The Multiple Group Case
Alternative
explanations
Measure
baseline
O
Administer
treatment
X
O
Measure
baseline
Measure
outcomes
O
O
Do not
administer
treatment
Measure
outcomes
Alternative
explanations
Example



Suppose USAID looked before and after
at countries where it did and didn’t run
governance programs in the last decade
Pre-post program-comparison group
design
Measures (O) are all of the things Clark
hates, but let’s set that aside for now.
Selection Threats
O
O


X
O
O
Any factor other than the program that
leads to posttest differences between
groups.
USAID did not randomly select the
countries in which it ran programs, and
sent aid to those with the lowest-rated
governments
Selection-History Threat
O
O


X
O
O
Any other event that occurs between
pretest and posttest that the groups
experience differently.
For example, countries that begin with
more stable democracies faced fewer
challenges in the past decade.
Selection-Maturation Threat
O
O


X
O
O
Differential rates of normal growth
between pretest and posttest for the
groups.
It is easier to move from a semidemocracy to a full democracy than it is
to move from a non-democracy to a
semi-democracy
Selection-Testing Threat
O
O


X
O
O
Differential effect on the posttest of
taking the pretest.
At least these measures are
“unobtrusive,” so this probably is not a
grave threat
Selection-Instrumentation Threat
O
O


X
O
O
Any differential change in the test used
for each group from pretest and
posttest
For example, the Polity measures may
give some countries credit for having a
USAID program
Selection-Mortality Threat
O
O


X
O
O
Differential nonrandom dropout
between pretest and posttest.
Perhaps the countries with weak
governments are more likely to cease
being a country over the past decade.
Selection-Regression Threat
O
O


X
O
O
Different rates of regression to the
mean because groups differ in
extremity.
For example, the countries that USAID
chooses may have nowhere to go but
up.
“Social Interaction” Threats to
Internal Validity
What Are “Social” Threats?
• All are related to social pressures in the
research context, which can lead to
posttest differences that are not directly
caused by the treatment itself.
• Most of these can be minimized by
isolating the two groups from each
other, but this leads to other problems
(for example, hard to randomly assign
and then isolate, or may reduce
generalizability).
Types of Designs
Types of Designs
Random assignment?
Types of Designs
Random assignment?
Yes
Types of Designs
Random assignment?
Yes
Randomized or
true experiment?
Types of Designs
Random assignment?
Yes
Randomized or
true experiment?
No
Types of Designs
Random assignment?
Yes
Randomized or
true experiment?
No
Control group or
multiple measures?
Types of Designs
Random assignment?
Yes
Randomized or
true experiment?
No
Control group or
multiple measures?
Yes
Types of Designs
Random assignment?
Yes
Randomized or
true experiment?
No
Control group or
multiple measures?
Yes
Quasi-experiment
Types of Designs
Random assignment?
Yes
Randomized or
true experiment?
No
Control group or
multiple measures?
Yes
Quasi-experiment
No
Types of Designs
Random assignment?
Yes
Randomized or
true experiment?
No
Control group or
multiple measures?
Yes
Quasi-experiment
No
Nonexperiment
Design Notation Example
R
R
O
O
X
O
O
Os indicate different
waves of
measurement.
Elements of a Design





Observations and measures
Treatments
Groups
Assignment to group
Time
Design Notation Example
Vertical alignment
of Os shows that
pretest and posttest
are measured at same time.
R
R
O
O
X
O
O
Design Notation Example
X is the treatment.
R
R
O
O
X
O
O
Design Notation Example
There are two
lines, one for
each group.
R
R
O
O
X
O
O
Design Notation Example
R
R
R indicates
the groups
are
randomly
assigned.
O
O
X
O
O
Design Notation Example
R
R
O1 X
O1
Subscripts
indicate
subsets of
measures.
O1, 2
O1, 2
Design Notation Example
R
R
O
O
X
O
O
Pretest-posttest (before-after)
Treatment versus comparison group
Randomized experimental design
Design Example
Posttest Only Randomized Experiment
Design Example
Posttest Only Randomized Experiment
R
R
X
O
O
Design Example
Pretest-Posttest Nonequivalent Groups
Quasi-Experiment
Design Example
Pretest-Posttest Nonequivalent Groups
Quasi-Experiment (note multiple groups
or multiple observations are REQUIRED
to have a quasi-experiment)
N
N
O
O
X
O
O
Design Example
Posttest Only Nonexperiment
Design Example
Posttest Only Nonexperiment
X
O