Transcript Threats to Internal Validity

Educational Action
Research
Todd Twyman
Summer 2011
Week 1
Agenda



Discuss Internal and External Validity
Review Threats to Internal Validity
Consider Precautions for Minimizing
Threats to Internal Validity
Why are you here?


How can you use the results from your
action research project to inform your
practices?
Why is action research important for
improving schools?
Internal Validity
The internal validity of a study relies
on the researcher’s ability to rule out
factors other than the intervention as
possible causes of changes in the
dependent variable.
Improving Internal Validity



Maximize systematic variance
 Make the groups as different as possible
in regard to the intervention
Minimize error variance
 Reduce the differences between the
participants
Control extraneous variance
 Manipulate the environment to maximize
the controls
Improving Internal Validity

Focus is on trying to “control” extraneous
factors (or confounds). Identify threats and
then either delete or reduce threats to
validity through your research design.
Thinking about Internal
Validity…





What could have caused these findings?
What could have influenced the results
besides the intervention?
What are alternative hypotheses for the
results?
What decisions would you make based on
these results?
What are the implications of the findings?
Threats to Internal Validity








History
Maturation
Testing
Instrumentation
Statistical regression
Differential selection
Experimental
mortality
Selection-maturation
interaction




Experimental
treatment diffusion
Compensatory rivalry
by the control group
Compensatory
equalization of
treatments
Resentful
demoralization of the
control group
Threats to Internal Validity

History: Events may occur that influence the
observed effects.

Maturation: Physical/psychological changes
that influence the observed effects.

Testing: Pre-test influences the post-test.

Instrumentation: Changes in the
measurement tool influence the observed
effects.
Identifying Threats

A tenth-grade biology teacher institutes a
new approach to the teaching of onecelled animal life based on a series of
seven-minute single-concept films. Prior to
starting the new unit, she develops a test
covering the material and administers it to
her class before and after the unit, noting
with satisfaction a dramatic improvement
by her pupils on the test.
Threats to Internal Validity

Statistical regression: Test scores move
toward the mean.

Differential selection: Participants are
differentially selected into groups.
 Attrition: Participants differentially drop
out of the study.
 Selection-maturation interaction:
Participants differ based on maturation.
Identifying Threats

A school principal has received a federal grant
to institute a new mathematics enrichment
program. Using school records, he computes the
average mathematics achievement score on a
nationally standardized examination for his fifth-,
ninth-, and twelfth-grade pupils during each of
the preceding three years. He plans to have the
same tests administered for the two years
following the institution of the new program. He
wishes to compare the relative performance of
the three classes before and after the
implementation of the new program.
Threats to Internal Validity

Treatment diffusion: Control group seeks
access to treatment.

Compensatory rivalry: Control group works
harder to make up for group placement.

Compensatory equalization: Outsiders
provide other treatment to compensate.

Demoralization of control group: Control
group performs worse due to discouragement.
Identifying Threats

A researcher uses a table of random numbers to
subdivide 15 junior high school classes into
three groups of five classes each (30 pupils per
class) as follows: “Treatment A,” “Treatment B,”
and “Control.” The treatments are administered
during the first four months of the school year,
and at mid-semester a posttest is given to all of
the 450 pupils involved.
3 Steps to Strengthening
Treatment Fidelity



Write precise specifications for dependent
and independent variables.
TRAIN extensively to ensure treatment is
delivered as intended.
Collect data on fidelity of treatment
implementation.
Internal Validity Review
Threats to Internal Validity
1. History (H)
2. Maturation (M)
3. Testing (T)
4. Instrumentation (I)
5. Statistical regression
(SR)
6. Differential selection
(DS)
7. Experimental mortality
(EM)
8. Selection-maturation
interaction (SMI)
9. Experimental treatment
diffusion (ETD)
10. Compensatory rivalry
by the control group
(CR-control)
11. Compensatory
equalization of
treatments (CE)
12. Resentful
demoralization of the
control group (Rdem)





Control group works harder to make up for
group placement
Pre-test influences the post-test
Control group performs worse due to
discouragement
Test scores move toward the mean
Events may occur that influence the
observed effects




Control group seeks access to treatment
Participants are differentially selected into
groups
Physical/psychological changes that
influence the observed effects
Participants differentially drop out of the
study



Participants differ based on maturation
Changes in the measurement tool
influence the observed effects
Outsiders provide other treatment to
compensate
Quiz
Steps to Reducing Threats to
Validity







Debrief with peers: identify strengths/
weaknesses/ biases.
Align data sources with research questions.
Triangulate data sources.
Record data accurately and without bias.
Embed fidelity checks.
Thoroughly describe the setting and study.
Engage in continuous and ongoing reflection.
Read with an Informed Eye
Assignment 3 will ask you to critique an
Action Research article of your choice.
To prepare:
 Look carefully at the methodology of
research articles.
 Evaluate the degree to which the authors
have controlled threats to internal validity.
 Question everything!
Critiquing Research

Are the problems and their educational
importance explained clearly?

Does the literature review sufficiently set
up a rationale for the need for this study?

Are the research variables easily identified
from the hypothesis or purpose
statement?
Critiquing Research

What instruments are used in the study?
Are they appropriate for the research
question and for the participants?

Is the research design appropriate to the
research question?

How does the study control for threats to
internal or external validity?

Are the conclusions logically related to the
questions posed in the research question?
Your own study



Identify the goals for conducting your
study
Identify the types of validity that are
applicable to your study (consider the
purpose of the study, activities, audience,
data sources)
Create a plan for reducing the threats to
internal validity that is aligned with the
types of validity addressed above.