Transcript Experimental Design Brian Mennecke College of Business Iowa State University The Source…  The source of much of this information comes from Campbell & Stanley… –

Experimental Design
Brian Mennecke
College of Business
Iowa State University
The Source…
 The source of much of this information
comes from Campbell & Stanley…
– Campbell, D. T. and J.C. Stanley (1963).
Experimental and Quasi-Experimental
Designs for Research. Chicago: Rand
McNally College Publishing Company.
The Goal of Research
 When one conducts science, the goal is
to seek out the truth.
– Question: How does one identify truth?
 Experimentation is one mechanism for
identifying causation, which is a step
toward understanding how one set of
factors influence another set of factors
Causation and Positivism
 Positivism is a research perspective that has
as its premise that inferences about cause
can be made.
 David Hume espoused the conditions by
which inference could be made; these
include…
– Contiguity between the cause and effect
– Temporal precedence
– Constant conjunction (i.e., when the effect is seen,
the cause is always present)
But how do we know something
is true?
 Some propositions are not true; how do
we know when something is true or
not?
 One approach is to test for validity.
– Validity is a term used to describe whether
the conclusions one draws about a
proposition are true or false
Types of validity
 Internal Validity: How sure are we that the cause
leads to the expected results? In other words, is it
appropriate for us to infer that the relationship
between variables is causal
 External Validity: How sure are we that we can
generalize the finding of causation to other
populations, settings, or variables?
 Construct Validity: How sure are we that the variables
we are using actually measure the concept (i.e., the
construct) that we are seeking to measure?
 Statistical Conclusion Validity: Do the statistical tests
that we perform accurately measure the relationships
between the variables under study?
Threats to Internal Validity
(Campbell & Stanley)








History: events that occur between the first and second measurement
that are unrelated to the experiment but that could affect the results.
Maturation: Changes in the participants that occur as a function of the
passage of time and not specific to the experiment.
Testing: The effects of taking a first test on the scores of a second test.
Instrumentation: Changes in the measurement instrument or changes
or the observers make changes in the obtained measurements.
Statistical regression (toward to mean): Groups having extreme scores
on the pretest (or selected on the basis of extreme scores) will tend to
have scores closer to the mean on the posttest.
Selection: Biases resulting in differentials selection of respondents for
the comparison groups.
Experimental mortality: Differential loss of respondents from the
comparison groups.
Selection-maturation interaction, other interaction effects:
Threats to External Validity
 Reactive or interaction effects of testing: The pretest
itself might be a learning experience such that by
taking the pretest students gain information that will
affect posttest results
 Interaction of selection and the experimental variable:
Different groups may respond differently to the
experimental variable.
 Reactive effects of experimental arrangements:
Subjects respond differently because they know they
are in an experiment (i.e., the Hawthorne effect)
 Multiple treatment interference: Multiple treatments
applied to the same respondents; the effects of prior
treatments cannot be erased.
What is the basis for asking
questions about causation?
 The source for all questions pertaining
to research experimentation is theory
– Why is theory important?
 Theory should always drive research
because it defines expectations about
the relationships that exist between
variables.
Before we get started…
 Some definitions:
– Construct: An idea or concept that you are
attempting to measure
• Latent Construct: A construct that cannot be measured
directly (e.g., group cohesion)
– Independent Variable: Variables that are presumed
to be the cause of an effect being studied;
independent variables are manipulated to examine
their impact on results
– Dependent Variables: Variables that are observed
to understand the result of causation.
– Hypothesis: A statement of a possible explanation
for causation. An hypothesis is tested by drawing
conclusions from an experimental examination of
the variables that are expected to be related
Types of Experimental Designs
 Pre-experimental designs: One group designs and
designs that compare pre-existing groups
 Quasi-experimental designs: Experiments that have
treatments, outcome measures, and experimental
conditions but that do not use random selection and
assignment to treatment conditions.
 True experimental designs: Experiments that have
treatments, outcome measures, and experimental
conditions and use random selection and assignment
to treatment conditions. This is the strongest set of
designs in terms of internal and external validity.
Pre-Experimental Designs
 Design 1: One-Shot Case Study: A
single group is studied once after some
intervention/treatment that is presumed
to cause change.
– For example, a training program is
implemented and participants are given a
posttest at the conclusion of the training.
X
O
Pre-Experimental Designs
 Design 2: One-Group Pretest-Posttest Design:
One group, not randomly selected nor
randomly assigned, is given a pretest, followed
by a treatment/intervention, and finally a
posttest. There is no comparison group.
Generally done with intact groups.
– For example, a classroom teacher gives her
students a pretest then implements an instructional
strategy followed by a posttest.
O1
X
O2
Pre-Experimental Designs
 Design 3: The Static-Group Comparison: One group
which has experienced a treatment/intervention (X) is
compared to another group that has not had the
intervention. The groups are not randomly selected nor
randomly assigned and are generally pre-existing
groups. There is no pre-observation/pretest.
– For example, comparison of GRE scores for students who
attended a rural high school versus those who attended an
urban high school.
X1
X2
O
O
True Experimental Designs
 Design 4: Pretest-Posttest Control Group Design: One
group is administered a treatment while the other is
not; all groups are observed before and after the
treatment is administered.
– For example, 50 freshman students are randomly selected to
participate in a tutoring study. Half are randomly assigned to a
tutor for their first semester and half are not. All students are
given a pretest at the beginning of the term and a posttest at
the end of the term.
R
R
O1
O1
X
O2
O2
True Experimental Designs
 Design 5: Solomon Four-Group Design: This design involves four
experimental groups. Two of the groups parallel the structure of
Design 4 while the remaining two groups include no pre-test (so
that the effects of the pretest can be evaluated).
– For example, 100 freshman students are randomly selected to
participate in a tutoring study. 25 are randomly assigned to a tutor for
their first semester and given a pretest. 25 are randomly assigned to
a group where no tutor is assigned and they are given a pretest.
Another 25 are randomly assigned to a tutor but not given a pretest.
The remaining 25 are randomly assigned to a group where no tutor is
assigned and they are not given a pretest.
Whew!
R
R
R
R
O1
O1
X
X
O2
O2
O2
O2
True Experimental Designs
 Design 6: Posttest Only Control Group Design: One
group is administered a treatment while the other is
not; all groups are observed after the treatment is
administered BUT not before the treatment.
– For example, students are randomly assigned to two groups of
50 each. The experimental (treatment) group receives a new
teaching method during a special class session. The second
group (the control) receives a traditional teaching method
during a special class session. No pretest is used for each
group. Issues such as existing grades, SAT scores, and other
factors are examined as covariates.
R
R
X
O2
O2
Quasi-Experimental Designs
 Design 7: The Time-Series Experiment: This
design involves periodic measurements of
some group or individuals and the introduction
of a change into the conditions during the
series.
– For example, studying a group of workers over time
and taking several measures of productivity during
this period. At some point a new work process is
introduced and measures of productivity are taken
over several weeks following the intervention.
O1
O2
O3
X
O4
O5
O6
Quasi-Experimental Designs
 Design 8: Equivalent Time-Samples Designs:
This design involves periodic introduction of
treatments followed by measurements with the
treatments varied consistently over time.
– For example, to study the effect on student
discussions of having an observer appear in a
classroom. At time period one, an observer is
present and a measure of discussion level is made.
At time two, no observer is present and a measure
of discussion level is made. At time three an
observer is present, a measure is taken. At time four
an observer is not present, a measure is taken. Etc.
X1
O
X2
O
X1
O
X2
O
Quasi-Experimental Designs
 Design 9: The Equivalent Materials Design: This
design involves giving equivalent samples of materials
to subjects, imparting interventions, and then making
observations.
– For example, subjects are asked to complete a survey
instrument about their opinions related to current events. The
students are then split into two groups and given two different
sets of (falsified) survey results indicating how other students
answered the survey. Both groups are then asked to complete
the survey again to observe how they respond.
Experimental Materials A(O) X0 O
Experimental Materials B(O) X0 O
Quasi-Experimental Designs
 Design 10: Nonequivalent Control Group: This design
involves an experimental and control group with both
given pretests and posttest; however, these groups are
not randomly selected because they constitute
naturally assembled groups (e.g. classrooms). The
assignment of X (the treatment) to one group or the
other is randomly selected by the researcher.
– For example, four sections of a course are chosen to
participate in a study of teaching methods. Half are randomly
assigned a new teaching method and half are not. All are
given pretests at the beginning of the term and all are given
posttests at the end of the semester.
O
O
X
O
O
Quasi-Experimental Designs
 Design 11: Counterbalanced Designs: In this design all subjects
receive all treatments but in a different order. Each treatment
occurs once at each time period and once for each treatment
group. A Latin-square design is a type of counterbalanced design
in which four treatments are applied to four naturally assembled
pools of subjects.
– For example, consider a study of the effect of different training
methods on learning. Subjects are placed into four groups (A,B,C, D)
for different training methods, X1-X4.
Group A
Group B
Group C
Group D
X1O
X2O
X3O
X4O
X2O
X4O
X1O
X3O
X3O
X1O
X4O
X2O
X4O
X3O
X2O
X1O
Quasi-Experimental Designs
 Design 12: The Separate Sample Pretest-Posttest Design: Often
used with large populations (i.e., in public opinion studies) where
the researcher cannot randomize or segregate subgroups for
different treatments. Two equivalent groups are identified, one
sample is measured prior to the treatment and a different (but
equivalent) sample is measured after the treatment. This design is
also called the "simulated before and after" design.
– For example, 100 community members are randomly surveyed
concerning their opinions about local government policies. A PR
campaign is then conducted for six weeks. A follow-up survey is then
conducted with 100 different residents who are randomly selected.
R
R
O
X
X
O
Quasi-Experimental Designs
 Design 13: The Separate Sample Pretest-Posttest
Control Group Design: This design is similar to Design
12; however, a control group is added to the design.
– For example, consider the PR campaign described in Design
12. In this case, the same design is used, but, in addition, the
measurements are made in a similar nearby city where no PR
campaign is run.
R
R
O
R
R
O
X
X
O
O
Quasi-Experimental Designs

Design 15: Recurrent Institutional Cycle Design (A "Patched-Up" Design):
This is an approach used in field research. A researcher begins with an
inadequate design and then adds features to control for one or more
sources of invalidity. The result is an "inelegant accumulation of
precautionary checks." The researcher is aware of rival interpretations
(sources of internal invalidity) and incrementally identifies other data that
would rule out rivals. The design exploits contextual features to refine the
research as it progresses.
– For example, this design would combine a longitudinal and cross sectional
structure. One group will be exposed to X and measured at the same time as
a second group that is just about to be exposed to X. A comparison of the two
groups would be able to be made because it is equivalent to a static group
comparison. The second group would be remeasured (posttest), which would
make the design comparable to the one group pretest-posttest design.
Group A
Group B
X
O1
O1
X
O2
My Research Agenda
 So, what type of research approach do
you think I use?
General Research Themes
 Geographic Information Systems (GIS)
and Location Intelligence
– Studies of the use of GIS as a decision
support tool
– The use of GIS in Businesses and
Organizations
– Location intelligence and the use of
location in decision making
– Perceptions of space and geography
General Research Themes
 Studies of Teams, Collaboration, and
technology
– Virtual Teams
– Team History
– Individual Characteristics
General Research Themes
 Virtual Worlds
– The application of VW to education and
learning
– Perceptions of avatars, space and location
in VWs
– Legal, tax, and social issues in VWs
– Communication and collaboration in VWs
General Research Themes
 Mobile Commerce, Computing, and
Virtual Teams
– Mobile Device Interfaces
– Impressions of Mobile Device Users
– Applications of Mobile Devices
General Research Themes
 Applications of Conjoint to IS Research
– Human Resources
– Information Systems Analysis
– IT Planning
General Research Themes
 IT Adoption and Implementation
– User Acceptance of Mobile Devices
– The Use of Mobile Devices in Commerce
General Research Themes
 The Application of IT for Training and
Learning
– The Application of Technology in Education
– The Role of Communication Technology in
Learning
A Recent Study
 Question: What is the impact of video
conference technology and training
methodology on student learning
 IV:
– Training Mode:
• Enactive Mastery
• Vicarious Experience
– Communication Media
• Face to Face
• Video Conferencing
Results
12
11
10
VE
EM
9
8
7
6
FTF
DVC
Results
Tests of Between-Subjects Effects
Dependent Variable: DQ - Total
Source
Corrected Model
Intercept
MIDGSETO
PFTSCORE
MEDIA
TRAINING
MEDIA * TRAINING
Error
Total
Corrected Total
Type III Sum
of Squares
2765.596a
112.075
339.197
1506.172
198.404
75.519
232.381
7060.445
53232.000
9826.041
df
5
1
1
1
1
1
1
141
147
146
a. R Squared = .281 (Adjusted R Sq uared = .256)
Mean Square
553.119
112.075
339.197
1506.172
198.404
75.519
232.381
50.074
F
11.046
2.238
6.774
30.079
3.962
1.508
4.641
Sig .
.000
.137
.010
.000
.048
.221
.033