Transcript Document

Week 1: Introduction
and Research Design
Adv. Experimental
Methods & Statistics
PSYC 4310 / COGS 6310
Michael J. Kalsher
Department of
Cognitive Science
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
1
Course Overview
• Review of research design.
• Review of SPSS (data entry; setting up variables; graphing;
syntax; etc.)
• Statistical analysis techniques
–
–
–
–
–
Correlation and simple regression
Multiple regression
t-tests, ANOVA / ANCOVA / MANOVA
Non-parametric statistics
Factor analysis
• Grading requirements
– Labs, Problem Sets, Data Collection/Analysis Project
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
2
Research Methods & Design:
Establishing Control over your variables
• Historical foundations of scientific research in
the behavioral and social sciences.
• The importance of research design
– Ruling out alternative explanations.
– Establishing control of IVs.
• Research Design vs. Statistical Analysis
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
3
Methods of Establishing Truth
• Tenacity
– “It’s so because it’s so”
• Authority
– “Aristotle said it’s so”
• Logical Deduction (Rationalism)
– Aristotle said women have fewer teeth than men (Premise)
– You are a woman
– Therefore, you have fewer teeth than I
• Empiricism
– Combines Logical Deduction with observation
(measurement)
– “Let’s count your teeth”
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
Scientific Method
• Shared observations
– Rules out individual experiences like religious
revelations or esthetic experiences (William
James).
• Reproducible Effects
– “No miracles”
• Conditional Truths
– Premises may be wrong
– Necessary Connection may be wrong
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
Types of Relationship
(between two concepts)
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
Spurious Relationships
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
Spurious Relationships
Ice Cream Sales
Heat Wave
Swimming Pool Drownings
A city's ice cream sales are found to be highest when the rate of drownings in the city’s swimming
pools is highest. To allege that ice cream sales cause drowning, or vice-versa, would be to imply a
spurious relationship between the two. In reality, a third variable, in this instance a heat wave, more
likely caused both.
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
Sets of Relationships (a theory)
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
Elements of a Theory
Sensory
Experience
Theoretical
Definition
Operational
Definition
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
Theoretical Linkage
• Verbal World
– Provides Necessary Connection
– Example:“Newspaper Readership will increase
with Age because older persons will be more
interested in world affairs, and will have more
leisure time”
• Provides a reasonable process by
which the cause produces the effect
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
Operational Linkage
• Measurement World
– Provides a description (logical or mathematical) of the
change in the effect produced by a change in the
cause
– Example:
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
Nonlinear Operational Linkages
• Linear linkages are
typically assumed by
common statistical
procedures.
• Nonlinear linkages are
more realistic, but more
complex.
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
A Model of the Research Process:
Levels of Constraint
(Model used to illustrate the continuum of demands placed on the adequacy of the
information used in research and on the nature of the processing of that information.)
High
Low
PSYC 4310/6310
Experimental Research
Differential Research
Correlational Research
Case-study Research
Naturalistic Observation
Exploratory Research
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
Research plan
becomes increasingly
detailed (e.g., precise
hypotheses and
analyses) but less
flexible.
Research plan may be
general, ideas, questions,
and procedures relatively
unrefined.
Classes of Research Variables:
Variables defined by their use in research
Independent variable
A variable that is actively manipulated
by the researcher to see what its impact
will be on other variables.
Dependent variable
A variable that is hypothesized to be
affected by the independent-variable
manipulation.
Extraneous variable
Any variable (usually unplanned or
uncontrolled factors), other than the
independent variable, that might
the dependent measure in a study.
affect
A constant
Any variable prevented from varying
(by holding variables constant, they do not affect the outcome of the
).
research
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
Classes of Research Variables:
The Measurement Model
Variable values are represented by numbers, but these numbers
may not demonstrate all the characteristics of true numbers.
1. Nominal. A variable made up of discrete, unordered categories. Each
category is either present or absent and categories are mutually exclusive
and exhaustive (e.g., gender).
2. Ordinal. A variable for which different values indicate a difference in the
relative amount of the characteristic being measured.
3. Interval. A variable for which equal intervals between variable values
indicate equal differences in amount of the characteristic being measured.
4. Ratio. Ratios between measurements as well as intervals are meaningful
because there is a starting point (zero).
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
Scales of Measurement:
Some Examples
Levels of Measurement
Nominal
Examples
Ordinal
Ratio
Diagnostic categories
Socioeconomic
Test scores;
Weight; length;
brand names; political
class; ranks
personality and
reaction time;
attitude scales
# of responses
Identity; magnitude
Identity; magnitude;
equal intervals
equal intervals;
or religious affiliation
Properties
Interval
Identity
Identity; magnitude
true zero point
Mathematical
Operations
Type of Data
Typical
Statistics
PSYC 4310/6310
None
Rank order
Add; subtract
Add; subtract;
multiply; divide
Nominal
Ordered
Score
Score
Chi Square
Mann-Whitney
t-test; ANOVA
t-test; ANOVA
U-test
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
The Role of Variance
- In an experiment, IV(s) are manipulated to cause variation between
experimental and control conditions.
- Experimental design helps control extraneous variation--the variance
due to factors other than the manipulated variable(s).
Sources of Variance
- Systematic between-subjects variance
Experimental variance due to manipulation of the IV(s) [The Good Stuff]
Extraneous variance due to confounding variables.
[The Not-So-Good Stuff]
Natural variability due to sampling error
- Non-systematic within-groups variance
Error variance due to chance factors (individual differences) that affect some
participants more than others within a group
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
18
Separating Out The Variance
SST
SSM
PSYC 4310/6310
Advanced Experimental Methods and Statistics
SST = Sums of Squares Total
SSM = Sums of Squares Model
SSR = Sums of Squares Error
SSR
© 2012, Michael Kalsher
19
Controlling Variance in Experiments
In experimentation, each study is designed to:
1. Maximize experimental variance.
2. Control extraneous variance.
3. Minimize error variance.
• Good measurement
• Manipulated and Statistical control
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
20
Controlling Variance in Observational
Studies
• Choose IV’s with large natural variance
• Control for alternate explanations by
measuring confounding variables and
statistically removing their variance
• Minimize error variance
– Good measurement
– Statistical control
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
21
Maximizing Experimental Variance:
Strong manipulations and Manipulation Checks
Experimental Variance
(The Good Stuff)
Due to the effects of the IV(s) on the DV(s)
Ensure that experimental manipulations are strong and
reliable!
Manipulation Check
Procedures designed to determine whether manipulation
of the IV(s) had the intended effect(s) on the DV(s)
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
22
Controlling Extraneous Variance
Extraneous variables: Between-group variables--other
than the IV(s)--that have effects on whole groups and thus
may confound the results.
Goal: To prevent extraneous variables from differentially affecting the groups.
Solution: Take steps to ensure that: (1) the experimental and control groups
are equivalent at the beginning of the study; and (2) groups are treated exactly
the same--save for the intended manipulation (of the IV).
Methods (for controlling extraneous variance):
1.
2.
Random Assignment
Select participants on the basis of one or more potentially confounding
variables (e.g., age, ethnicity, social class, IQ, sex).
3. Build the confounding variables into the study as additional IVs.
4. Match participants on the confounding variable or use within-subjects
design
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
23
Test Statistics
Essentially, most test statistics are of the following
form:
Systematic variance
Test statistic =
Unsystematic variance
Test statistics are used to estimate the likelihood that an
observed difference is real (not due to chance), and is
usually accompanied by a “p” value (e.g., p<.05, p<.01,
etc.)
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
24
Types of Mistakes
Statistical decision
Reject Ho
Don’t reject Ho
PSYC 4310/6310
Advanced Experimental Methods and Statistics
True state of null hypothesis
Ho true
Ho false
Type I error
Correct
© 2012, Michael Kalsher
Correct
Type II error
25
Effect Sizes:
The coefficient of determination
The statistical test only tells us whether it is safe to
conclude that the means come from different populations.
It doesn’t tell us anything about how strong these
differences are. So, we need a standard metric to gauge
the strength of the effects.
r2 (r-Square), or the “Coefficient of Determination”, is one
metric for gauging effect size.
Rules of Thumb regarding effects sizes:
Small effect: 1-3% of the total variance
Medium effect: 10% of the total variance
Large effect: 25% of the variance
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
r2
=
SSM
SST
26
Essential Elements of Research:
Reliability, Validity, Control and Importance
Reliability
Getting the same result when a measurement device is applied to the
same quantity repeatedly.
Validity
The extent to which a measurement tool (test,
device) measures what it purports to measure.
Control
Behavior can be influenced by many factors, some known and others
unknown to the researcher. Control refers to the systematic
methods employed by a researcher to reduce threats to the
the study posed by extraneous influences on the behavior of
participants and the observer.
validity of
both the
Importance
PSYC 4310/6310
Does the research question we are trying to answer warrant the
expenditure of resources (i.e., time, money, effort) that will be
required to complete the study).
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
Types of Reliability
Test-retest Reliability
Consistency of measurement over time
Internal Consistency
Inter-item correlation
Interrater Reliability
Level of agreement between independent
Agreement
observers of behavior(s). Assessed via Agreement + Disagreement x 100
correlation or the procedure at right.
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
Evaluating Measures:
Effective Range
Effective Range:
Scales sensitive enough to detect differences among one
group of subjects may be insensitive to detect differences
among another.
Scale Attenuation (or range restriction).
A problem associated with scales not ranging high enough,
low enough, or both.
Leads to “ceiling” effects and “floor” effects that distort data
by not measuring the full range of a variable.
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
Types of Validity
Face validity. The (non-empirical) degree to which a test
appears to be a sensible measure.
Content validity. The extent to which a test adequately
samples the domain of information, knowledge, or skill that
it purports to measure.
Criterion validity.
Now (concurrent) and Later (predictive).
Involves determining the relationship (correlation) between
the predictor (IV) and the criterion (DV).
Construct validity. The degree to which the theory or
theories behind the research study provide(s) the best
explanation for the results observed.
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
Internal vs. External Validity
Internal Validity
Extent to which causal/independent variable(s) and no
other extraneous factors caused the change being
measured.
External Validity (generalizability)
Degree to which the results and conclusions of your
study would hold for other persons, in other places,
and at other times.
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
Threats to Internal Validity:
Factors that reduce our ability to draw valid conclusions
Selection
History
Maturation
Repeated Testing
Instrumentation
Regression to the mean
Subject mortality
Selection-interactions
Experimenter bias
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
Reducing Threats to Internal Validity
The role of Control
Behavior is influenced by many factors termed—confounding
variables—that tend to distort the results of a study, thereby
making it impossible for the researcher to draw meaningful
conclusions. Some of these may be unknown to the researcher.
Control refers to the systematic methods (e.g., research
designs) employed to reduce threats to the validity of the study
posed by extraneous influences on both the participants and the
observer (researcher).
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
Group/Selection threat
Occurs when nonrandom procedures are used to assign
subjects to conditions or when random assignment fails
to balance out differences among subjects across the
different conditions of the experiment.
Example:
A researcher is interested in determining the factors most likely to
elicit aggressive behavior in male college students. He exposes
subjects in the experimental group to stimuli thought to provoke
aggression and subjects in the control group to stimuli thought to
reduce aggression and then measures aggressive behaviors of the
students. How would the selection threat operate in this instance?
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
History threat
Events that happen to participants during the
research which affect results but are not linked to
the independent variable.
Example:
The reported effects of a program designed to improve
medical residents’ prescription writing practices by the
medical school may have been confounded by a self-directed
continuing education series on medication errors provided to
the residents by a pharmaceutical firm's medical education
liaison.
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
Maturation threat
Can operate when naturally occurring biological or
psychological changes occur within subjects and
these changes may account in part or in total for
effects discerned in the study.
Example:
A reported decrease in emergency room visits in a long-term
study of pediatric patients with asthma may be due to subjects
outgrowing childhood asthma rather than to any treatment
regimen introduced to treat the asthma.
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
Repeated testing threat
May occur when changes in test scores occur not
because of the intervention but rather because of
repeated testing. This is of particular concern when
researchers administer identical pretests and
posttests.
Example:
A reported improvement in medical resident prescribing
behaviors and order-writing practices in the study previously
described may have been due to repeated administration of the
same short quiz. That is, the residents simply learned to provide
the right answers rather than truly achieving improved
prescribing habits.
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
Instrumentation threat
When study results are due to changes in instrument
calibration or observer changes rather than to a true
treatment effect, the instrumentation threat is in
operation.
Example:
In Kalsher’s Experimental Methods and Statistics course, he
evaluates students progress in understanding principles of research
design at week 3 of the semester. A graduate T.A. evaluates the
students at the conclusion of the course. If the evaluators are
dissimilar enough in their approach, perhaps because of lack of
training, this difference may contribute to measurement error in
trying to determine how much learning occurred over the semester.
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
Statistical Regression threat
The regression threat can occur when subjects
have been selected on the basis of extreme
scores, because extreme (low and high) scores in
a distribution tend to move closer to the mean (i.e.,
regress) in repeated testing.
Example:
if a group of subjects is recruited on the basis of extremely high
stress scores and an educational intervention is then implemented,
any improvement seen could be due partly, if not entirely, to
regression to the mean rather than to the coping techniques
presented in the educational program.
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
Experimental Mortality threat
Experimental mortality—also known as attrition,
withdrawals, or dropouts—is problematic when there
is a differential loss of subjects from comparison
groups subsequent to randomization, resulting in
unequal groups at the end of a study.
Example:
Suppose a researcher conducts a study to compare the effects of a
corticosteroid nasal spray with a saline nasal spray in alleviating
symptoms of allergic rhinitis (irritation and inflammation of the nasal
passages). If subjects with the most severe symptoms preferentially
drop out of the active treatment group, the treatment may appear
more effective than it really is.
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
Selection Interaction threats
A family of threats to internal validity produced
when a selection threat combines with one or
more of the other threats to internal validity.
When a selection threat is already present, other
threats can affect some experimental groups,
but not others.
Example:
If one group is dominated by members of one fraternity
(selection threat), and that fraternity has a party the night
before the experiment (history threat), the results may be
altered for that group.
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
Threats to External Validity:
Ways you might be wrong in making generalizations
People, Places, and Times
Demand Characteristics
Hawthorne Effects
Order Effects (or carryover effects)
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
People threat:
Are the results due to the unusual
type of people in the study?
Example:
You learn that the grant you submitted to assess average
drinking rates among college students in the U.S. has been
funded. In late November, you post an announcement
about the study on campus to get subjects for the study.
100 students sign up for the study. Of these, 78 are
members of campus fraternities; the other 22 are members
of the school’s football team.
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
Places threat:
Did the study work because of the
unusual place you did the study in?
Example:
Suppose that you conduct an “educational” study in a
college town with lots of high-achieving educationallyoriented kids.
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
Time threat:
Was the study conducted at a peculiar time?
Example:
Suppose that you conducted a smoking cessation study
the week after the U.S. Surgeon General issued the well
publicized results of the latest smoking and cancer studies.
In this instance, you might get different results than if you
had conducted the study the week before.
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
Demand Characteristics
Participants are often provided with cues to the
anticipated results of a study.
Example:
When asked a series of questions about depression, participants
may become wise to the hypothesis that certain treatments may
work better in treating mental illness than others. When participants
become wise to anticipated results (termed a placebo effect), they
may begin to exhibit performance that they believe is expected of
them.
Making sure that subjects are not aware of anticipated outcomes
(termed a blind study) reduces the possibility of this threat.
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
Hawthorne Effects
Similar to a placebo, research has found that the mere
presence of others watching a person’s performance
causes a change in their performance. If this change is
significant, can we be reasonably sure that it will also
occur when no one is watching?
Addressing this issue can be tricky but employing a
control group to measure the Hawthorne effect of those
not receiving any treatment can be very helpful. In this
sense, the control group is also being observed and will
exhibit similar changes in their behavior as the
experimental group therefore negating the Hawthorne
effect.
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
Order Effects (carryover effects)
Order effects refer to the order in which treatment
is administered and can be a major threat to
external validity if multiple treatments are used.
Example:
If subjects are given medication for two months, therapy for another
two months, and no treatment for another two months, it would be
possible, and even likely, that the level of depression would be least
after the final no treatment phase. Does this mean that no treatment
is better than the other two treatments? It likely means that the
benefits of the first two treatments have carried over to the last phase,
artificially elevating the no treatment success rates.
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
The Role of Experimental Design
In most social and behavioral research studies, we attempt to
obtain at least one score from each participant (usually
more!). Any obtained score is comprised of a number of
components:
1. A ‘true score’ for the thing we hope we are measuring.
2. A ‘score for other things’ that we measure inadvertently.
3. Systematic (non-random) bias (usually ok as long as it affects all
participants equally).
4. Random (non-systematic) error (which should cancel out over large
numbers of observations).
We want our obtained score to consist of as much ‘true
score’, and as little of the other factors, as possible.
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
49
Research Study Control
Control removes sources of error in inferences
– Reduces the chance of wrong conclusions
– Increases the power of statistics to find
relationships in the presence of random error
(“noise”)
Types of Control
– Direct Manipulation
– Randomization
– Statistical Control
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
50
Types of Control:
Direct Manipulation
Sources of error held constant by
research design or sampling decisions
– Example: a researcher investigating the effects of
seeing justified violence in video games on
children knows that young children cannot
interpret the motives of characters accurately.
She decides to limit her study to older children
only, to eliminate random responses or
unresponsiveness of younger children.
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
51
Types of Control:
Randomization
Unknown sources of error are equalized
across all research conditions by randomly
assigning subjects or by randomly choosing
experimental materials.
– Example: Many different factors are known to affect
the amount of use of Internet social networking sites. A
researcher wants to test two different site designs. He
randomly assigns subjects to work with each of the two
designs. This equalizes the amount of confounding
error from unknown factors in both groups.
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
52
Types of Control:
Statistical Control
Known confounding variables are measured,
and mathematical procedures are used to
remove their effect.
– Example: A political communication researcher
interested in studying emotional appeals versus
rational appeals in political commercials suspects that
the effects vary with the age of the viewer. She
measures age, and uses it as an independent predictor
(with multivariate statistics) to isolate, describe, and
remove its effect.
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
53
Contrasting Methods of Control
Type of
Control
Strength
Weakness
Direct
Manipulation
• Removes effect completely
• Must know source of effect
• Reduces generalizability
Randomization
• Don’t have to know source of
effect
• Equalizes effect so there is no
systematic confound
• Reduces statistical power
by adding to unsystematic
error variance
Statistical
control
• Estimates effect of
confounding variables
• Expands theoretical model
• Must know source of effect
• Requires more complex
statistics
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
54
Basic Types of Research
• Observational Methods
• Quasi-Experimental Designs
• True Experimental Designs
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
55
Observational Methods
No direct manipulation of variables by the
researcher. Behavior is merely recorded--but
systematically and objectively so that the
observations are potentially replicable.
Advantages
•
•
Reveals how people normally behave.
Experimentation without prior careful observation can lead to a
distorted or incomplete picture.
Disadvantages
•
•
Generally more time-consuming.
Doesn’t allow identification of cause and effect.
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
56
Quasi-Experimental Design
In a quasi-experimental study, the experimenter
does not have complete control over manipulation
of the independent variable or how participants
are assigned to the different conditions of the
study.
Advantages
•
•
Natural setting
Higher face validity (from practitioner viewpoint)
Disadvantages
•
Not possible to isolate cause and effect as conclusively as with a
“true” experiment.
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
57
Types of
Quasi-Experimental Designs
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
58
One Group Post-Test Design
Measurement
Treatment
Time
Change in participants’ behavior may or may not be
due to the intervention.
Prone to time effects, and lacks a baseline against
which to measure the strength of the intervention.
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
59
One Group Pre-test Post-test Design
Measurement
Treatment
Measurement
Time
Comparison of pre- and post-intervention scores
allows assessment of the magnitude of the
treatment’s effects.
Prone to time effects, and it is not possible to
determine whether performance would have
changed without the intervention.
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
60
Interrupted Time-Series Design
Measurement
Measurement
Time
Measurement
Treatment
Measurement
Measurement
Don’t have full control over manipulations of
the IV. No way of ruling out other factors.
Potential changes in measurement.
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
Measurement
61
Static Group Comparison Design
Group A:
Treatment
Measurement
(experimental group)
Group B:
No Treatment
Measurement
(control group)
Time
Participants are not assigned to the conditions randomly.
Observed differences may be due to other factors.
Strength of conclusions depends on the extent to which
we can identify and eliminate alternative explanations. 62
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
Experimental Research:
Between-Groups and
Within-Groups Designs
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
63
Between-Groups Designs
Separate groups of participant are used for each
condition of the experiment.
Within-Groups (Repeated Measures) Designs
Each participant is exposed to each condition of the
experiment (requires less participants than between
groups design).
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
64
Between-Groups Designs
Advantages
•
•
•
Simplicity
Less chance of practice and fatigue effects
Useful when it is not possible for an individual to
participate in all of the experimental conditions
Disadvantages
•
•
Can be expensive in terms of time, effort, and number of
participants
Less sensitive to experimental manipulations
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
65
Examples of
Between-Groups Designs
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
66
Post-test Only / Control Group Design
Group A:
Measurement
Treatment
(experimental group)
Random
allocation:
Group B:
Measurement
No Treatment
(control group)
Time
If randomization fails to produce equivalence, there is no way
of knowing that it has failed. Experimenter cannot be certain
that the two groups were comparable before the treatment.
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
67
Pre-test / Post-test Control Group
Design
Group A:
Measurement
Treatment
Measurement
No Treatment
Measurement
Random
allocation:
Group B: Measurement
Time
Pre-testing allows experimenter to determine equivalence
of the groups prior to the intervention. However, pretesting may affect participants’ subsequent performance.
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
68
Random allocation:
Solomon Four-Group Design
Group A: Measurement
Treatment
Measurement
Group B: Measurement
No Treatment
Measurement
Group C:
Treatment
Measurement
Group D:
No Treatment
Measurement
Time
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
69
Within-Groups Designs:
Repeated Measures
Advantages
• Economy
• Sensitivity
Disadvantages
• Carry-over effects from one condition to another
• The need for conditions to be reversible
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
70
Repeated-Measures Design
Treatment
Measurement
No Treatment
Measurement
Measurement
Treatment
Measurement
Random Allocation
No Treatment
Time
Potential for carryover effects can be avoided by randomizing the order
of presentation of the different conditions or counterbalancing the order
in which participants experience them.
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
71
Latin Squares Design
Three Conditions or Trials
order of conditions or trials:
One group of participants
A
B
C
Another group of participants
B
C
A
Yet another group of participants
C
A
B
Order of presentation of conditions in a within-subjects design can be
counterbalanced so that each possible order of conditions occurs just once.
Problem not completely eliminated because A precedes B twice, but B precedes
A only once. Same with C and A.
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
72
Balanced Latin Squares Design
Four Conditions or Trials
order of conditions or trials:
One group of participants
A
B
C
D
Another group of participants
B
D
A
C
Yet another group of participants
D
C
B
A
And yet another group of participants
C
A
D
B
Note: This approach works only for experiments with an even number of conditions. For
additional help with more complex multi-factorial designs, see: http://www.jic.bbsrc.ac.uk
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
73
Factorial Designs
• include multiple independent variables
• allow for analysis of interactions
between variables
• facilitate increased generalizability
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
74
Important Concepts
Alternative hypothesis
Dispersion
Null hypothesis
Score-level variable
Standard Deviation
Between-groups design
Effect Size
Observational study
Skew
Standard Error
Categorical variable
Experimental research
One-tailed test
Standard Deviation
Systematic variation
Central tendency
Face validity
Ordinal variable
Standard Error
Two-tailed test
Confidence intervals
Frequency distribution
Outcome variable
Systematic variation
Type I error
Confounding variable
Independent variable
Platykurtic
Two-tailed test
Type II error
Construct validity
Kurtosis
Power
Type I error
Unsystematic variation
Content validity
Leptokurtic
Practice effects
Type II error
Validity
Continuous variable
Level of Measurement
Predictor variable
Unsystematic variation
Variance
Correlational research
Mean
Quasi-exp. research
Validity
Within-groups design
Counterbalancing
Measurement error
Randomization
Variance
z-scores
Criterion validity
Median
Range
Within-groups design
Degrees of Freedom
Mode
Reliability
z-scores
Dependent variable
Nominal variable
Repeated measures
Score-level variable
Discrete variable
Normal Distribution
Sampling distribution
Skew
PSYC 4310/6310
Advanced Experimental Methods and Statistics
© 2012, Michael Kalsher
75