Transcript Practice Effects

Research Methods in Psychology
Repeated Measures Designs
Repeated Measures Designs
 Each individual participates in each
condition of the experiment
• completes the DV with each condition
• hence “repeated measures”
 Also called “within-subject” design
• entire experiment is conducted “within” each
subject
Repeated Measures Designs, continued
 Why Use a Repeated Measures Design?
• no need to balance individual differences
across conditions of experiment
 all participants are in each condition
• fewer participants needed
• convenient and efficient
• more sensitive
Sensitivity
 A sensitive experiment
• can detect the effect of an independent
variable
• even if the effect is small
 Repeated measures designs are more
sensitive than independent groups designs
• “error variation” is reduced
 same people participate in each condition
 variability due to individual differences eliminated
Practice Effects
 Main disadvantage of repeated measures
designs is practice effects
• People change as they are tested repeatedly
 performance may improve over time
 people may become bored or tired over time
 Practice effects become a potential
confounding variable if not controlled
Practice Effects, continued
 Example:
• Suppose a researcher compares two different
study methods, A and B
 Condition A: participants use a highlighter to mark
key points while reading a text, then take a test on
the material
 Condition B: participants read a text, then make up
sample test questions and answers, then take a
test on the material
Practice Effects, continued
• Suppose
 all participants first experience Condition A and
then Condition B
 results indicate test scores are higher in Condition
A compared to Condition B
• Is marking text with highlighter (A) better than
writing sample questions/answers (B)?
 impossible to know
• confounding of IV with order of presentation
• practice effects (boredom, fatigue) may account for
poorer performance in Condition B
Practice Effects, continued
 Practice effects must be balanced, or
averaged, across conditions
• Counterbalancing the order of conditions
distributes practice effects equally across
conditions
 half of the participants do Condition A, then B
 the remaining participants to Condition B, then A
 Conditions A and B then have equivalent practice
effects
 practice effects aren’t eliminated, but they are
averaged across the conditions of the experiment
Counterbalancing Practice Effects
 Two types of repeated measures designs
• Complete and Incomplete
• purpose of each type of design is to
counterbalance practice effects
• each design uses different procedures for
counterbalancing practice effects
Complete Design
 Practice effects are balanced within each
participant in the complete repeated
measures design
• each participant experiences each condition
several times, using different orders each time
• a complete repeated measures design is used
when
 each condition is brief (e.g., simple judgments
about stimuli)
Complete Design, continued
 Two methods for generating orders of
conditions
• block randomization
• ABBA counterbalancing
Complete Design, continued
 Block randomization
• a block consists of all conditions (e.g., 4
conditions: A, B, C, D)
• generate a random order of the block (ACBD)
• participant completes condition A, then C,
then B, then D
• generate a new random order for each time
the participant completes the conditions of the
experiment (e.g., DACB, CDBA, ADBD)
Complete Design, continued
 Block randomization
• balances practice effects only when
conditions are presented many times
• practice effects are averaged across the many
presentations of the conditions
• practice effects are not balanced if conditions
are presented only a few times to each
participant
Complete Design, continued
 ABBA counterbalancing
• used when conditions are presented only a
few times to each participant
• procedure: present one random sequence of
conditions (e.g., DABC), then present the
opposite of the sequence (CBAD)
• each condition has the same amount of
practice effects
Complete Design, continued
 ABBA counterbalancing
• balance practice effects that are “linear”
 linear practice effects
• participants change in the same way following each
presentation of a condition
 nonlinear practice effects
• participants change dramatically following the
administration of a condition
• example: participant experiences insight about how to
complete an experimental task (“aha … now I get it”)
• likely to use this insight in subsequent conditions
Complete Design, continued
• Example of linear practice effects
 suppose participants gain “one unit” of practice
with each administration (“trial”) of a condition
• there are zero practice effects with the first administration
Trial 1 Trial 2 Trial 3 Trial 4 Trial 5 Trial 6
Condition
A
B
C
C
B
A
Practice Effects +0
+1
+2
+3
+4
+5
Practice effects are balanced because total practice effects is +5
for each condition:
A: 0 + 5
B: 1 + 4
C: 2 + 3
Complete Design, continued
• Example of nonlinear practice effects:
 Suppose a participant figures out a method for
completing the task on the third trial, and then uses
the new method for subsequent trials
Trial 1 Trial 2 Trial 3 Trial 4 Trial 5 Trial 6
Condition
A
B
C
C
B
A
Practice Effects +0
+1
+5
+5
+5
+5
Practice effects are not balanced across the conditions:
A: 0 + 5 = 5
B: 1 + 5 = 6
C: 5 + 5 = 10
Complete Design, continued
• Nonlinear practice effects create a
confounding
 differences in scores on the DV may not be caused
by the IV (conditions A, B, C)
 differences on DV may be due to different amounts
of practice effects associated with each condition
• ABBA counterbalancing should not be used
 when practice effects are likely to vary or change
over time (i.e., nonlinear practice effects)
 use block randomization instead
Complete Design, continued
• ABBA counterbalancing should not be used
when anticipation effects can occur
 participants develop expectations about which
condition will appear next in a sequence
 responses may be influenced by expectations
rather than actual experience of each condition
 if anticipation effects are likely, use block
randomization
Incomplete Design
 Each participant experiences each
condition of the experiment exactly once
• complete design: more than once
 Practice effects are balanced across
participants in the incomplete design
• complete design: practice effects balanced
within each subject
Incomplete Design, continued
 General rule for balancing practice effects
• each condition (e.g., A, B, C) must appear in
each ordinal position (1st, 2nd, 3rd) equally
often
• if this rule is followed, practice effects
 will be balanced across conditions
 will not confound the experiment
Incomplete Design, continued
 Two techniques for balancing practice
effects in an incomplete repeated
measures design
• all possible orders
• selected orders
Incomplete Design, continued
 All possible orders
• use when there are four or fewer conditions
• two conditions (A, B) → two possible orders: AB, BA
 half of the participants would be randomly assigned to do
condition A first, followed by B
 other half of participants would complete condition B first,
followed by A
• three conditions (A, B, C) → six possible orders:
ABC, ACB, BAC, BCA, CAB, CBA
 participants would be randomly assigned to one of the six
orders
Incomplete Design, continued
• four conditions (ABCD) → 24 possible orders
(ABCD, ABDC, ACBD, ACDB, ADBC, etc.)
• five conditions → 120 possible orders
• six conditions → 720 possible orders
• at least one participant must receive each
order of the conditions
 therefore, all possible orders is used for
experiments with four or fewer conditions of the IV
Incomplete Design, continued
 Selected orders
• select particular orders of conditions to
balance practice effects
• two methods
 Latin Square
 random starting order with rotation
• each condition appears in each ordinal
position exactly once
• each participant is randomly assigned to one
of the orders of conditions
Incomplete Design, continued
• Procedure for Latin Square
 randomly order the conditions of the experiment
(e.g., ABCD)
 number the conditions (A = 1, B = 2, C = 3, D = 4)
 use this rule for generating the 1st order:
1, 2, N, 3, N – 1, 4, N – 2, 5, N – 3, 6, etc.
where N = last number of conditions
• the first order of four conditions would be 1 2 4 3
Incomplete Design, continued
 to generate 2nd order of conditions add 1 to each
number in the first order (1 2 4 3)
• “N” represents the number of conditions (e.g., 4); we
can’t use N + 1 because this would create a 5th condition
• additional rule: N + 1 always is “1” -- the first condition
• the second order of conditions is 2 3 1 4
 to generate 3rd order of conditions add 1 to each
number in the second order (again, N + 1 = 1)
• the third order of conditions is 3 4 2 1
 follow the same procedure for each subsequent
order
• The number of orders is the same as the number of
conditions (e.g., 4 conditions → 4 orders)
Incomplete Design, continued
 Match letters of conditions to their numbers to
create the Latin Square
1st
1
2
2
3
3
4
2nd 3rd
3rd
4th
4th
1st
2nd
4
3
A
B
D
C
1
4
2
B
1
C
C
A
D
D
B
3
2
D
A
C
4
A
1
Incomplete Design, continued
• Each condition appears in each ordinal
position equally often, which balances
practice effects
 For example, condition “A” appears in each ordinal
position:
1st
2nd 3rd
4th
A
B
D
C
B
C
A
D
C
D
B
A
D
A
C
B
Incomplete Design, continued
• Another advantage of Latin Square
 each condition precedes and follows every other
condition once (e.g., AB and BA, BC and CB)
1st
2nd 3rd
4th
A
B
D
C
B
C
A
D
C
D
B
A
D
A
C
B
 this helps to control for potential order effects
Incomplete Design, continued
 Random starting order with rotation
• generate a random order of conditions (e.g., ABCD)
• rotate the sequence by moving each condition one position to
the left each time
1st
2nd
3rd
4th
A
B
C
D
B
C
D
A
C
D
A
B
D
A
B
C
 each condition appears in each ordinal position to balance
practice effects
 unlike Latin Square, order of conditions is not balanced
Data Analysis
of Repeated Measures Designs
 Complete repeated measures designs
require an additional step
• Because participants complete each condition
many times, a summary score (e.g., mean) is
computed for each participant for each
condition
• this represents each participant’s average
performance in each condition
Data Analysis, continued
• Suppose
 two participants complete two conditions (A, B) of
an experiment four times each
 the DV is their rating on a 1–5 scale
 assume IV (conditions A, B) represents two types
of stimuli participants are asked to judge (e.g., size
of the stimuli)
Data Analysis, continued
 Suppose the following data are observed in an
ABBA design:
Condition
A
B
B
A
B
A
A
B
Participant 1
2
4
5
1
3
1
2
5
Participant 2
1
4
4
1
5
2
3
5
Data Analysis, continued
 To analyze these data we first need to compute the average
rating for each condition (A, B) for each participant:
Condition
A
B
B
A
B
A
A
B
Participant 1
Participant 2
2
4
5
1
3
1
2
5
A: (2+1+1+2)/4 = 1.50
B: (4+5+3+5)/4 = 4.25
1
4
4
1
5
2
3
5
A: (1+1+2+3)/4 = 1.75
B: (4+4+5+5)/4 = 4.50
Data Analysis, continued
 Next calculate the mean for each condition across
all participants
 In this example with two participants, the means
for conditions A and B are
Condition A
participant 1 1.50
participant 2 1.75
mean 1.625
Condition B
4.25
4.50
4.375
 Null hypothesis testing or confidence intervals
would be used to determine whether this difference
between means is reliable
The Problem of Differential Transfer
 Repeated measures designs should not
be used when differential transfer is
possible
• occurs when the effects of one condition
persist and affect participants’ experience of
subsequent conditions
• use independent groups design instead
• assess whether differential transfer is a
problem by comparing results for repeated
measures design and random groups design
Comparison of Two Designs
 Differences between repeated measures design
and independent groups design
• Independent variable
 repeated measures: each participant experiences every
condition of the IV
 independent groups: each participant experiences only one
condition of the IV
• What is balanced (averaged) across conditions to rule
out alternative explanations for findings?
 repeated measures: practice effects
 independent groups: individual differences variables