Transcript Document

Toward quantifying the effect of
prior training on task performance
MURI Annual Review
September 26-27, 2006
Bill Raymond
Overview
• Project goal: Quantify the effects on performance of different
training methods for complex military tasks.
• Feature decomposition:
1. Task type
2. Training method
3. Performance assessment (context & measures)
4. Training principles
• Planning matrix:
- Capture where we know of, and can quantify in terms of
performance measures, effects of training method and
performance context on task components.
• Quantify principles:
- Derive performance functions for points in the feature space
using empirical data from laboratory tasks.
- Generalize performance functions for implementation in IMPRINT
modeling tool to simulate training effects on task performance.
Decomposition issues
• Constraints on decompositions
Features must relate to experimental designs
 Must be able to describe all experimental tasks.
 Task, training, and performance context features
can be no finer than experimental manipulations.
Features may be different for research and IMPRINT
 Can’t control training in the real world as carefully
as in the laboratory
 Not all experimental results will be major effects.
 IMPRINT task categories are already defined.
 Planning features should converge to final IMPRINT
features, diverging from research features
Planning matrix issues
• What will the matrix construction provide?
 Current
and planned research coverage of space
 May be used by us or others for future planning
 Approximation of final IMPRINT training features
 Initial step in determining the generality of
performance functions in the space
Starting point:
Analyzing training and performance
• Training variables - during skill learning:
 How was the skill taught?
 What kind of practice did learners get?
 How did practice relate to the way the skill
will be used?
Pedagogy
}
• Performance context variables - at skill use:
 How does expected performance relate to
training?
 How long has it been since training?
 Did learners get refresher training?
}
Practice
Performance
Task, training, and performance matrix
Training features
IMPRINT task taxons
Task components
Visual
Pedagogy Practice
Performanc
e context
Data entry
Data entry
Data entry
Information
processing
Data entry
Data entry
Data entry
Fine motor - discrete
Data entry
Data entry
Data entry
Numerical Analysis
Fine motor continuous
Gross motor - light
Gross motor - heavy
Communication
(reading & writing)
Communication (oral)
Pedagogy parameters
• Method
Instruction (=default)
Demonstration
Simulation
Discovery
Modeling/mimicking
Immersion
• Learning location (local = default, remote/distance)
• Discussion/Question and answer (no = default, yes)
• Individualization (no = default, yes)
Task by Pedagogy parameters
Pedagogy
Task components
IMPRINT task taxons
Method
Visual
Data entry (Instruction)
Numerical Analysis
Information
processing
Data entry
(Instruction)
Classificatio Inst/Discovery
n
Fine motor - discreteData entry
Fine motor - continuous
Gross motor - light
Gross motor - heavy
Communication (reading
& writing)
Communication (oral)
(Instruction)
Learning
location
Discussion
/Q&A?
Individualized?
Practice parameters
• Scheduling of trials and sessions
Number
Spacing
(massed = default, spaced, expanding/contracting)
Distribution (mixed = default, blocked)
• Scope of practiced task (partial, whole = default, whole +
supplemental)
•
•
•
•
•
•
Depth of processing (no = default, yes)
Processing mediation (no = default, yes)
Stimulus–response compatibility (yes = default, no)
Time pressure (no = default, yes)
Feedback - presence (no = default, all trials, periodic)
Context of practice
Distractor
(no = default, yes)
Secondary activity (no = default, yes)
Task by practice
IMPRINT task taxons
Task
components
Visual
Practice
Scheduling
Scope
Processing
depth
Part/
whole
Yes
(presentation
format)
Processing
mediation
Stimulusresponse
compatibility
Time
pressure
Feedback
Context
Yes
(response
&
accuracy)
Distractor/2nda
ry
activity (vocal
activity)
Yes
(response
&
accuracy)
Distractor/2nda
ry
activity (vocal
activity)
Data entry
Numerical
Analysis
Item
InformationData entry repetition,
# Sessions,
processing
Spacing
Data entry
Fine motor discrete
Fine motor continuous
Gross motor - light
Gross motor heavy
Communication
(reading & writing)
Communication
(oral)
Item
repetition,
# Sessions,
Spacing
Part/
whole
Yes (prior
knowledge)
No (Inputoutput
Format)
Performance context parameters
• Transfer
New context (relative to training)
New task (relative to training)
• Delay interval (default = none, time period)
• Refresher training (default = no, schedule)
Task by performance parameters
Performance context
IMPRINT task taxons
Task components
Data entry
Visual
New
context
New task
Delay
interval
Refresher
training
Data entry
Numerical Analysis
Information
processing
Data entry
Fine motor - discrete Data entry
Fine motor - continuous
Gross motor - light
Gross motor - heavy
Communication (reading &
writing)
Communication (oral)
Yes (typing
hand, output
configuration)
Yes
Yes (typing
hand, output
configuration)
Yes
Quantifying training principles
• Data Entry used as an example
• Consider two principles
Practice  Learning (Power law of practice)
 Skill practice - no item repetition
 Specific learning - item repetition
Prolonged work  Diminished performance
• Quantify effects for each taxon
Cognitive (“Information processing”)
Physical (“Fine motor - discrete”)
• …and performance context
Transfer to new items (similarity dimension)
Retention of learned skill (refresher training)
Skill practice: Quantifying learning
Total RT for 1 subject
5
y = -0.0005x + 2.6806
4.5
RT (sec)
4
3.5
3
2.5
2
1.5
1
0
100
200
300
400
500
600
Item
• Skill practice improves performance .5 msec/item
Mean decreases 300 msec in 640 (unique) items
• Where does skill practice come from?
Repetition of individual digits (and pairs of digits?)
Cognitive or physical learning?
Individual differences?
Skill practice: Origin or learning
Pair repetition?
Chunking effect
1.2
RT (sec)
1
0.8
0.6
0.4
0.2
0
Digit 1
Digit 2
Digit 3
Digit 4
Ente r
Keystroke
• Subjects appear to “chunk” digits 1 & 2, digits 3 & 4
so they may be learning something about pairs of digits
Skill practice: Origin of learning
Pair repetition?
Pair practice for one subject
5
y = 0.3473x + 2.478
Total RT (sec)
4.5
R2 = 0.0007
4
3.5
3
]
2.5
2
1.5
1
0.06
0.16
0.26
0.36
0.46
Pair repetition per item practiced
• Effect of 2-digit chunk practice appears minimal
Skill practice is general facility at number typing
Skill practice: Type of learning
Physical or cognitive?
Keystroke speed improvement
from Block 1 to Block 5
RT change (msec)
15
10
5
0
-5
Digit 1
Digit 2
Digit 3
Digit 4
Enter
-10
-15
-20
-25
Keystroke
• Speed improvement occurs on digits 1 and 3
Learning is cognitive
Skill practice: Individual differences
Component times for chunkers and nonchunkers
1.4
Digit 1
1.2
RT (sec)
1
0.8
Digit 3
0.6
Digit 2
Digit 4
0.4
Ente r
0.2
0
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
Ke y strok e a nd block
• “Chunkers” are 15% slower than “non-chunkers”
Appears to be a strategy choice
Pedagogy - advantage for instruction over
“discovery”?
Specific learning: Quantifying learning
Total RT for one subject
Total RT for 1 subject
5
5
y = -0.0005x + 2.6806
4.5
4.5
Total
(sec)
RT RT
(sec)
44
3.5
3.5
33
2.5
2.5
22
1.5
1.5
1
1
0
0
100
100
200
200
300
400
300
500
400
600
500
Item
Ite m
• Repetitious practice improves performance faster initially
Power law of practice
General learning functions
Total RT (sec)
Total RT
.
.
.
?
0
0
Item
• Performance as a function of number of repetitions
Planned experiment
General learning functions
Total RT
Total RT (sec)
New items?
Old items?
...
Learning
0
Transfer
0
Ite m
• Transfer and retention
Planned experiment
Retention
Prolonged practice
Accuracy decline with prolonged skill practice
0.12
0.115
Error rate
0.11
0.105
0.1
0.095
0.09
0.085
0.08
1
2
3
4
5
Block
• Prolonged work results in an increase in errors
Accuracy rate decline of about 1% over 320 items
• Where does the decline in accuracy originate?
Cognitive or physical fatigue?
Prolonged practice:
Type of performance decline
• Two types of errors:
Stimulus adjacency errors: 1234  1244
Key adjacency errors:
1234  1264
• 90% of errors are of these two types
• Origin of errors
Stimulus adjacency = cognitive
Key adjacency =motor phase, which could be motor
output planning (cognitive) or motor execution
(execution)
Prolonged practice:
Type of performance decline
Accuracy for two errors types
(no feedback)
Stimulus adjacency errors
16
Key adjacency errors
Number of errors
14
12
10
8
6
4
2
0
0
10
20
30
40
50
60
Items (/10)
• Practice results in an increase in key adjacency errors
Accuracy decline occurs during the motor phase
(which may be both cognitive and physical)
Prolonged practice:
Type of performance decline
Accuracy for two errors types
(with feedback)
16
Stimulus adjacency errors
key adjacency
Number of errors
14
12
10
8
6
4
2
0
0
10
20
30
40
50
60
Items (/10)
• Feedback eliminates the speed-accuracy trade-off
If feedback is cognitive, then the relevant
processes in the motoric phase must be cognitive
Summary
IMPRINT task taxons
Task
components
Training features
Pedagogy
Method:
Practice
Performance
context
Scheduling:
Transfer:
• no reps - speed decrease linear (.5 Retention:
• Instruction strategy
msec/item)
(planned
instruction may • item reps - power law (parameters to experiment)
Information improve speed be determined)
processing • “Discovery” - Feedback:
(Cognitive) some Ss 15% • no feedback - accuracy decline
slower
(1%/300 items)
• typing/accuracy feedback - no
decline
Transfer:
Retention:
Fine motor discrete
(Physical)
(planned
experiment)