Transcript Two basic approaches to modeling human/system performance

The final resting place
for all this research…
Ron Laughery, Ph.D.
University of Colorado
Items to be covered…
• What is the problem this
research is trying to solve from
an operational perspective?
• What is the basic human
performance modeling and
simulation approach that this
research will feed?
• What is the specific tool and
architecture that we are
working to advance?
• What are the issues in moving
this research into practice?
What is the problem this research is trying to
solve from an operational perspective?
• The £5,000,000,000 question…
– In about 1995, Robin Miller, an operational
analyst with the MoD asked us this question
and made this statement at a meeting:
• “A question I get all the time can be summed up as this
– should we invest £5B in new kit, or should we instead
invest that £5B in training? If your models can’t help
me answer that question, you’re not doing your job.”
• We are trying to ensure that we are doing
our job in Mr. Miller’s eyes
What is the basic human performance
modeling and simulation approach that
this research will feed?
• In military and civilian systems, decisions are
increasingly being made on the basis of model
based analyses
– System effectiveness depends upon…
Humans
Hardware
Software
Two basic approaches to modeling
human/system performance
• Reductionist
– Breaking human activity and interaction
with the system into discrete activities
Advantages/disadvantages of
reductionist modeling approach
• Advantages
–
–
–
–
Intuitive
Level of detail determined by need
Basic data are usually available or easily obtained
Consistent with many military systems and
operational analysis models
• Disadvantages
– Often requires extensive subject-matter expert
input
Second approach to modeling
human/system performance
• First principled/cognitive models
– Based on theories of the underlying
mechanisms that facilitate human behavior
perception
Iconic
storage
Central
processing
Working
memory
Long-term
memory
Response
mechanisms
Example: ACT-R Representation
and Equations
W  S
B  ln  t
Activation Ai  Bi 
Learning
j
Intentions
ji
 A
j
d
j
i
Goal
Ti  F e Ai
Ui  Pi G  Ci   U
Succ i
Learning Pi 
Succ i  Faili
Retrieval
Productions
j
Latency
Memory
Visual
Manual
Utility
Motor
Vision
IF the goal is to categorize new stimulus
and visual holds stimulus info S, F, T
THEN start retrieval of chunk S, F, T
and start manual mouse movement
World
Size Fuel Turb Dec
Stimulus
Bi
Chunk
S 20 1
SSL
L 20 3
S13
Y
Advantages/disadvantages of
the first principle approach
• Advantages
– Requires less data input from either
experiments or subject matter experts
– More first-principle based and, if component
models are valid, easier to defend
• Disadvantages
– Model construction can be quite cumbersome
for simple tasks
– We don’t have enough real first-principle
models of human performance
A strategy that has worked- a
hybrid approach
• The flexibility of reductionist models combined
with the power of first principles of human
behavior is the formula for success
Iconic
perception storage
Central
processing
Working
memory
Long-term
memory
Response
mechanisms
Reductionist modeling with
Task Network Modeling
• Largely involves the extension of a task
analysis into a network defining sequencing
Going from a task network to a
running computer model
• Add timing information and task/system
interdependencies
Add human decision making
strategies
• Any defined branch point represents a
need for a decision
• Logic and rule sets, goal seeking,
naturalistic
?
Then, develop a scenario, equipment
model and/or links to other simulations
Run the model to collect
human/system performance data
Combining First Principles of human
behavior with Task Network Models
• For the past 16 years, we have been embedding and linking
first principle models of human performance into our tools
including
–
–
–
–
–
–
–
–
–
Cognitive workload and human response
Micro models of human time and accuracy
Human error and system response to error
Performance shaping factor effects
Linkage to anthropometric, biomechanical models
Goal driven task scheduling
Naturalistic Decision Making
Situation awareness modeling
Integration of cognitive engineering models such as ACT/R
• Predicting training effects is still the
weakest link!
Improved Performance Research
Integration Tool (IMPRINT):
Capability and Application
What Does IMPRINT Do?
It helps you...

Set realistic system requirements
 Identify future manpower & personnel constraints
 Evaluate operator & crew workload
 Test alternate system-crew function allocations
 Assess required maintenance manhours
 Assess performance during extreme conditions
 Examine performance as a function of personnel
characteristics, training frequency & recency
 Identify areas to focus test and evaluation resources
IMPRINT Architecture Operations Modeling
Improved Performance Research
Integration Tool (IMPRINT)
IMPRINT Architecture Maintenance Modeling
Send systems on missions as defined by scenario
Simulate need for maintenance
corrective
& continue
mission
corrective
& stop
mission
combat
damage
preventive
Repair systems
Repair Parts
Systems Ready
for Next Mission
Manpower Pool
Who Has IMPRINT?
 Army
 Navy
 Air
Force
 Other Government
 Contractors
 University
108
23
9
12
108
19
279 and growing
Mental Workload
Mission Tasks
1. monitor
alarms
Which Brain
Resources
Involved?
Degree of Resource Use?
Visual
Visual
Auditory
Psychomotor
2. decide
response
action
Cognitive
3. pull trigger
Auditory
.
.
.
n. task n
Cognitive
0.0
1.0
1.2
3.7
4.6
5.3
6.8
Psychomotor
7.0
- No Cognitive Activity
- Automatic (simple association)
- Alternative Selection
- Sign/Signal Recognition
- Evaluation/Judgment (consider
single aspect)
- Encoding/Decoding, Recall
- Evaluation/Judgment (consider
several aspects)
- Estimation, Calculation,
Conversion
Current IMPRINT Implementation:
Stressors by Task Type
MOPP Heat Cold Noise Sleepless Hours
Task type (Taxon*)
Visual
T
A
T
Numerical
A
TA
Cognitive
A
TA
Fine Motor Discrete
T
A
T
Fine Motor Continuous
Gross Motor Light
T
T
Gross Motor Heavy
Commo. (Read & Write)
A
Commo. (Oral)
T
A
A
T = affects task time, A = affects task accuracy, TA= affects both
* O’Brien, L. H., Simon R. and Swaminathan, H. (1992). Development of the Personnel-Based System
Evaluation Aid (PER-SEVAL) Performance Shaping Functions. ARI Research Note 92-50
Approach to modeling human
response to stressors
P
e
r
f
o
r
m
a
n
c
e
m
u
l
t
i
p
l
i
e
r
s
a
s
a
f
u
n
c
t
i
o
n
o
f
t
i
m
e
s
i
n
c
e
s
l
e
e
p
p
e
r
f
o
r
m
a
n
c
e
m
u
l
t
i
p
l
i
e
r
1
.
2
The general
effects...
1
0
.
8
a
t
t
e
n
t
i
o
n
p
e
r
c
e
p
t
i
o
n
c
o
g
n
i
t
i
o
n
0
.
6
p
s
y
c
h
o
m
o
t
o
r
0
.
4
p
h
y
s
i
c
a
l
0
.
2
0
6
1
2
1
8
2
4
3
0
3
6
4
2
4
8
5
4
6
0
6
6
7
0
t
i
m
e
s
i
n
c
e
s
l
e
e
p
On a specific Detect ring - 50% attention, 50% perception
Select menu item using a mouse - 40% attention, 60% psychomotor
task….
Interpret customer’s request for information - 100% cognitive
 attention performance multiplier = .82
 perception performance multiplier = .808
 cognition performance multiplier = .856
 psychomotor performance multiplier = .784
task time = 112.3% of normal
Time to Prepare to Engage
at 20 Hours Since Sleep
0.08
-
physical performance multiplier = .727
0.07
0.06
Relative Freq.
under specific
conditions...
leads to this specific
effect at this time…..
0.05
preptime
0.04
0.03
0.02
0.01
0
0
5
10
15
Tim e, sec
20
25
Use task network models to
study aggregate effects of PSFs
Time to Prepare to Engage
at 20 Hours Since Sleep
0.08
0.07
0.07
0.06
0.06
0.05
preptime
0.04
0.03
Relative Freq.
Relative Freq.
Time to Prepare to Engage
at 20 Hours Since Sleep
0.08
0.05
0.03
0.02
0.01
0.01
0
0
0
5
10
15
Tim e, sec
20
25
preptime
0.04
0.02
0
5
10
15
20
25
Tim e, sec
Time to Prepare to Engage
at 20 Hours Since Sleep
0.08
0.07
Relative Freq.
0.06
0.05
preptime
0.04
0.03
0.02
0.01
0
0
5
10
15
20
25
Tim e, sec
Net Performance Op 6
Time to Prepare to Engage
at 20 Hours Since Sleep
0.08
Time to Prepare to Engage
at 20 Hours Since Sleep
120
Relative Freq.
0.06
0.07
0.06
0.05
preptime
0.04
0.05
preptime
0.04
18
0.03
0.02
0.03
0.01
0.02
100
0
0
0.01
5
10
15
Tim e, sec
0
10
15
20
20
25
16
25
Tim e, sec
14
80
12
10
60
8
Time to Prepare to Engage
at 20 Hours Since Sleep
0.08
40
0.07
0.06
6
0.05
preptime
0.04
0.03
0.02
4
0.01
20
0
0
5
10
15
20
25
Tim e, sec
2
0
0
Time (sec)
Utilization
5
Workload
0
Relative Freq.
Relative Freq.
20
0.07
0.08
What view of training is in
IMPRINT now?...
Performance
What we really need for a reasonably
accurate representation of training…
Classroom
training
Simulator
training
Field
training
No
training
Retraining
Amount of training received
• We need these functional relationships…
– For different task types (the taxonomy)
– For different “types” of training
Big questions…
• Purpose of models
– Design of optimal training systems
– Design of systems considering training
• Taxonomies
– Training environment
– Task type
• Scope/complexity of tasks studied
– Do small tasks scale to large tasks?
• How do we treat Retention