Transcript Modeling Space Fotress: CMU Effort

Modeling Space Fortress
Abraham Anderson
John Anderson
Shawn Betts
Dan Bothell
Jennifer Ferris
Jon Fincham
Michelle Moon
Ben Poole
Yulin Qin
The Decomposition Hypothesis
•Performance on complex tasks can be decomposed into primitive sets of mental
operations of the sort revealed in simple laboratory experiments.
•This assumption underlies much of the work on latency in basic neuroimaging
analyses.
•Nonetheless there has been very mixed evidence on part-to-whole training.
•This is generally thought to reflect that fact that one cannot use “pure insertion”
because the one task will change when you insert another.
•Most sophisticated analyses of reaction time and imaging rely on some sort of
“additive factors” where you keep all the processes but just make them more or
less difficult.
•Still within the context of a cognitive model one ought to be able to decide when
pure insertion will work and when it will not.
Pure Insertion in Space Fortress
•Insert Mines or Fortress into game and check brain pattern
•Naïve Insertion: Full Game = Fortress+Mines
•However, this ignores the fact that we are really getting a double count of
navigation and visual processing on the right hand side.
-Mines
+Mines
-Fortress
Orbit
Mines
+Fortress
Fortress
Both
•Sophisticated Insertion: Orbit+Both=Mines+Fortress
•However, this ignores that there are points in time when one cannot do both
fortress & mines in the both condition.
•One needs a model that will perform all of the conditions and look at its
predictions for these conditions.
Game structure
Minute 1
Fixation
3 IFF letters
Condition
Game Starts
Maybe Mine
Minute 2
Maybe Mine
Maybe Mine
Maybe Mine
Minute 3
Maybe Mine
Maybe Mine
Maybe Mine
Minute 4
Maybe Mine
Game Ends
Feedback
• Played 16 games per session
– Four of each condition
• 10 sessions 2/day
• Sessions 2 and 10 with fMRI
– Analyzed the 20s “Maybe mine” segments
Fixation
Changes from “regular” SF
•
•
•
•
•
Created the separate conditions
Use a keyboard instead of joystick
Fixed timing for mine onsets
No need to choose between points and shots
Fortress vulnerable while mine on screen
Points and Point Additivity
7000
6000
Points
5000
4000
3000
2000
Orbit
Mines
Mines+Fortress
1000
0
1
2
3
4
5
Fortress
Both
Orbit+Both
6
7
Sessions
•The major point growth is in the fortress conditions
•There is a necessary sub-additivity in points
8
9
10
Modeling Plan
• Model the well practiced performance
• Consider learning later
• Use participants fortress condition for sessions 7-9
as the basis for the model’s navigating
• Run model through all conditions
• Compare it to session 9 behavioral data and session
10 fMRI data
Flight data
speed
orbits
3
20
18
2.5
16
2
14
12
10
human
1.5
8
model
1
human
model
6
0.5
4
2
0
0
orbit
mines
fortress
orbit
full
mines
fortress
distance
160
140
120
100
80
human
60
model
40
20
0
orbit
mines
fortress
full
full
-50
thrust
right
left
iff
fire
shots
full
fortress
mines
orbit
full
fortress
mines
orbit
full
fortress
mines
orbit
full
fortress
mines
orbit
full
fortress
mines
orbit
full
fortress
mines
orbit
Keypresses per game
350
300
250
200
150
human
model
100
50
0
Keypress duration percentages
orbit thrust
mines iff
0.4
0.35
0.3
0.25
0.2
human
0.15
model
0.1
0.05
0
0
1
2
3
4
5
6
7
8
9
10
11
12
0.5
0.45
0.4
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0
human
model
0
fortress right
0.4
0.4
0.35
0.35
0.3
0.3
0.25
0.25
1
2
3
4
5
6
7
8
full fire
9
10
11
12
0.2
human
0.2
human
0.15
model
0.15
model
0.1
0.1
0.05
0.05
0
0
0
1
2
3
4
5
6
7
8
9
10
11
12
0
1
2
3
4
5
6
7
8
9
10
11
12
Scores
2500
2000
1500
human
model
1000
500
0
PNTS
CNTRL VLCTY
ORBIT
SPEED
PNTS
CNTRL VLCTY
MINES
SPEED
PNTS
CNTRL VLCTY
FORTRESS
-500
• Mine + Fortress = 7373.425 > Orbit + Full = 6959.95
SPEED
PNTS
CNTRL VLCTY
FULL
SPEED
Model total scores
Changes to ACT-R
• Special device for interfacing with the game
• Using the Temporal module
• Modified
– Motor module
– Visual module
– Imaginal module
Motor module changes
• Set motor feature prep time to 0ms
– Suggested by Wayne based on Kieras’s 2009 ICCM paper
• Added mostly independent hands
– Two execution paths
• Added individual finger queries
?finger-check>
left-index busy
• Added new motor actions
–
–
–
–
Hold-key
Release-key
Release-all-keys
Delayed-punch
Delayed-punch
• Holds a finger down for a prespecified time
+manual>
isa delayed-punch
hand left-or-right
finger finger-name
{delay time}
• Substitute for a strong visual-motor interface
• For the model (mostly) only a few times used
–
–
–
–
Orbiting 90ms, 125ms, 160ms
Firing 70ms, 90ms, 125ms
IFF 125ms, 160ms
Mine aiming special
Visual features
• Standard text items
– foes, conditions, bonus symbols, vlnr count, iff letter
• “Featureless” indicators – no information other
than presence or absence
– fortress, fortress hit, fortress explosion, ship
explosion, mine, mine explosion, trial over
• Ship
– Contains most of the critical information
Ship chunk
MY-SHIP
ISA SHIP
SCREEN-POS, VALUE, STATUS,
COLOR, HEIGHT, WIDTH, X, Y
ORIENTATION
DIST
VEL
ANGLE
VDIR
HEX-HIT
MINE-DIST
SHOOT-AT-MINE
HIT-FORTRESS-FIRST
New visual request
• Attend-and-track
– Combines the separate move-attention and starttracking requests into one action
New visual buffer
• Visual-search
– A search buffer
– Makes visicon chunks available for searching
in production matching
(p start-playing
=goal>
isa goal
state start-playing
• Visual-location information only
=visual-search>
isa sf-visual-location
kind ship
– Similar to the old LHS !find-location!
• Less flexible than a visual-location request
– No :attended, :nearest, or current
?visual>
state free
==>
+visual>
isa attend-and-track
screen-pos =visual-search
…
• Two primary reasons for adding it
– Buffer stuffing not available due to tracking
– Allows for modeling control over “stuffing”
• Utility can decide which feature is more
important when co-occurrences
)
Using the visual-search buffer
(p detect-bonus
=goal>
isa goal
check-bonus nil
=visual-search>
isa sf-visual-location
kind new-bonus
=imaginal>
isa game-state
==>
=imaginal>
+goal>
check-bonus check)
(p detect-hit
=visual-search>
isa sf-visual-location
kind fortress-hit
?imaginal>
state free
=imaginal>
isa game-state
shot-count =count
< shot-count 10
counted nil
==>
!bind! =new-count (1+ =count)
+imaginal>
shot-count =new-count
counted t)
(p see-mine-onset
=goal>
isa goal
state play
=visual-search>
isa visual-location
kind mine
=visual>
isa ship
?imaginal>
state free
…
==>
+imaginal-action>
isa generic-action
action predict-mine-shooting
…)
Special Imaginal request
• Prediction for shooting the mine
– 250ms, slightly more than a standard imaginal request
– Uses existing imaginal-action buffer
+imaginal-action>
isa generic-action
action predict-mine-shooting
• Determines through simulation how to best shoot at the mine
given current ship details
MINE-PREDICT
ISA MINE-HEADING
ACTION TURN
FINGER RING
DELAY 0.099
DIST 260.8793
HIT-DIST 180.8793
Mine aiming productions
(p adjust-mine-heading-with-fortress
=goal>
isa goal
cond2 "+fort“
=imaginal>
isa mine-heading
action turn
finger =finger
delay =delay
dist =dist
=visual>
isa ship
<= mine-dist =dist
=visual-search>
isa visual-location
kind mine
…
==>
+manual>
isa delayed-punch
hand left
finger =finger
delay =delay
…)
(p reassess-mine
=goal>
isa goal
=visual-search>
isa visual-location
kind mine
=imaginal>
isa mine-heading
action drift
hit-dist =delay
=visual>
isa ship
< mine-dist =delay
…
==>
+imaginal-action>
Isa generic-action
action predict-mine-shooting
…)
Model overview
• 77 productions covering eight basic tasks
–
–
–
–
–
–
–
–
Encoding pre-trial info and rehearsal
Attend ship then initial thrust and turn to start orbiting
Orbiting the center hex
Shoot at fortress and count hits
Encode and respond to bonuses
Identify mine, aim, and shoot at it
Correct orbiting problems (hit hex or too far out)
Detect ship destruction or trial over
Orbiting productions
• Model tries to stay aimed at the center and
orbit clockwise about 95 pixels out
• Broken into groups by ship speed
< 1.0 tries to speed up
1.0 – 1.7 maintains the normal orbiting
> 1.7 tries to slow down
Normal speed
(p right-norm-short (p right-norm-default (p right-norm-long (p thrust-norm-default (p left-norm-default
=goal>
=goal>
=goal>
=goal>
=goal>
isa goal
isa goal
isa goal
isa goal
isa goal
state play
state play
state play
state play
state play
dont-turn nil
dont-turn nil
dont-turn nil
=visual>
dont-turn nil
=visual>
=visual>
=visual>
isa ship
=visual>
isa ship
isa ship
isa ship
isa ship
<= vel 1.7
<= vel 1.7
<= vel 1.7
<= vel 1.7
<= vel 1.7
>= vel 1.0
>= vel 1.0
>= vel 1.0
>= vel 1.0
>= vel 1.0
>= angle 5
<= angle 10
> vdir 92
?manual>
preparation free
?finger-check>
left-index free
==>
+manual>
isa delayed-punch
hand left
finger index
delay fast)
>= angle 5
<= angle 20
> vdir 88
?manual>
preparation free
?finger-check>
left-index free
==>
+manual>
isa delayed-punch
hand left
finger index)
>= angle 15
<= angle 30
> dist 93
>= vdir 92
<= angle 2
>= angle -4
< angle -14
?manual>
preparation free
?manual>
preparation free
?finger-check>
?manual>
?finger-check>
left-index free
preparation free
left-ring free
==>
left-index free
+manual>
?finger-check>
left-middle free
isa delayed-punch left-middle free
==>
hand left
left-index free
+manual>
finger index
left-ring free
isa delayed-punch
delay slow)
==>
hand left
+manual>
finger ring)
isa delayed-punch
hand left
finger middle)
Consider the BOLD response
• Use the BOLD prediction tools built into ACT-R to
determine activity in the modules
• Few changes needed
– Made all goal and imaginal modifications in the model
+’s so they’re counted
– Added new buffers to track left and right hand actions
separately
• Only tracks the execution
– Modified visual tracking so that it is only periodically
busy instead of constantly
• One attention shift per 2 seconds
no mine no fortress
mine no fortress
no mine fortress
mine fortress
goal
3
visual
2.5
0.12
2
0.1
1.5
0.08
1
0.06
0.5
0.04
0
0.02
1
-0.5
2
3
4
5
6
7
8
9
10
0
1
2
3
4
5
6
7
8
9
10
retrieval
imaginal
0.3
0.04
0.25
0.03
0.2
0.02
0.15
0.1
0.01
0.05
0
1
0
-0.05
1
2
3
4
5
6
7
8
9
2
3
4
5
6
7
8
left-hand
right-hand
production
9 10
-0.01
10
0.4
0.5
0.3
0.4
0.4
0.3
0.3
0.2
0.2
0.2
0.1
0.1
0.1
0
0
1
1
2
3
4
5
6
7
8
9
10
-0.1
2
3
4
5
6
7
8
9
10
0
1
2
3
4
5
6
7
8
9
10
Orbit
Mines
Fortress
Manual
Vocal
LIPFC
1.00%
Anterior
Prefrontal
Posterior
Parietal
0.80%
Auditory
Fusiform
0.60%
0.40%
0.20%
0.00%
-0.20%
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
% Change from Baseline
Both
Left Motor Right Motor
ACC
Parietal
Prefrontal
Caudate
Fusiform
Orbit
Mines
Fortress
Both
model orbit
model mines
model fortress
model both
0.80%
0.60%
0.40%
0.20%
0.00%
-0.20%
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
% Change from Baseline
1.00%
Left Motor
Right Motor
ACC
Parietal
Prefrontal
Caudate
Fusiform
Model’s BOLD motor mismatch
Orbit
mines
900
800
700
600
500
400
300
200
100
0
model right
human right
model left
human left
1
2
3
4
5
6
7
8
9
900
800
700
600
500
400
300
200
100
0
10
model right
human right
model left
human left
1
2
3
4
5
6
fort
7
8
9
10
both
900
900
800
800
700
700
600
model right
600
model right
500
500
human right
400
human right
400
300
model left
300
model left
200
human left
200
human left
100
100
0
0
1
2
3
4
5
6
7
8
9
10
1
2
3
4
5
6
7
8
9
10
Curious…
right-hand
0.4
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0
1
2
3
4
5
6
7
8
9
10
fort
both
900
900
800
800
700
700
600
600
500
model right
model
right
500
400
400
300
human right
300
200
200
100
100
human
right
0
0
1
2
3
4
5
6
7
8
9
10
1
2
3
4
5
6
7
8
9
10
Future work
• Address the issues with the model’s BOLD
results in the motor system
• Model the performance on the early days and
the learning that takes place over days