Transcript How does the brain decide where to look next?

How does the brain decide
what to look at next?
John Findlay
University of Durham
(acknowledgements to Val Brown)
Saccades are quintessentially voluntary movements.
The gaze selects informative detail but eye scans also
appear random and arbitrary
from Yarbus (1967)
Pathways involved in saccade
generation
Visual pathways form a massively
interconnected neural network
Retinotopic mapping is maintained through to
the saccadic generator in the SC
Multiple interconnected maps
offer the possibility of selection
by biased competition
“ Some kind of short-term description of the
information currently needed must be used to
control competitive bias in the visual system,
such that inputs matching that description are
favoured in the visual cortex.”
(Desimone and Duncan, Annual Review of Neuroscience, 1995)
Biased competition creates
salience maps
•
A salience map is a two-dimensional map in which
a single scalar quantity (salience) is represented at
each point.
•
Biased competition results in similarity to the
search target being represented as salience.
•
Instantiated in various models
Itti L and Koch C (2000). A saliency-based search mechanism for
overt and covert shifts of attention. Vision Research, 40, 1489-1506.
Hamker FH (2004). A dynamic model of how feature cues guide
spatial attention. Vision Research, 44, 501-521.
Saccade target selection in the
superior colliculus
The SC is the main final site for
selection of saccade destinations
Activity in the superior colliculus related to saccades
Wurtz R H (1996). Vision for the control of movement. The Friedenwald
Lecture Investigative Ophthalmology and Visual Science, 37, 2131-2145.
How does the brain decide
what to look at next?
•
During visual search, biased competition
creates a salience map and processes,
probably in the SC, select the point of
highest salience to convert to an orienting
saccade.
• Supported by detailed studies of saccades during
visual search (Findlay, Vision Research, 1997; Motter
and Belky, Vision Research, 1998a,b)
Task:
Search for a
red cross
(Look at it)
Findlay J M (1997). Saccade target selection during visual search. Vision Research, 37, 617-631
Properties of first saccades
(Findlay, 1997)
Short latency (~ 250 ms) – very similar for
saccades to target and to distractor
Properties of first saccades
(Findlay, 1997)
Short latency (~ 250 ms) – very similar for
saccades to target and to distractor
Frequently (75%) on target when target is
in inner ring, occasionally (26%) when
target in outer ring.
.
Properties of first saccades
(Findlay, 1997)
Short latency (~ 250 ms) – very similar for
saccades to target and to distractor
Frequently (75%) on target when target is
in inner ring, occasionally (26%) when
target in outer ring.
Incorrect saccades go preferentially to
distractor sharing a feature with the
target.
Monkey visual search
(Motter & Belky, Vision Research, 38, 1007-1022; 1885-1815, 1998)
• Monkeys trained to search for a conjunction target
(colour and orientation)
Saccade selection in visual search
•
The conclusion in both the Findlay and the Motter &
Belky studies was that the biased competition/
salience map approach provided the most
satisfactory account of saccadic selection.
• In particular, no evidence for a rapid covert
attentional scan (favoured by many psychologists).
• This conclusion was reached earlier in physiological
studies of single cell responses in the visual
system of primates carrying out search tasks
FEF
IT
Schall & Hanes (1993)
Chelazzi, Miller, Duncan & Desimone (1993)
How does the brain decide what to
look at next during visual search ?
•
During visual search, biased
competition creates a salience map
and processes, possibly in the SC,
select the point of highest salience to
convert to an orienting saccade.
How does the brain decide
where to look next?
•
Selection from a salience map is basic
•
Supplementary processes
1. Inhibition of return
4.
Implicit learning
5.
Contingent learning
2. Saccade pipelining
6.
Neuro-economics (Glimcher)
7.
Task specific requirements for
3. Strategies
information acquisition
(Land, Hayhoe, Ballard)
How does the brain decide
where to look next?
•
Selection from a salience map is basic
•
Supplementary processes
1. Inhibition of return
4.
Implicit learning
5.
Contingent learning
2. Saccade pipelining
6.
Neuro-economics (Glimcher)
7.
Task specific requirements for
3. Strategies
information acquisition
(Land, Hayhoe, Ballard)
NO OTHER ATTENTIONAL SELECTION
•
Selection from a salience map is basic
How does the system avoid ‘salience loops’?
(B is the most salient location when A is fixated, then A
becomes the most salient when B is fixated)
Salience map alone would give A > B > A > B . . . . .
Proposed answer - Inhibition of Return (IOR)
An attended location is subject to some form of inhibition
when attention is shifted elsewhere
Klein R M and MacInnes W J (1999). Inhibition of return is a foraging facilitator in visual search.
Psychological Science, 10, 346-352
The rings task
Scan order partly
specified
Centre - red – free
scan through
blacks - blue
Count target letters
and make
Yes/No response
The Rings Task
3
6
9
12
Scan through the rings, starting with the red and
ending with the blue
(Scans from 6 individuals)
Rings task - typical eye scan
Deviations from sequential scan
BACKTRACK 1
OMISSION
BACKTRACK 1
BACKTRACK 1
BACKTRACK 1
BACKTRACK 2
Trials with deviations from sequential scan
Omissions
Backtrack 1
Backtrack 2
Backtrack > 2
BR
15
15
1
15
JP
9
15
6
9
LS
6
12
3
4
LW
5
29
5
4
PB
5
39
5
10
SL
3
13
7
13
Proportion
0.09
0.26
0.06
0.11
Error rate*
47 %
6%
15 %
23 %
(target omissions)
* Error rate on trials with standard scan
4%
Backtracking in visual search
BACKTRACK 1
Found by other workers
BACKTRACK 1
(Motter & Belky, 1998)
(Peterson et al. 2001)
BACKTRACK 1
• IOR time course may relate to visual processing
slower with increased processing demands, so not always immediate
• Backtracking sequences may be pre-planned
(pipelined saccades)
In A1, B, A2 fixation sequences, the B fixation was normal duration (228 ms)
but A1 and A2 were both shorter than normal (~ 170 ms).
The saccade following a backtracking sequence tended to follow the
direction of the last saccade in the sequence.
Backtracking in visual search
BACKTRACK 1
X
BACKTRACK 1
Found by other workers
(Motter & Belky, 1998)
(Peterson et al. 2001)
BACKTRACK 1
• IOR time course may relate to visual processing
• Backtracking sequences may be pre-planned
(pipelined saccades)
In A1, B, A2 fixation sequences, the B fixation is normal duration but A1 and
A2 are both shorter than normal
The saccade following a backtracking sequence tends to follow the
direction of the last saccade in the sequence
Visually-guided and memoryguided saccades
How does the brain decide which?
Hikosaka et al (2000) argue for basal
ganglia pathway (blue route)
Inhibitory effects on SC, others
excitatory
Separate sets of cells in caudate
and in SNr are active during visually
guided and memory guided
movements.
Directional strategies
(Convex Hull)
COUNT
THE
DOTS
Directional strategies
COUNT
THE
DOTS
Directional strategies are one form of memory (Gilchrist & Harvey)
NO OTHER ATTENTIONAL SELECTION
‘Visual attention selects the saccade target’
Statement supported by the finding that visual
information at the destination point of a forthcoming
saccade receives preferential pre-processing
(Deubel and Schneider, 1996; Kowler et al. 1995)
Biased competition is a form of attentional selection
but does not operate in a localised region of the
visual field.
Visual attention is commonly thought of as selection
of a localised region.
‘Visual attention selects the
saccade target’
Why I don’t like this statement
1.
It’s getting close to a homunculus view
2.
Localised visual attention should be able to
eliminate distractor interference.
Visual attention should be able to
eliminate distractor interference.
next
saccade ?
Visual attention should be able to
eliminate distractor interference.
Attentional spotlight
selects next target
Visual attention should be able to
eliminate distractor interference.
Attentional spotlight
selects next target
Visual attention should be able to
eliminate distractor interference.
Attentional spotlight
selects next target
No effective spotlight
Visual attention should be able to
eliminate distractor interference.
Attentional spotlight
selects next target
Global effect
No effective small spotlight
The Global Effect
Saccades to neighbouring target pairs tend to land towards
a centre-of-gravity position.
Findlay, 1981, 1982; Deubel, 1982; Ottes, Van Gisbergen and Eggermont, 1984
Reliably found with onset stimuli: does it occur in free scanning?
How accurate are scanning
saccades ?
SACCADE
TO CENTRE
OF GRAVITY ?
Is there a global effect in free
scanning ?
• Are saccades less
accurate when there is
a distractor present in
the critical sector (as
defined by Walker et
al. 1997)?
Subject
No
distractor
With
distractor
JP
16.9
43.3
LW
14.9
36.7
PB
20.4
38.0
SL
22.8
55.1
Percentage of inaccurate saccades
Probability of inaccurate saccade
for distractors in different
locations relative to
the target
Is accuracy higher follower longer fixations ?
JP
SL
PB
LW
mean
accuracy
coding
Short
Medium
Long
(< 200 ms) (200 – 300 ms) (> 300 ms)
2.00
2.17
1.86
2.60
2.47
2.19
2.66
2.42
2.33
2.55
2.26
2.49
2.45
2.33
2.22
No
distractor
2.73
2.95
3.01
2.84
2.88
Accuracy is highest following short
duration fixations (although distractors
still decrease it).
1
4
3
2
This is the opposite to a speed-accuracy
trade off.
Attentional selection and
saccades
• Saccades during a free scan of a set of
identical elements show the global effect.
Thus no evidence here for a spotlight-like
attentional selection.
• In most practical situations, elements are not
identical; hence biased competition will act to
reduce the global effect
How does the brain decide
where to look next?
•
Selection from a salience map is basic
•
Supplementary processes influencing
salience
1. Inhibition of return
2. Saccade pipelining
3. Strategies
4. Implicit learning
5. Etc. etc. etc.
THE END
Thank you for your attention
Kowler et al (1995)
Do distractors that have been already scanned
reduce accuracy ?
accuracy
coding
JP
SL
PB
LW
Mean
1
4
3
2
no
distractor
scanned
distractor
new
distractor
2.73
2.95
3.01
2.84
2.88
2.06
2.41
2.48
2.37
2.33
2.13
2.30
2.63
2.48
2.38
Accuracy is reduced both by scanned
distractors and by new ones.
How replicable are scanning patterns ?
(repeat run with one subject - different trial order)
REPLICA TRIALS
NON REPLICA TRIALS
NEAR REPLICA TRIALS
REPLICAS
26% overall, 68% ring count 3
REPLICAS and NEAR REPLICAS 45% overall
Replicability of directional selection
Saccade direction histograms
Heuristic scanners
Strategic scanners
Saccade direction change histograms
Saccade Landing Points
SACCADES WITH NO DISTRACTOR IN SECTOR
Subject PB
Accuracy is largely independent of saccade size
Saccade undershoot
10%
5%
1%
Saccade variability : on-axis
10%
10%
5%
5%
2%
1%
Saccade variability : off-axis
5%
2%
1%
Corrective saccades in the
multi-element scanning task
Oculomotor capture in the
multi-element scanning task