Transcript How We Cognize Space Zenon Pylyshyn Rutgers Center fir Cognitive Science Rutgers University, New Brunswick, NJ.

How We Cognize Space
Zenon Pylyshyn
Rutgers Center fir Cognitive Science
Rutgers University,
New Brunswick, NJ
How we cognize space
Is There an Image Space in the Head?


The most common approach to the
question of how we represent a spatial
layout is that we represent it in the form of
a mental image. The format of mental
images is supposed to be particularly
suited for representing spatial information.
The mind’s great illusion: That you see a
world inside your head when you imagine
 the “Picture Theory” of mental imagery
The four-part plan of this lecture:
1. First I will talk a little about “the imagery debate” and
introduce recent neuroscience evidence.
2. I will then focus on what I consider the core of the
debate: How mental images represent space.
3. Then I will talk about the special case of images that
are projected onto the perceived world.
4. To show the generality of projected images that I will
need to spend a few minutes to introduce the idea of
visual indexes or FINSTs, as a type of a deictic or
demonstrative reference.
5. Finally I will combine the idea of indexes with the
evidence on mental imagery and suggest how
generalized indexes may allow spatial properties of the
currently-perceived world to translate into apparent
spatial properties of images.
I. The imagery debate:
A capsule overview
The main question is whether thoughts
experienced as “mental images” or as “seeing
with the mind’s eye” are different from other
thoughts, and if so how.
 The dominant view is the “picture theory” of
mental images, which assumes that images
stored in a spatial medium and are examined
by the visual system the same way that the
original scene would be.

Does imagery use the visual
system and if so, what does that
tell us about the nature of images?



There is some evidence that the visual system is
“active” during imagery
This has led to the view that the visual system
must be examining some not-yet-interpreted
image, just as it was thought to do in visual
perception.
But the last step is unwarranted because even if
the visual system was involved, it would only
mean that both vision and imagery use some of
the same processes and the same kind of
representations, but neither need be pictorial.
Failure of the picture-theory in vision
In vision the picture theory was meant to explain why
our perception is panoramic and stable while the visual
inputs are highly local, partial and constantly changing
But the picture theory of vision has been thoroughly
discredited: There is no rich panoramic display in vision
(e.g., see change blindness, superposition studies, …)
The picture theory of vision
is a non-starter, even for cats
(Cartoon by Kliban)
A more plausible theory of vision
(even for cats)
II. The newest round of
the imagery debate


In recent years the picture theory has been revived,
due largely to two neuroscience findings:
1. The visual cortex (V1) is activated during imagery
2. The visual cortex is retinotopically organized (i.e.,
it appears to map the retina in a topographically
continuous or homeomorphic manner).
From this, people have concluded that mental
imagery uses a literal spatial display, located in V1.
The goal of neuroscience research on
mental imagery is to find a display of the
imagined pattern in visual cortex
We already know that
there is a topographical
projection of retinal
activity in visual cortex
The tool of choice has
been the use of brain
scans (esp fMRI, PET)
Tootell, R. B., Silverman, M.
S., Switkes, E., & de Valois, R.
L. (1982). Deoxyglucose
analysis of retinotopic
organization in primate striate
cortex. Science, 218(4575),
What do recent neuroscience results
tell us about mental imagery?
None of the brain-scan (fMRI, PET) results
supports the picture theory of mental
images for reasons that I will discuss next
1. Even if there is a 2D mapping of retinal
activity in visual cortex (V1), this should
not be identified with the mental image.
2. Patterns in V1 do not function the same
way as mental images for several reasons.
3. Even if dynamic 3D patterns were found in
V1 it would not explain most mental
imagery research findings.
The topographical structure of the visual
cortex could not support mental images
1. Even of there is a 2D mapping of retinal
activity in V1, this cannot be identified with
the mental image which is panoramic,
3-dimensional, dynamic and has many other
properties that could not be mapped onto V1,
so we would need a different theory for them.
Why activity in visual cortex could
not correspond to a mental image
2. Patterns in V1 are different from mental images:
a) Patterns in V1 are foveal and retinocentric while
mental images are panoramic and allocentric
b) There is no spontaneous 3D interpretation of
patterns in mental images <parallelogram example>
c) There is no amodal completion of patterns in
mental images <Kanizsa example>
d) Order of access of information in a mental image
is not free <name letters of a familiar word backwords>
e) Emmert’s law does not hold for images <unlike
afterimages>
f) There is no visual (re)interpretation of images
<Slezak example>
Why activity in visual cortex could
not correspond to a mental image
2. Patterns in V1 are different from mental images:
a) Patterns in V1 are foveal and retinocentric while
mental images are panoramic allocentric
b) There is no spontaneous 3D interpretation of
patterns in mental images <parallelogram example>
c) There is no amodal completion of patterns in mental
images <Kanizsa example>
d) Order of access of information in a mental image is
not free <letter reading example>
e) Emmert’s law does not hold for images
f) There is no visual (re)interpretation of images
<Slezak example>
Imagine two parallelograms (as
below) one above the another
Close your eyes and then imagine
these two parallelograms
Connect the corresponding top and bottom vertices
What do you see? Keep looking to see if anything changes
Did you see this? Did it flip?
Why activity in visual cortex could
not correspond to a mental image
2. Patterns in V1 are different from mental images:
a) Patterns in V1 are foveal and retinocentric while
mental images are panoramic allocentric
b) There is no spontaneous 3D interpretation of
patterns in mental images <parallelogram example>
c) There is no amodal completion of patterns in
mental images <Kanizsa example>
d) Order of access of information in a mental image is
not free <letter reading example>
e) Emmert’s law does not hold for images
f) There is no visual (re)interpretation of images
<Slezak example>
Amodal completion in imagery?
Amodal completion in imagery?
Why activity in visual cortex could
not correspond to a mental image
2. Patterns in V1 are different from mental images:
a) Patterns in V1 are foveal and retinocentric while
mental images are panoramic allocentric
b) There is no spontaneous 3D interpretation of
patterns in mental images <parallelogram example>
c) There is no amodal completion of patterns in mental
images <Kanizsa example>
d) Order of access of information in a mental image is
not free <letter reading example>
e) Emmert’s law does not hold for images
f) There is no visual (re)interpretation of images
<Slezak example>
Slezak figures
Pick one (or two) of these animals and
memorize what they look like. Now
rotate it in your mind by 90 degrees
clockwise and see what it looks like.
Rotated Slezak figures
 No subject was able to recognize
the mentally rotated figure
 Subjects remembered the figures
well enough so if they drew it they
could recognize the rotated figure
Even if patterns in visual cortex were
isomorphic to those in the mental
image, it still would not explain most
results of mental imagery research!
3. The reason that patterns of activation in
striate cortex would not explain most of
the results of mental imagery research is
that the results are largely cognitively
penetrable and therefore require the
appeal to knowledge, goals, utilities, etc
and inferences over them. In other words
they require a cognitive explanation.
Task Demands and the tacit
knowledge explanation
The task of “imagining X” is the task of
pretending that you are seeing X and simulating
as much of that event as seems relevant to the
task using your tacit knowledge about how the
event might unfold. The task also requires
certain other skills (e.g., estimating time-tocollision, generating time intervals, etc) but it
does not require that you use a spatial display.
 Examples…

There are many examples showing
that the result that was attributed to
the mental image format is actually
due to tacit knowledge



Color mixing example to illustrate the difference
between the two sources of observations <slide>
Imagine dropping weights from different heights
Mental Image size (It has been shown that it
takes longer to report small details from a
small image than from a large on. What does
this mean? What would you think if the
result showed the opposite?)

Mental scanning <example slide>
Color mixing example
Studies of mental scanning
A window on the mind?
2
1.8
1.6
Latency (secs)
1.4
scan image
imagine lights
show direction
1.2
1
0.8
0.6
0.4
0.2
0
Relative distance on image
(Pylyshyn & Bannon. See Pylyshyn, 1981)
Mental representation of space:
The core of the imagery debate


It seems to be almost impossible to deny that thinking
using mental images exploits spatial properties of images
in some important sense. In what sense? (Do images
“preserve metrical spatial properties”?)
It is always possible to encode spatial relations in any
form of representation that has a numeral system, so why
assume that the representation of space is itself spatial?
1) Phenomenology: we see things as “laid out in space”!
2) Psychophysical evidence from projected images
(illusions, S-R compatibility…)
3) Clinical evidence (visual/imaginal neglect)
Use of real visible space in
“projected” mental imagery

“Projected images” serve to directing attention and to
associate thoughts with selected visible objects. Examples:
 Robust version of mental scanning (scanning with eyes open)
 Visual illusions involving projected images <Bernbaum & Chung,
1981>
 Projected memory images act like displays <Podgorny & Shepherd>
 S-R Compatibility with images (Tlauka & McKenna, 1998)
 Visuomotor (prism) adaptation from mental images <Finke, 1980>
Shepard & Podgorny experiment
Both when the displays are seen and when the F is
imagined, RT to say whether the dot was on the F was
fastest when the dot was at the vertex of the F, then when
on an arm of the F, then when far away from the F – and
slowest when one square off the F.
S-R Compatibility Effect with display
S-R Compatibility Effect with Images
Might all spatial images
work like projected images?

There are three key ideas behind the proposal that
spatial mental images are the projection of the spatial
layout of imagined objects onto a perceived scene
1. Recognition that the spatial properties exhibited in
experiments with projected images depend only on the
location of a few items and not on other visual properties
2. The idea of a limited-capacity amodal indexing
mechanism or deictic reference: FINSTs and Anchors.
3. The idea of a primitive amodal spatial sense that allows
us to perceive and recall the location of objects in an
allocentric frame of reference, independent of the
objects’ perceptual properties or of sense modality, and
automatically updated by our movements
We don’t need a spatial display in our
head if we have the right kind of deictic
contact with real (perceived) space



None of the experiments that are alleged to show the
existence of a spatial display (in visual cortex) need to
appeal to anything more than a small number of
imagined locations. (e.g., Shepherd & Podgorny, Finke, Tlauka,…)
If we can index a small number of (occupied) locations
in real space (using FINSTs) we can use them to
allocate attention or to program motor commands.
If these indexed objects are also bound to objects of
thought this will result in our thoughts (i.e. images)
having persisting spatial relations.
Aside on Visual Index (FINST) Theory

FINSTs are direct, unmediated, nonconceptual connections
between objects in the world and mental symbols
 FINSTs serve as visual demonstratives (like “this” or “that”).


Such direct references are essential for solving the
correspondence problem in vision – especially in the case
of visual representations built up incrementally over
different glances or “noticings”.
Some instances where we need Indexes:
 Visual stability, recognizing n-place relations, subitizing, and
multiple-object tracking
Several objects must be picked out
at once in relational judgments

When we judge that certain objects are
collinear, we must have picked out the relevant
individual objects first.
Several objects must be picked out
at once in relational judgments

The same is true for other relational judgments
like inside or on-the-same-contour… etc. We
must pick out the relevant individual objects first.
A concrete demonstration of
what visual indexes can do
Multiple Object Tracking studies (MOT)
 Basic finding: People can track up to 5 individual
objects that do not have a unique description
 We have shown that it is unlikely that the tracking
is done by updating locations but rather that
individuating and keeping track of certain kinds of
individuals is a primitive visual operation
 Tracking is primitive and likely both preconceptual
and preattemtive
 The mechanism for tracking is the same as the
mechanism that is used for picking out elements
when images are “projected” onto a scene.
How do we do it? What properties
of individual objects do we use?
How do we do it? What properties
of individual objects do we use?
But you can also imagine in the
dark or with your eyes closed!


Does imagery work differently in the dark or
with eyes closed?
Must indexes be visual?
The Sense of Space




This phrase is meant to denote an extremely welldeveloped human capacity to recall and orient to
locations in space; a space that is independent of
modality and is anchored to real allocentric space.
There is a major difference between a sense of space
and a visual image. The sense of space is not a
subjective experience but a skill that is largely
unconscious. There has long been a suspicion that what
has been studied under the name “mental imagery” is
really spatial ability (e.g., unconscious images?).
The sense of space does not need an internal spatial
medium; it can derive spatial properties by binding
mental particulars to real perceived space.
Perceptual Indexes (I.e., FINSTs and Anchors) are
mechanisms that allow representations to inherit some
of the spatial properties of the perceived world.
Some illustrations of the sense of space
Many phenomena that have been cited in support of the
picture theory of mental imagery only implicate a spatial
sense, not the visual perception of a mental display

Sense of space is not specific to (or parasitic on) vision
 Blind people exhibit all the observed phenomena of mental
imagery

Responses to images exhibit S-R compatibility and the
Simon effect – i.e., reactions made towards a stimulus are
faster than ones made away from it.
 The space that is relevant to the Simon effect is amodal (you get
cross-modal Simon effects)


Hemispatial Neglect is a deficit in orienting attention to
real locations – that’s why it may be mirrored in imagery
Mental Images can induce visuomotor adaptation
 But only location, not visual pattern, plays a role (R. Finke)

Observations such as the mental scanning effect, when
they are not due to task demands, can be explained in
terms of scanning through perceived space
Conclusion




Many of the “mental imagery” findings in the literature are the
result of subjects using their tacit knowledge to simulate what it
would be like to see the situation described.
The neuroscience evidence does not show that there is a 2D
display in visual cortex on which we “draw” images when we
imagine. The activity in visual cortex is of the wrong kind to
underwrite mental imagery.
More interesting are the studies in which people project images
onto perceived scenes because these studies do show the
involvement of spatial properties. But these experiments never
need to assume that a picture-like pattern is projected. All they
need to assume is that a few objects in the visual scene are
indexed and associated with objects of thought. The rest of the
spatial properties come from perception.
Although the clear cases are when images are projected onto a
visual scene, the same is likely true of other modalities that
contribute to our sense of space.
Representing space
The spatial character of mental
images (and other “spatial”
representations) comes from
binding objects of thought to
real objects in 3D space. The
space in mental imagery comes
from real concurrently-perceived
spatial relations, which give us
our exquisite sense of space.
References
Bernbaum, K., & Chung, C. S. (1981). Müller-Lyer illusion induced by imagination. Journal of Mental Imagery,
5(1), 125-128.
Kanizsa, G., & Gerbino, W. (1982). Amodal completion: Seeing or thinking? In B. Beck (Ed.), Organization and
Representation in Perception (pp. 167-190). Hillsdale, NJ: Erlbaum.
Pylyshyn, Z. W. (1973). What the Mind's Eye Tells the Mind's Brain: A Critique of Mental Imagery.
Psychological Bulletin, 80, 1-24.
Pylyshyn, Z. W. (1981). The imagery debate: Analogue media versus tacit knowledge. Psychological Review, 88,
16-45.
Pylyshyn, Z. W. (1989). The role of location indexes in spatial perception: A sketch of the FINST spatial-index
model. Cognition, 32, 65-97.
Pylyshyn, Z. W. (2000). Situating vision in the world. Trends in Cognitive Sciences, 4(5), 197-207.
Pylyshyn, Z. W. (2001). Visual indexes, preconceptual objects, and situated vision. Cognition, 80(1/2), 127-158.
Pylyshyn, Z. W. (2002). Mental Imagery: In search of a theory. Behavioral and Brain Sciences, 25(2), 157-237.
Pylyshyn, Z. W. (2003). Return of the Mental Image: Are there really pictures in the brain? Trends in Cognitive
Sciences, 7(3), 113-118.
Pylyshyn, Z. W. (in press). Seeing and visualizing: It's not what you think. Cambridge, MA: MIT Press/Bradford
Books.
Pylyshyn, Z. W., & Storm, R. W. (1988). Tracking multiple independent targets: evidence for a parallel tracking
mechanism. Spatial Vision, 3(3), 1-19.
Shepard, R. N., & Metzler, J. (1971). Mental rotation of three dimensional objects. Science, 171, 701-703.
Shepard, R. N., & Podgorny, P. (1978). Cognitive processes that resemble perceptual processes. In W. K. Estes
(Ed.), Handbook of learning and cognitive processes (Vol. 5). Hillsdale, NJ: Erlbaum.
Slezak, P. (1995). The `philosophical' case against visual imagery. In P. Slezak, T. Caelli & R. Clark (Eds.),
Perspective on Cognitive Science: Theories, Experiments and Foundations (pp. 237-271). Stamford, CT: Ablex.
Tlauka, M., & McKenna, F. P. (1998). Mental imagery yields stimulus-response compatibility. Acta
Psychologica, 67-79.