Transcript More on Representation of space and arguments from

What do experiments on mental
imagery tell us about consciousness?
• Rediscovery of mental imagery in the 1950s marked the
return of cognitive concepts within a behaviorist context.
• The term mental image has been universally assumed to
mean a conscious mental state, similar to a perceptual state.
Thus it provides a way to examine the cortical embodiment
of one of the most frequently studied conscious states.

Some empirical properties of mental images, together with
assumptions about its connection with perception, has lead many
people to search for a specific pictorial objects in the brain.
 NB The question whether there are (or can be) unconscious mental
images has, to my knowledge, never been raised. This is an interesting
idea which runs counter to everyone’s assumption about images (and
therefore worth exploring).
One of the strongest cases for the causal
role of consciousness comes from the
study of reasoning with mental images
●
The ‘psychological reality’ of mental images is unquestioned
because many experiments have shown that manipulating imagery
instructions results in a large and reliable effect on behavior.
For example, there are many demonstrations that in order to recall
something you must first recall an image of something else, and
then examine that image for the answer:

On what side of your front door is the handle on? How many
windows are there in your kitchen? What is 6 feet behind you? What
is in your pocket? Did any US president have a beard (or glasses)?
D
●
J


Imagine a cube on its corner and point and count its corners.
What do you ‘see’ if you rotate a ‘ ’ by 90° and put it on a ‘J’
Among the early experiments suggesting the
causal role of consciousness comes from the
use of mental images in learning and memory
●
A good mnemonic for recalling a list of items is to combine an image of each
item on the list with a place on the image of a familiar path (street, sidewalk...).
Then imagine walking along the path and leaving items from your list at
familiar places. To recall the list you reverse this walk and notice the items.


●
E.g., items might be a shopping list, such as bread, milk, eggs, …
Then imagine walking along a path from the store to this room, dropping off the items on the
list: the bread goes at the corner, the milk goes under the tree, the eggs go by the light, etc..
To recall the list imagine walking back along that path while looking for any objects you
recognize as being on the list. You might see the eggs, then the milk, then the bread.
This “Method of loci” is used by actors for obvious reasons.
In order to remember pairs of words, do this: Imagine the objects denoted by
each of the word pairs interacting in some way – the more bizarre the better.
Imagine the pairs and remember the image of each pair. When its time to
recall you will be given one of the pair. Then you must find the image that
contains that object and see what other object is with it.
One of the strongest cases for the causal
role of consciousness comes from the
study of reasoning using mental images
●
In the 1970’s most research on mental imagery studied the effect of imagery
(or “imigability rating of words”) on learning and memory. This research
brought the idea of mental imagery back into mainstream psychology, but the
next big thing in imagery research was to ask subjects to do things with their
image – to derive conclusions by examining their image. This era was most
visibly occupied by Steven Kosslyn (with whom I have debated often – see my
paper “The Imagery Debate: …” in your reading list).
●
Research in this period found more and more ways in which imagining
something was like seeing a picture of that thing. It was a case study in the
Stimulus Error or the Intentional Fallacy. Before describing some of the
research and the way conclusions were drawn from them I will reiterate why
this has something to do with the course theme: Consciousness, and will also
give you a demonstration of the Intentional Fallacy at work.
Methodological aside:
On the difference between explanations
that appeal to mental architecture and
those that appeal to tacit knowledge
This difference is closely related to the intentional
fallacy and so deserves a special aside.
Aside on the parable of the mystery box
●
A Cognitive Scientist, out walking in a field, comes
upon a black box which happens to have a meter and
recorder that records the meter’s changes over time.
●
The Cognitive Scientist examines lots of records
generated by the box and finds the pattern to be quite
systematic. It follows the following regular pattern:
An illustrative example: Mystery Code Box
What does this behavior pattern tell us about the nature of the box?
An illustrative example: Mystery Code Box
Careful study revealed that pattern #2 only occurs in
this special context when it is preceded by pattern A
What does this behavior pattern tell us about the nature of the box?
The punch line:
●
The black box is transmitting International Morse Code
– in this case English messages in IMC.
●
The Morse code for e is ▪, for i is ▪ ▪ , for c is ▬ ▪ ▬ ▪
●
There is a spelling ‘rule’ in English; Roughly it’s i (▪ ▪)
before e (▪) except after c (▬ ▪ ▬ ▪).
●
The pattern that was observed is due entirely to this rule
of English spelling applied to Morse code.
●
Because this box can presumably transmit any pattern of
dots and dashes, the regularity will not be found in the
way it is wired, but in what it (correctly) represents.
The Moral:
Regularities in behavior may be due to either:
1.
2.
The inherent nature of the system (to its
structure), or its Architecture
The nature of what the system represents
(what it “knows”) or its Representations
Given this regularity, are we now in a
position to figure out how the box works?
●
Does the regularity itself place constraints on potential
structural-functional properties of the black box?
●
Will it help in inferring how the box works?
In general the behavioral repertoire does
not determine the structure of the box
●
●
That’s because the particular behavior of the box in the
example is not constrained by its structure, but by its
relation to its environment, and its function as described at a
particular level– in an intentional vocabulary.
To find out what vocabulary is the correct one a scientist
must study the behavior of the box in special test situations
(not in “ecologically valid” ones). Only rarely will the
vocabulary be obvious, which is why science is hard.
Note the parallel with regularities attributed to the nature of
mental images. In most cases it is attributable to what the
subject knows about the imagined situation.
Before turning to the puzzle of conscious
thoughts, here is a short detour to the intuitive
idea that we think in words or pictures
●
Recall the seductive idea behind the dual code theory of the mental.


We experience our conscious thoughts as either sensory or verbal (spoken)
What else could they be? What would it be like to experience something
consciously that we could not in principal have perceived?
How about pain (headache), dizziness, elation, fear, happiness? Are these
basic, learned or inferred experiences? NB James-Lange theory of emotions.
Consider first the intuitive idea that we think in our native language
(e.g., English). This idea is so natural that most people do not even
consider that it might need support!
● I claim the problem with either option in the dual code view is this:
 Neither words nor pictures have the right variable grain or
specificity to represent the content of thought.
●
Inner speech is not thinking!
As I sit here, writing some text to go with the heading I just typed, I am
aware of imagining saying (“thinking”) to myself:
“I’d better hurry and finish this section or I will be late for my meeting.”
In “saying” that sentence to myself in my inner voice, I meant
something more than what appears in the sequence of words. I
knew which particular text I meant when I thought “this section”; I
knew how much time would have to pass (roughly minutes rather
than hours or days) in order for it to count as my being “late” for a
meeting; I knew which meeting I had in mind even though I only
thought “my meeting”; and I knew what counts as “hurrying” when
typing a section of text, as opposed to running a race.
And for that matter I knew what “I” referred to, although the
sentence did not specify who that was (“I” refers to different people
on different occasions).
Imagined speaking is not thinking!
What is going on in my mind when I imagine speaking, is the
same sort of thing that goes on when I am actually speaking to
someone (or to myself, as I sometimes do). It involves
deciding how best to communicate some idea to another person
by uttering a sequence of words, and then using the resources
of the grammar of my language, as well as principles of
conversation, to construct a sentence that conveys that meaning
when conjoined with my other beliefs, including what I
believe my hearer already knows (or is in a position to infer in
that context). The sentence I decide to speak or imagine
speaking is the end product of such a thought process, it is not
the thought.
Neither inner nor outer speech
expresses all the thoughts behind them
Sentences alone never express all and only what their speakers
mean or think. Because what we experience when we seem to
be “thinking in words” is an imaginary dialogue, the sentences
of this imagined dialogue follow rules of “discourse” such as
those referred to as Gricean Maxims. These include principles
of cooperation such as “make your statements as informative as
required but not more informative than necessary” (i.e., don’t
express what your hearer already knows). They include such
principles as Truth (don’t say what you know to be false),
Informativeness (only say what your hearer does not know),
Relevance and Clarity (e.g., avoid ambiguity, vagueness,
redundancy). Simple as these maxims are, they apply to all
human dialogues, including imagined ones.
What we just saw is a case where much of
the regularity does not reflect the nature of
thoughts, but the constraints on discourse
●
Like the case of the long-short dashes in the code-box
example, much of the pattern is not caused by the
nature of the intentional representations, but of the
rules of discourse. The person determines the content
by virtue of his thoughts but the rest of the regularities
are determined by the constraints on discourse.
●
The person controls the content of thoughts, just the
way he does in a conversation, but this content is
given prior to the sentences being formed.
Preview of what is to come…
●
Most, if not all the work on mental imagery in the past
40 or so years has been fueled by phenomenology and
by the metaphor of mental space and mental pictures,
which in turn has been used to explain empirical findings
the way one might be tempted to explain the behavior of
the black box – by looking for things inside instead of
things outside the mind.
●
In the case of mental imagery, much of the work has
appealed to relevant spatial properties, but has done so
by assuming that these properties are replicated inside
the head!
Speculative aside concerning the role of space
Space and self-consciousness?

Some writers have suggested that self-consciousness
arises from the ability to distinguishing self from other
on the basis of location.

So being conscious of oneself is being able to situate
oneself apart from others in space.

If there is anything to this view of consciousness, the
work on imagery could help place it in context since
empirical findings concerning mental imagery have
been essentially about how we mentally represent space.
What constraints does imagery
impose on representations?
●
●
Recalling the code-box example we might ask whether the
regularities are due to the nature of our cognitive architecture
or to what you we want or believe?
As with the code-box example we should ask what constraints
are imposed by the mental architecture. Can your image have
any properties you choose? If not, ask; Why not? of examples;
 Can you imagine an object viewed from all directions at once, or from no
particular direction?
 Can you imagine a 4-dimensional object?
 Can you imagine a written character that is neither upper nor lower case?
Or a figure that does not have a particular size or shape (e.g. Berkeley asked:
can you imagine a triangle that is neither equilateral nor isosceles, nor scalene, …etc)?
 Can you imagine a 3x3 array of numbers and read them back in any order?
What constraints does imagery
impose on representations?
Images are better at representing spatial layouts than temporal patterns
L
M
Q
G
F
S
E
Y
T
More on Consciousness of space and
arguments from neuroscience
●
The intuition that images are spatial (or, as some put it, that
they “have space”) is deeper than most other questions
about images.
I have discussed this in mind-numbing detail in “Things and Places” (Chapter 4).
●
According to Pictorialists (like Kosslyn), the argument
over imagery is now in its third and final stage, where
evidence from neuroscience has essentially put an end to
the debate. I will look at some of this alleged debateending evidence and will show that it is characterized by
the pervasive acceptance of the intentional fallacy.
One of the earliest and most-cited studies that
have been interpreted as showing that images
are laid out in space: Mental Scanning – a ‘window on the mind’
• The experiments seem to show that when you
move your attention across a mental image, it
takes longer (in real time) to move it a greater
(imagined) distance – i.e. time & distance on
the image appears to follow the same laws as
it does in the real world – i.e. time increases
linearly with distance when speed is constant.
Mental Scanning (“Window on the Mind”)
• Kosslyn (and many hundreds of researchers and
students) showed that if you ask subjects to
move their attention from one place on an
imagined map, it takes longer to do so when the
distance represented is greater.
• This series of studies was judged by the journal
editor as an example of important new findings
in psychology. It has become a classic.
• You might keep watch for possible cases of the
intentional fallacy or the stimulus error!
Studies of mental scanning
Does it show that images have metrical space?
2
1.8
1.6
Latency (secs)
1.4
scan image
imagine lights
show direction
1.2
1
0.8
0.6
0.4
0.2
0
Relative distance on image
(Pylyshyn & Bannon. Described in Pylyshyn, 1981)
Conclusion: The image scanning effect is Cognitively Penetrable
 i.e., it depends on Tacit Knowledge.
Does visual imagery use the visual system?
• It depends on what you mean by use and visual system



To use vision can mean to visually perceive the image – a non-starter
Only early vision is relevant to this use of “perceive”, since general
vision can involve all of cognition. This point requires a whole lecture!
This question is of interest to picture theorists because if vision is
involved it would suggest that images are uninterpreted picture-like
spatial displays. The visual module cannot be applied to alreadyinterpreted data structures – we don’t see symbolic descriptions.
• Let’s assume that “the visual cortex” is “active” during
mental imagery as some fMRI studies have suggested,

What follows from that? Does it tell us what the image itself must be like?
• Why should we believe that vision is involved?

In the last 15 years the main support for the assumption that the visual
system is involved has come from neuroscience.
Reasons for thinking that images
are interpreted by the visual system
• Similar phenomenology of imagining & seeing

This reason actually overshadows all others
• Re-perceiving and novel construals

A large but very problematic literature
• Superposition & interference studies
• Visual illusions with projected images

The ubiquitous role of attention
Reasons for thinking that images
are interpreted by the visual system
• Similar phenomenology of imagining & seeing

This phenomenology is what leads to the intentional
fallacy as shown in the next cartoons from Kliban
This is what our conscious experience
suggests goes on in vision…
Kliban
This is what the demands of explanation
suggests must be going on in vision…
Conceptual representation
– not a picture or icon
More demonstrations of the relation
between vision and imagery
• Images constructed from descriptions

The two-parallelogram example
• “Seeing” mental images lacks the critical signature
properties of vision
 Involuntary properties of vision such as amodal
completion, automatic 3D & apparent motion, different
off-retina analysis, sensitivity to hints, …
• Reconstruals: Slezak
this
imagery
Connect eachDo
corner
of the top
parallelogram with the
Now
imagine
an exercise:
identical
Imagine
acorner
parallelogram
likethis
thisone
one
corresponding
of the bottom
parallelogram:
parallelogram
directly
below
What do you see?
What do you see when you imagine the connections?
Did the imagined shape look (and change) like the one you see now?
Viewing Mental Images lacks
signature properties of vision
• No ‘Amodal Completion’ of partly-occluded objects.
• Off-retina figures do not combine.
• Superposition not as common as thought – does not
occur in general (so assumed independent motivation
for an internal screen not supported).
• Most psychophysical phenomena do not occur in
viewing mental images (notwithstanding some bad
published studies)
“Amodal completion” in images?
Is this what you saw?
Off-retinal information is different from foveal info
Off-retinal information is different from retinal info
(which itself suggests that all mental information is different from visual inputs)
Standard view of saccadic
integration by superposition
Is it plausible? It is one of the arguments given in
support of the view that mental imagery is like vision
except it enters higher up in the ‘visual pathway’.
Superposition does not work (O’regan 1983)
More questions about the relation
between vision and imagery
• Conceptual information is never iconic or
graphic, as picture theorists must assume.
• Images constructed from descriptions


The D-J example(s)
The two-parallelogram example
• Amodal completion
• Reconstruals: Slezak
Labels propagate over picture
Visual-Motor adaptation and
image-motor adaptation
●
The basic prism adaptation setup: arm movement towards
a target while wearing prism glasses
●
Now repeat with arm unseen but subject told where it is
(actually where it would have been in the prism case)
●
Get adaptation {Finke, R. A. (1979). The Functional Equivalence of Mental Images and
Errors of Movement. Cognitive Psychology, 11, 235-264.}
●
But in the original experiment it has been shown that you
don’t need to see a hand, any indicator of where the hand
is will do as long as the subject believes his hand is where
indicated: Being your hand is irrelevant.
Can images be visually reinterpreted?
• There have been many claims that people can visually
reinterpret images

These have all been cases where one could easily figure out
what the combined image would look like without actually
seeing it (e.g., the J – D superposition). (This example was
mentioned earlier)
• Peterson’s careful examination of visual “reconstruals”
showed (contrary to her own conclusion) that images are
never bistable (no Necker cube or figure-ground reversals)
and when new construals were achieved from images they
were quite different from the ones achieved in vision
(more variable, more guessing from cues, etc).
• The best evidence comes from a philosopher (Slezak, 1992, 1995)
Slezak figures
Pick one (or two) of these animals and
memorize what they look like. Now
rotate it in your mind by 90 degrees
clockwise and see what it looks like.
Slezak figures rotated 90o
What do parallels between
seeing and imaging show?
Phenomenology is ambiguous between:
Showing that imagery uses vision to examine
an internal picture, or
2. Showing that vision does not involve an
internal picture either!
There is little doubt that (2) is the correct
1.
option. Why do we need a second (internal)
image when we have the original to look at!
As Nelson Goodman said, ‘One of the damn things is enough’!
Neuroscience: The picturetheory’s last hope
Are there pictures in the brain?
There is no evidence of cortical displays of
the right kind to explain imagery phenomena.
Here is what there is that gives hope to
picture-theorists:
Neuroscience has shown that the retinal pattern of
activation is displayed on the surface of the cortex
There is a topographical projection
of retinal activity on the visual
cortex of the cat and monkey.
Tootell, R. B., Silverman, M. S., Switkes, E., & de Valois, R. L
(1982). Deoxyglucose analysis of retinotopic organization in
primate striate cortex. Science, 218, 902-904.
Problems with drawing conclusions about the nature
of mental images from neuroscience data
1. The capacity for imagery and for vision are known to be
independent. Also all imagery results are observed in the blind.
2. Cortical topography is 2-D, but mental images are 3-D – all
phenomena (e.g. rotation) occur in depth as well as in the plane.
3. Patterns in the visual cortex are in retinal coordinates whereas
images are in world-coordinates
 Your image stays fixed in the room when you move your eyes or turn your
head or even walk around the room
4. Accessing information from an image is very different from
accessing it from the perceived world. Order of access from
images is highly constrained.
 Conceptual rather than graphical properties are relevant to image
complexity (e.g., mental rotation).
Problems with drawing conclusions about mental
images from the neuroscience evidence
5. Retinal and cortical images are subject to Emmert’s Law,
whereas mental images are not;
6. The signature properties of vision (e.g. spontaneous 3D
interpretation, automatic reversals, apparent motion, motion
aftereffects, and many other phenomena) are absent in images;
7. A cortical display account of most imagery findings is
incompatible with the cognitive penetrability of mental
imagery phenomena, such as scanning and image size effects;
8. The fact that the Mind’s Eye is so much like a real eye (e.g.,
oblique effect, resolution fall-off) should serve to warn us that
we may be studying what observers know about how the world
looks to them, rather than what form their images take.
Problems with drawing conclusions about mental
images from the neuroscience evidence
9.
Many clinical cases can be explained by appeal to tacit
knowledge and attention



The ‘tunnel effect’ found in vision and imagery (Farah) is likely due to
the patient knowing what things now looked like to her post-surgery
Hemispatial neglect seems to be a deficit in attention, which also
explains the “representational neglect” in imagery reported by Bisiach.
A recent study shows that imaginal neglect does not appear if patients
have their eyes closed. This fits well in the account I have offered in
which the spatial character of a mental images derives from concurrently
perceived space.
10. What if colored three-dimensional images were found in visual
cortex? What would that tell you about the role of mental
images in reasoning? Would this require a homunculus?
This is MORE than enough
for now!
So why does it not feel like we are
doing computations?
 The content of our conscious experience is a very poor
guide to what is actually going on that causes both our
experiences and our behavior. Science is concerned with
causes, not just correlations.
 We have learned (since Galileo) that we can’t assume
that the way things seem has much to do with how it
works (consider the example of language understanding)
 As in most sciences, the essential causes are far from obvious
(e.g., How can the moon exert a pull on the earth without
contacting it? What is this table made of? etc.).
 In the case of cognition, what is going on is a delicate mixture
of the obvious (what Granny or Shakespeare knew about
people, and why they do what they do) and the incredible.
Is there very short-lasting iconic storage?
●
Although the idea of pictorial long-term memory has
been known to be false ever since Sir Fredrick
Bartlett’s book on memory (and Hebb’s work on
neural basis of memory), there is some provisional
evidence that sensory information may outlast the
duration of the stimulus. Many people have studied
these “sensory buffers” including George Sperling
and Michael Posner. But the story remains murky
even there. Visual information loses its spatiality
very soon after the retina.
Sperling’s partial report method
for showing an iconic memory
Mental Scanning
●
●
Some hundreds of experiments have now been done
demonstrating that it takes longer to scan attention
between places that are further apart in the imagined
scene. In fact the relation is linear between time and
distance.
These have been reviewed and described in:
 Denis, M., & Kosslyn, S. M. (1999). Scanning visual mental
images: A window on the mind. Cahiers de Psychologie
Cognitive / Current Psychology of Cognition, 18(4), 409-465.
 Rarely cited are experiments by Pylyshyn & Bannon which I
will summarize for you.
Studies of mental scanning
Does it show that images have metrical space?
2
1.8
1.6
1.4
scan image
imagine lights
Latency (secs)
show direction
1.2
1
0.8
0.6
0.4
0.2
0
Relative distance on image
(Pylyshyn & Bannon. Described in Pylyshyn, 1981)
Conclusion: The image scanning effect is Cognitively Penetrable
 i.e., it depends on Tacit Knowledge.
What is assumed in imagist
explanations of mental scanning?
In actual vision, it takes longer to scan a longer distance because
real distance, real motion, and real time is involved, therefore this
equation holds due to natural law:
Time = distance
speed
But what ensures that a corresponding relation holds in an image?
The obvious answer is: Because the image is laid out in space!
 But what if that option is closed for empirical reasons?
● Imagists appeal to a “Functional Space” which they liken to a
matrix data structure in which some cells are adjacent to other
cells, some are closer and others further away, and to move from
one to another it is natural that you pass through intermediate cells
● Question: What makes these sorts of properties “natural” in a
matrix data structure?
●
Thou shalt not cheat
●
●
There is no natural law that requires the representations of
time, distance and speed to be related according to the
motion equation. You could equally easily imagine an object
moving instantly or according to any motion relation you
like, since it is your image!
There are two possible answers why the relation
Time = Representation of distance
Representation of speed
typically holds in an image-scanning task:
1.
2.
Because subjects have tacit knowledge that this is what
would happen if they viewed a real display, or
Because the matrix is taken to be a simulation of a realworld display, as it often is in computer science, we are
tempted to think of the 2D display, whose presence is
only heuristic: There is no 2D surface anywhere!!
Mental Rotation has been one of the
most cited demonstrations of all
●
Look at the following 3D figures and judge which
pairs are the same except for orientation. The other
pair are enantiomorphs – 3D mirror images so they
can’t be put into correspondence by 3D rotation only.
Mental rotation
Time to judge whether (a)-(b) or (b)-(c) are the
same except for orientation increases linearly
with the angle between them (Shepard & Metzler, 1971)
What do you do to judge whether these
two figures are the same shape?
Is this how the process looked to you?
When you make it rotate in your mind, does it seem
to retain its rigid 3D shape without re-computing it?
Thou shalt not cheat
● What happens in ALL imagist accounts of
phenomena from mental scanning to mental rotation
is that they assume the properties of real space in
order to provide a principled explanation, then retreat
to something not-quite-real space when it is pointed
out that they are assuming that images are laid out in
real space-in-the-head.
● This happens with mental rotation as well, even
though the tacit knowledge account is not plausible
there (it is an involuntary and universal way of
solving the rotated-figure task so long as the task
involves tokens of enantiomorphs).
The missing bit of logic:
● According to Prinz (2002) p 118,
“If visual-image rotation uses a spatial medium of the
kind Kosslyn envisions, then images must traverse
intermediate positions when they rotate from one
position to another. The propositional system can be
designed represent intermediate positions during
rotation, but that is not obligatory.”
● Given that this happens in 3D so that it can’t be a
literal brain space, the question arises: What makes
this obligatory in “functional Space”?
Sources of obligatory constraint
●
●
It could be that there is a medium or a set of analog
properties that together happen to simulate a virtual
space, and the physical properties of this medium enforce
the continuity of motion through it. This is very unlikely.
It could be that the constraint comes from the fact that it
holds in the world. Then its transfer from the real to the
mental world occurs either ;
 Voluntarily, because we know how it would happen and we can
use that fact to solve the problem
 Naturally, because the constraint got built in over time through
evolutionary pressures
 Naturally, but not because the constraint is built in, but because
of other properties of the architecture that make it more efficient
to compute the rotated shape incrementally until there is a match
Do images have low-level
psychophysical properties?
● Bring the bars closer and closer together. In which
can you see the bars when the spacing is closest?
● In experiments, it was shown that the oblique effect
occurs in mental images just as it does in vision
 Kosslyn, Thompson & Ganis (2006) argued that this is
because there are more vertical and horizontally tuned cells
than obliquely tuned cells in visual cortex. Does this
explain the finding?
Do images have low-level visual properties?
● Imagine a grating in which the bars are:
1. Vertical
2. Horizontal
3. Oblique (45°)
● Which of these (identical) gratings can continue to
be discriminated as the figure becomes smaller?
A final point…
● In Kosslyn, Thompson & Ganis (2007) the authors cite
Ned Block to the effect that one does not need an actual
2D surface, so long as the connections upstream from the
cortical surface can decode certain pairs of neurons in
terms of their imagined distance. Imagine long stretchy
axons going from a 2D surface to subsequent processes.
Now imagine that the neurons are randomly moved
around so they are no long on a 2D layout. As long as
the connections remain fixed it will still behave as
though there was a 2D surface.
● Call this the “encrypted 2D layout” version of literal
space.
The encrypted-spatial layout alternative
● By itself the encrypted-layout alternative will not do because
without referring to the original 2D locations, the relation
between pairs of neurons and scan time is not principled. In
the end the only principle we have is Time=distance/speed so
unless the upstream system decrypts the neuron locations into
their original 2D surface locations the explanation for the
increase in time with increased imagined distance remains a
mere stipulation. It stipulates, but does not explain why, when
two points are further away in the imagined layout it takes
longer to scan between them or why scanning between them
requires that one visit intermediate locations along the way.
● But this is what we needed to explain! One can apply such a
mere stipulation to any form of representation. What was a
principled explanation with the literal 2D display has now
been given up for a mere statement of how it shall be.
The ‘Imagery Debate’ Redux
●
According to Kosslyn there have been 3 stages in the
debate over the nature of mental images:
1. The role of images in learning and memory (Paivio’s
Dual Code theory). Influential at the time but now
abandoned except for a few recidivists like Barsalou.
2. Spatial properties of images and their role in dynamic
processes, as assessed by reaction time measures
(Kosslyn’s research on ‘metric properties of images’)
3. Discovery of brain mechanisms underlying visual
imagining, which led to ‘the resolution of the
imagery debate’.
Mental imagery and neuroscience
●
●
Neuroanatomical evidence for a retinotopic display in
the earliest visual area of the brain (V1)
Neural imaging data showing V1 is more active during
mental imagery than during other forms of thought
 The form of activity differs for small vs large images in the
way that it differs when viewing small and large displays
●
●
●
Transcranial magnetic stimulation of visual areas
interferes more with imagery than other forms of thought
Clinical cases show that visual and image impairment
tend to be similar (Bisiach, Farah)
More recently psychophysical measures of images shows
parallels with comparable measures of vision, and these
can be related to the receptive cells in V1
Neuroscience has shown that the retinal pattern of
activation is displayed on the surface of the cortex
There is a topographical projection
of retinal activity on the visual
cortex of the cat and monkey.
Tootell, R. B., Silverman, M. S., Switkes, E., & de Valois, R. L
(1982). Deoxyglucose analysis of retinotopic organization in
primate striate cortex. Science, 218, 902-904.
Problems with drawing conclusions about the
nature of mental images from neuroscience data
1.
The capacity for imagery and for vision are known to be
independent. Also all imagery results are observed in the blind.
2.
Cortical topography is 2-D, but mental images are 3-D – all
phenomena (e.g. rotation) occur in depth as well as in the plane.
Patterns in the visual cortex are in retinal coordinates whereas
images are in world-coordinates
3.

4.
Your image stays fixed in the room when you move your eyes or turn
your head or even walk around the room
Accessing information from an image is very different from
accessing it from the perceived world. Order of access from
images is highly constrained.

Conceptual rather than graphical properties are relevant to image
complexity (e.g., mental rotation).
Problems with drawing conclusions about mental
images from the neuroscience evidence
5.
6.
7.
8.
Retinal and cortical images are subject to Emmert’s Law,
whereas mental images are not;
The signature properties of vision (e.g. spontaneous 3D
interpretation, automatic reversals, apparent motion, motion
aftereffects, and many other phenomena) are absent in images;
A cortical display account of most imagery findings is
incompatible with the cognitive penetrability of mental imagery
phenomena, such as scanning and image size effects;
The fact that the Mind’s Eye is so much like a real eye (e.g.,
oblique effect, resolution fall-off) should serve to warn us that
we may be studying what observers know about how the world
looks to them, rather than what form their images take.
Problems with drawing conclusions about mental
images from the neuroscience evidence
9.
Many clinical cases can be explained by appeal to tacit
knowledge and attention



The ‘tunnel effect’ found in vision and imagery (Farah) is likely due to
the patient knowing what things now looked like to her post-surgery
Hemispatial neglect seems to be a deficit in attention, which also
explains the “representational neglect” in imagery reported by Bisiach
A recent study shows that imaginal neglect does not appear if patients
have their eyes closed. This fits well in the account I will offer in which
the spatial character of a mental images derives from concurrently
perceived space.
10. What if colored three-dimensional images were found in visual
cortex? What would that tell you about the role of mental
images in reasoning? Would this require a homunculus?
This is MORE than enough for now!
Should we welcome back the homunculus?
●
●
●
In the limit if the visual cortex contained the contents of ones
conscious experience in imagery we would need an interpreter to
“see” this display in visual cortex
But we will never have to face this prospect because many
experiments (including ones by Kosslyn) show that the contents of
mental images are conceptual (or, as Kosslyn puts it, contain
“predigested information”).
And finally, it is clear to anyone who thinks about it for a few
seconds that you can make your image do whatever you want and
to have whatever properties you wish.
 There are no known constraints on mental images that cannot be attributed to
lack of knowledge of the imagined situation (e.g., imagining a 4D cube).
 All currently claimed properties of mental images are cognitively penetrable.
Explaining mental scanning, mental rotation and
image size effects in terms of “functional space”
●
●
●
When people are faced with the natural conclusion
that the “iconic” position entails space (as in scanning
and size effects) they appeal to “functional space”
A Matrix in a computer are often cited as an example
Consider a functional space account of scanning or of
mental rotation:
 Why does it take longer to scan a greater distance in a
functional space?
 Why does it take longer to rotate a mental image a greater
angle?
But there are examples of solving geometry
problems easily with imagery
• There are many problems that you can solve much
more easily when you imagine a layout than when
you do not.
• In fact many instances of solving problems by
imagining a layout that seem very similar to how
would solve them if one had pencil-and-paper.
• The question of how pictures, graphs, diagrams, etc
help in reasoning is very closely related to the
question of how imagined layouts function in
reasoning. That is not in question. What is in
question is what happens in either the visual or
imagined cases and how images can benefit from this
processes even though there is no real diagram.
How do real visual displays help thinking?
 Why do diagrams, graphs, charts, maps, icons and other visual
objects help us to reason and to solve problems?
 The question why visual aids help is nontrivial and Seeing &
Visualizing, chapter 8 contains some speculative discussion,
e.g., they allow the visual system to:
• make certain kinds of visual inferences
• make use of visual demonstratives to offload some of the memory load
• Capitalize on the fact that the displays embody the axioms of measure
theory and of geometry (which are then inherited by thought)
 The big question is whether any of these advantages carry over to
imaginal thinking! Do mental images have some (or any) of the
critical properties that make diagrams helpful in reasoning?
Visual inferences?
●
If we recall a visual display it is because we have encoded
enough information about its visual-geometrical properties that
we can meet some criteria, e.g., we can draw it. But there are
innumerably many ways to encode this information that are
sufficient for the task (e.g. by encoding pairwise spatial
relations, global spatial relations, and so on). For many
properties the task of translating from one form to another is
much more difficult than the task of visually encoding it – the
translation constitutes visual inference.
● The visual system generalizes from particular instances as part
of its object-recognition skill (all recognition is recognition-as
and therefore assumes generalization from tokens to types). It
is also very good at noticing certain properties (e.g., relative
sizes, deviations from square or circle, collinearity, inside, and
so on). These capabilities can be exploited in graphical layouts.
Memorize this map so you can draw it accurately
From your memory:
•
•
•
•
•
Which groups of 3 or more locations are collinear?
Which locations are midway between two others?
Which locations are closest to the center of the island?
Which pairs of locations are at the same latitude?
Which is the top-most (bottom-most) location?
 If you could draw the map from memory using whatever
properties you noticed and encoded, you could easily
answer the questions by looking at your drawing – even
if you had not encoded the relations in the queries.
Draw a rectangle. Draw a line from the bottom corners to a point
on the opposite vertical side. Do these two lines intersect? Is the
point of intersection of the two lines below or above the
midpoint? Does it depend on the particular rectangle you drew?
A
B
x
y
m
m’
D
C
Which properties of a real diagram also
hold for a mental diagram?
● A mental “diagram” does not have any of the properties that a
real diagram gets from being on a rigid 2D surface.
● When you imagine 3 points on a line, labeled A, B, and C, must
B be between A and C? What makes that so? Is the distance AC
greater than the distance AB or BC?
● When you imagine drawing point C after having drawn points A
and B, must the relation between A and B remain unchanged
(e.g., the distance between them, their qualitative relation such as
above or below). Why?
● These questions raise what is known as the frame problem in
Artificial Intelligence. If you plan a sequence of actions, how do
you know which properties of the world a particular action will
change and which it will not, given that there are an unlimited
number of properties and connections in the world?
If all else fails there is always parsimony and
generality…(they worked well in physics and linguistics!)