Transcript What is Cognitive Science?

Proseminar in Cognitive Science, 2009
Zenon Pylyshyn, Rutgers Center for Cognitive Science
What is Special About Cognition?
What’s in the mind? How do we know?
What is special about cognition?
• Cognition (from Latin “cogito”) refers to the capacity to
know, and by extension to reason, perceive, plan, decide,
solve problems, infer the beliefs of others, communicate
by language as well as by other ways, and all the other
capabilities we associated with intelligent activity.
• What is central to all such activity is that it relies on
representations of the (actual or imagined) world.
 Cognitive science is the study of systems that represent and that
use their representations rationally, e.g.,draw inferences.
 A computer is another such system, so computing has become
the basic paradigm of cognitive science.
• In the last 40 years, The Representational Theory of Mind
has become The Computational Theory of Mind
Cognitive science is a delicate mixture
of the obvious and the incredible
Granny was almost right:
Behavior really is governed by what we know and
what we want (together with the mechanisms for
representing and for drawing inferences from these)
It’s emic, not etic properties that matter
Kenneth Pike
 What determines our behavior is not how the
world is, but how we represent it as being
 As Chomsky pointed out in his review of Skinner, if
we describe behavior in relation to the objective
properties of the world, we would have to conclude
that behavior is essentially stimulus-independent
 Every behavioral regularity is cognitively penetrable
• Exceptions are physical ones like falling, or breaking
• Regularities attributable to the fixed architecture of the
mind/brain
It’s emic
states that
matter!
The central role of representation creates
some serious problems for a natural science
 What matters is what representations are about
 But how can the fact that a belief is about some
particular thing have an observable consequence?
• How can beliefs about ‘Santa Claus’ or the ‘Holy
Grail’ determine behavior when they don’t exist?
 In a natural science if “X causes Y” then X must
exist and be causally connected to Y!
• Even when X exists, it is not X’s physical properties
that are relevant, it’s what they are perceived as!
 e.g., the North Star & navigation
Is it hopeless to think we can have a
natural science of cognition?
William James’ “Science of Mental Life?”
Along comes The computational theory of mind
“the only straw afloat” {Fodor}
The major historical milestones
• Brentano’s recognition of the problem of
intentionality: Mental States are about something,
but aboutness is not a physical relation.
• The formalist movement in the foundations of
mathematics: Hilbert, Kurt Gödel, Bertrand Russell
& Alfred Whitehead, Alan Turing, Alonzo Church,
… provided a technique by which logical reasoning
could be automated.
• Representational/Computational theory of mind: The
modern era: Newell & Simon, Chomsky, Fodor
Intelligent systems behave the way
they do because of what the represent
• But in order to function according to physical
principles, the representations must have (or be
instantiated in) physical properties
• To encode knowledge in physical properties
one first encodes it in symbolic form (Proof
Theory tells us how) and then instantiates those
symbolic codes physically (computer science
and computer engineering tells us how)
How to make a purely mechanical system reason about things it does
not understand or know about? The discovery of symbolic logic.
(1) Married (John, Mary) OR Married (John, Susan)
and the equation or “statement”
(2) NOT [Married (John, Susan) ].
from these two statements you can infer,
(3) Married (John, Mary)
Notice that (3) follows from (1) and (2) regardless the parts written in this orange
color (i.e., everything except the symbols OR or NOT) so that you could write
down the equations without mentioning marriage or John or Mary or, for that
matter, anything having to do with the world. Try replacing these expressions
with the meaningless letters P and Q. The resulting inference still holds so long
as different orange symbols are replaced by unique meaningless symbols.
(1') P or Q
(2') not Q
therefore,
(3') P
Convert thought to symbol processing

If we encode beliefs, knowledge, thoughts, and other
“propositional attitudes” (e.g., want, fear, hope, know,
guess,…) in terms of symbolic codes, then we can use
the rules of logic to draw inferences from them to new
beliefs. This process is called “thinking.”

If we can design a machine so that its physical states
correspond to symbolic expressions, we may be able to
get the machine to change states in such a way as to
result in inferences and therefore to model thinking.
That’s just what a Turing Machine does!

A Turing Machine is a very simple theoretical machine
that changes its state depending on what symbols occur
on its tape and what state it is currently in.
(Statei, Input-symbolx)  (Statej, Output-symboly, movel,r)

Turing showed, by hard work and persistence, that such
a machine could compute any function we can describe.
In particular it can compute whatever function some
other Turing Machine would compute given a certain
input. The latter is called a Universal Turing Machine
and you will hear more about it in a few weeks.
A Turing Machine could not be simpler!
Turing Machine rules look like this:
Rule 1
State Si
If you are in state S1 and you read
G then move left and write D
Rule 2. If you are in state S2 and you read
‘or’ then move left and write Q
....
Write ‘Q’
# C§ Q T B
Read ‘or’
or
VC  | 
#
The very minimal you need to be a Turing-equivalent machine is the capacity to
read symbols, write symbols, move to a new tape cell, and to change states
depending on what symbol you read and what state you are in at the time.
You will hear more about Turing Machines in 2 weeks
Cognitive Science and the Tri-Level Hypothesis
Intelligent systems are organized at three
(or more) distinct levels:
1. The physical or biological level
2. The symbolic or syntactic level (Turing machine)
3. The knowledge or semantic level
 This means that different regularities may
need to be explained by different levels
The central role of representation creates
some serious problems for a natural science
 A serious problem arises because we are not
directly aware of our representations
 We are not aware of our thoughts …
 What we are usually aware of is what our thoughts are
about, not properties of the representation itself
 Need to distinguish properties of our thoughts and
properties of what they are about (e.g. mental images)
 We are not even aware of deciding, choosing or willing
an action (Wegner)
 Introspective evidence is just one type of
evidence and it has turned out to be unreliable
How can we find out?
 Given these serious problems in
understanding cognition, is it even possible
in principal to find out how the mind works?
 Is there even a fact of the matter about what
process is responsible for certain behaviors?
 Is the only road to understanding cognition
through neuroscience?
 How can we discover how mental processes
work?
Weak vs Strong Equivalence
• Is cognitive science concerned only with developing models that
generate the same Input-Output behavior as people exhibit in
carrying out certain tasks?
• A theory that correctly predicts I-O behavior is said to be weakly
equivalent to the psychological process it is supposed to explain.
It is what some people mean by “simulating behavior”.
• Everyone in Cognitive Science is interested in strong
equivalence – we want to explain not only the observed
behavior, but also how it is generated. The how will usually
takes the form of an algorithm.
Simulating the Input-Output function
Input
Black Box
Output
Can we do any better than I-O simulation
without looking inside the black box?
 If all you have is observed behavior, how
can you go beyond I-O simulation?
Simulating the Input-Output function
Think about this for a few minutes:

Is there any way to find out HOW a
person does a simple problem such as
adding two 4 digit numbers?
67 + 44 = ??

What are possible sources of evidence
that may be relevant to this question?
Modeling the Actual Process
(the algorithm used)
Input
Black Box
Output
Index of
process
If all you have is observed behavior, how can you
go beyond I-O simulation (mimicry)?
 Answer: Not all observations are Inputs or Outputs:
some are meta-behavior or indexes of processes.
Example:
Sternberg memory search experiment
• The initial input consists of the instructions and
the presentation of the memory set (n letters).
• On each trial the particular input to the black box
consists of the presentation of a target letter.
• The output consists of a binary response (present
or absent). The time taken to respond is also
recorded. That’s called the “Reaction Time”.
• Is the reaction time part of the output, to be
generated by the theory, as another response?
Example of the input-output of a
computational model of the Sternberg task
• Inputs: Memory set is (e.g.) C, D, H, N
• Inputs: Probe (e.g., C or F)
• Output: Pairs of Responses and Reaction Times
(e.g. output is something like “Yes, 460 msecs”)
• Does it matter how the Output is derived?
It doesn’t if all you care about is I-O behavior
It does if you care about Strong Equivalence (i.e.,
HOW it works)
Example of the input-output of a
computational model of the Sternberg task
• Inputs are: (1) Memory set = C,D,H,N
(2) Target probe = C (or F)
• Input-Output prediction using a table:
Input to model
C
N
R
H
M
Yes
Yes
No
Yes
No
Model prints out
460 ms
530 ms
600 ms
520 ms
620 ms
Is this model weakly- or strongly-equivalent to a person?
Example of a weakly equivalent
model of the Sternberg task
1.
2.
3.
4.
5.
6.
7.
8.
Store memory set as a list L. Call the list size = n
Read target item, call it  (If there is no , then quit)
Search through L for  until found or end of list encountered
If found in the list, assign Ans =“yes” otherwise Ans =“no”
(That provides the answer, but what about the time RT ?)
If Ans =“yes”, set RT = 500 + n * K  Random(20  x  50)
If Ans =“no” , set RT = 800 + n * K  Random(20  x  50)
Print Ans , Print RT
End of trial, repeat from #2.
Is this the way people do it? How do you know?
Results of the Sternberg memory search task
What do they tell us about how people do it? Is this Input-Output
equivalent or is it strongly equivalent to human performance?
Exhaustive search
Self-terminating search
What reasons do you have for
doubting that people do it this way?
 Because in this case time should not be one of the
computed outputs, but a measure of how many steps
it took.
 The same is true of intermediate states (e.g.,
evidence includes what subjects say, error rates, eye
tracking, judgments about the output, and so on.)
 Reaction time is one of the main sources of
evidence in cog sci.
 Question: Is time always a valid index of processing
complexity?
Studying human arithmetic
algorithms
• Arithmetic (VanLehn & Brown. “Buggy”)
Buggy – a model of children’s arithmetic – has about
350 “rules” which help uncover “deep bugs”
• Newell & Simon’s study of problem solving
Problem behavior graph and production systems
Use of protocols, eye tracking
• Information-Processing style of theory.
Computational but not always a computer model.
If cognitive processes are at a different level of
organization from the physical level, how can we ever
find out what they are – i.e., how can we discover what
algorithm is being used?
 We are limited only by the imagination of the
experimenter, e.g.,
 Relative complexity evidence (RT, error rates…)
 Intermediate state evidence
 Eye tracking






Stage analysis (additive factors method)
Event Related Potentials (EEG)
fMRI
clinical observations of brain damage
Psychophysical methods (SDT)
Etc…
Does intentionality (and the trilevel
hypothesis) only apply to high-level
processes such as reasoning?
• Examples from language.
John gave the book to Fred because he finished it
John gave the book to Fred because he wanted it
• The city council refused to give the workers a permit for a
demonstration because they feared violence
• The city council refused to give the workers a permit for a
demonstration because they were communists
Does meaning (and the trilevel
hypothesis) only apply to high-level
processes such as reasoning?
Representation in perception
• What does perception (especially vision)
tell cognition?
• What is the “output” of the visual system?
Representation in perception
• When we are aware of processes going on during
visual perception they are mostly misleading
Galileo, “... if men had been born blind, philosophy would be more perfect, because it
would lack many false assumptions that have been taken from the sense of sight.”
• So how are we to develop theories of vision?
• One method is to try to build models that are
sufficient to the task of seeing – i.e. by showing
that they can recognize certain familiar shapes, fail
in ways that are similar to how human vision fails,
and is able to plan motor commands towards things
in the visual world
Does intentionality (and the trilevel hypothesis) only
apply to high-level processes such as reasoning?
 Examples from vision seeing as: It’s what you see the figure as that
determines behavior – not its physical properties.
 What you see one part as determines what you see another part as.
Is it possible to specify a set of ways of physically presenting
a visual stimulus for it to be perceived in a certain way?
Can you think of other ways of presenting a
stimulus so it is perceived as e.g., a Necker Cube?
What does vision construct, and pass
on to cognition?
• Other experiments showing that what is
stored is a symbolic data structure
Errors in recall suggest how visual
information is encoded
Children have very good visual memory,
yet often make egregious errors of recall
• Errors in relative orientation often take a
standard “special” form
• Errors in reproducing a 3D image preserve
3D information
Errors in recall suggest how visual
information is encoded
• Children more often confuse left-right than
rotated forms
• Errors in imitating actions is another source of
evidence
Ability to manipulate and recall patterns depends
on their conceptual, not geometric, complexity
• Difficulty in superimposing shapes depends on
how they are conceptualized
Look at first two shapes and superimpose them in your mind;
then draw (or select one) that is their superposition
Many studies have shown that memory for
shapes is dependent on the conceptual
vocabulary available for encoding them
e.g., recall of chess positions by beginners and masters
There are many other examples showing that it is
how you represent something that is relevant to
cognitive science
Examples from color vision:
“Red light and yellow light mix to produce orange light”
This remains true for any way of getting red light and
yellow light:
e.g. yellow may be light of 580 nanometer wavelength, or
it may be a mixture of light of 530 nm and 650 nm
wavelengths.
So long as one light looks yellow and the other looks red
the “law” will hold.
This is what our conscious experience
suggests goes on in vision…
Kliban
This is what the demands of explanation
suggests must be going on in vision…
Visual representations are not like pictures:
They are mostly conceptual, like descriptions
• If they were pictures they would need to be perceived again!
• Forgetting never consists in segment of pictures being lost
(like a torn photograph)
• The pattern of errors and generalizations shows that they are
stored as conceptual (symbolic) structures.
• This was noted at the turn of the last century by Sir Fredrick
Bartlett who argued that our recall of visual information is
mostly reconstruction from conceptual memory.
Two other considerations that are special
to cognitively determined behavior
1. The Cognitive Penetrability of most cognitive
processes. A regularity that is based on representations
(knowledge) can be systematically altered by imparting
new information that changes beliefs.
2. The critical role of "Cognitive Capacity". Because of
an organism's ecological or social niche, only a small
fraction of its behavioral repertoire is ever actually
observed. Nonetheless an adequate cognitive theory
must account for the behavioral repertoire that is
compatible with the organism's structure, which we call
its cognitive capacity.
Strong Equivalence and the
role of cognitive architecture
The concept of cognitive architecture
 If differences among behaviors (including differences
among individuals) is to be attributed to different
beliefs or different algorithms, then there must be some
common set of basic operations and mechanisms. This
is called the Cognitive Architecture
• The concept of a particular algorithm, or of being “the same
algorithm” is only meaningful if two computers have the
same architecture. Algorithm is architecture-relative.
 The architecture is the part of the system that does not
change when beliefs change. So it defines the system’s
Cognitive Capacity.
On the difference between explanations
that appeal to mental architecture and
those that appeal to tacit knowledge
Suppose we observe some robust
behavioral regularity. What does it tell
us about the nature of the mind or
about its intrinsic properties?
An illustrative example: Mystery Code Box
What does this behavior pattern tell us about the nature of the box?
An illustrative example: Mystery Code Box
Careful study reveals that pattern #2 only occurs in
this special context when it is preceded by pattern A
What does this behavior pattern tell us about the nature of the box?
The Moral:
Regularities in behavior
may be due to either:
1. The inherent nature of the system or
its structure or architecture.
2. The content of what the system
represents (what it “knows”).
Why it matters:
 A great many regular patterns of behavior
reveal nothing more about human nature
than that people do what follows
rationally from what they believe.
 The example of human conditioning:
Brewer, W. F. (1974). There Is No Convincing Evidence for Operant or
Classical conditioning in Adult Humans. In W. B. Weiner & D. S. Palermo
(Eds.), Cognition and the Symbolic Processes. Hillsdale, N.J.: Erlbaum.
Another example where it matters:
The study of mental imagery
Application of the architecture vs knowledge
distinction to understanding what goes on when
we reason using mental images
Examples of behavior regularities
attributable to tacit knowledge
• Color mixing, conservation of volume
• The effect of image size ?
• Scanning mental images ?
Color mixing example
Conservation of volume example
Our studies of mental scanning
2
1.8
1.6
scan image
1.4
imagine lights
Latency (secs)
show direction
1.2
1
0.8
0.6
0.4
0.2
0
1
2
3
4
Relative distance on image
(Pylyshyn & Bannon. See Pylyshyn, 1981)
There is even reason to doubt that one can imagine scanning continuously (Pylyshyn & Cohen, 1998)
Can you rotate a mental image?
Which pair of 3D objects is the same except for orientation?
Do mental images have size?
Imagine a very small mouse. Can you see its whiskers?
Now imagine a huge mouse. Can you see its whiskers?
Which is faster?
Why do so many people deny these
obvious facts about mental imagery?
 The power of subjective experience (phenomenology).
The mind-body problem is everywhere: but subjective
experience does not cause behavior! (e.g., conscious will)
 The failure to make some essential distinctions
 Content vs form (the property of images vs the property of
what images are about) {compare the code box example}
 An image of X with property P can mean
1) (An image of X) with property P or
2) An image of (X with property P)
 Capacity vs typical behavior: Architecture vs knowledge
Are all the things we thought
were due to internal pictures
actually due to tacit knowledge?
Other reasons for imagery phenomena:
• Task demands: Imagine that X = What
would it be like if you saw X?
Are there pictures in the brain?
• There is no evidence for cortical
displays of the right kind to explain
visual or imaginal phenomena
So what is in the brain?
• The best hypothesis so far (i.e., the only one
that has not been shown to be clearly on the
wrong track) is that the brain is a species of
computer in which representations of the
world are encoded in the form of symbol
structures, and actions are determined by
calculations (i.e., inferences) based on these
symbolic encodings.
So why does it not feel like we are
doing computations?
 Because the content of our conscious experience is a
very poor guide to what is actually going on that causes
our experiences and our behavior. Science is
concerned with causes, not just correlations.
 Because we can’t assume that the way things seem has
much to do with how it works (e.g., language
understanding)
 As in most sciences, the essential causes are far from obvious
(e.g., why does the earth go around the sun? What is this
table made of ? etc.).
 In the case of cognition, what is going on is a delicate
mixture of the obvious (what Granny or Shakespeare knew
about why people do what they do) and the incredible
We can’t even be sure that we have
the right methods or instruments
If all else fails there is always parsimony and
generality…(they worked well in physics and linguistics!)