Transcript Document

Language Embedded
in Systems
of Perception and Action
Rolf A. Zwaan
Florida State University
Experiential Traces
Experiential Traces
“referential”
linguistic
Experiential Associations
“intra-referential”
intra-linguistic
Experiential Associations
cross-domain
Hebbian Learning

Co-occurrence primary factor in the
forming of associations
Intra-linguistic;
 Intra-referential;
 Cross-domain (grounding).

Experiential Traces

Linguistic Traces

Referential Traces

Auditory
Motor
Visual
Tactile

Visual
Motor
Auditory
Olfactory
Tactile
Somatosensory








Capturing
Second-order Correlations
cosine
LSA/Encyclopedia_all
0.5
0.45
0.4
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0
r2 = .30
0
1000
2000
distance from Miami (miles)
3000
4000
Capturing
Second-order Correlations
0.8
r2 =.65
0.7
LSA cosine
0.6
0.5
0.4
0.3
0.2
0.1
0
0
1
2
3
distance from the Netherlands (# of intervening countries)
Utility of Linguistic
Traces/Associations

Additional layer of associations.


May enhance “fluency.”
“Referential bootstrapping.”
Referential Bootstrapping

Name this animal:

What has horns like a giraffe, a deer head, a horse
neck, and legs like a zebra?
http://www.westerville.k12.oh.us/ZOO/Orlich/pages/Okafrtpg.htm
Referential Bootstrapping
Resonance

Experiential traces “resonate” with
linguistic input.
Visual resonance.
 Motor resonance.


The role of resonance in comprehension.
Predictions

Prediction 1


Prediction 2


Language comprehension affects subsequent
perceptual performance.
Incidentally acquired visual representations
affect subsequent language processing.
Prediction 3

Language comprehension affects concurrent
motor performance.
Comprehension Affects
Perceptual Performance
(Zwaan, Stanfield, & Yaxley, 2002)



The ranger saw the eagle in the nest.
[SAW, RANGER, EAGLE][IN, EAGLE, NEST]
The ranger saw the eagle in the sky.
[SAW, RANGER, EAGLE][IN, EAGLE, SKY]


The ranger saw the eagle in its nest.
The ranger saw the eagle in the sky.
Picture Recognition
and Naming Latencies
(Zwaan, Stanfield & Yaxley, 2002)
800
750
RT (ms)
Match
Mismatch
700
650
600
Recognition
Naming
Language may Involve
Dynamic Representations
(Zwaan. Madden, Yaxley, & Aveyard, 2004)
The midfielder passed the ball to you.
Zwaan, Madden, Yaxley, & Aveyard, 2004
Prediction 2

Incidentally acquired visual
representations affect subsequent
language processing.
The Visual-Memory Paradigm
(Aveyard, Zwaan, Radach, & Vorstius, in preparation)

Memory Phase


Word-picture verification (4 times)
Reading Phase (ostensibly unrelated)

Eye movements tracked
Memory Phase
eagle
Memory Phase
Reading Phase
The ranger was in the park on his morning
round. It was a bright day. In the sky there
was an eagle, soaring above the prairie.
First Fixations
650
600
550
500
Match
Mismatch
450
400
350
300
In the sky
an eagle
was
soaring
above the
…
Gaze Durations
850
800
750
700
Match
Mismatch
650
600
550
500
In the sky
an eagle
was
soaring
above the
…
Visual World




Situated
Visual Search
Auditory Presentation
DV = eye fixations
during display viewing
Visual Memory




Re-situated
Memory Search
Visual Presentation
DV = eye fixations
during reading
Prediction 3

Motor traces are activated by linguistic
input.
Action Compatibility Effect
(Glenberg & Kaschak, 2002)

Sensibility judgments
Close the drawer.
 Andy delivered the pizza to you.
 Liz told you the story.
 Break the cloud.

Action Parameters
and Motor Resonance

Actions have various parameters.
Direction.
 Force.
 Manner (e.g., open hand vs. closed hand).
 Speed.
 Duration.


Which action parameters exhibit
resonance in language comprehension?
Measuring Response Force
Force (kPa)
Measuring Response Force
45000
40000
35000
30000
25000
20000
15000
10000
5000
0
A
90%
T
10%
Time
Atmospheric pressure = 100 kPa
Response Force
(Zwaan & Taylor, in prep.)


Sensibility judgments.
Verbs.




He pushed the car. (high)
He started the car. (low)
He admired the car. (no)
Nouns.



He broke the baseball bat. (high)
He broke the bread. (low)
He broke the promise. (no)
Response Force: Verbs
21000
20000
Response
Amplitude
(kPa)
19000
High
Lo w
No
18000
17000
16000
15000
Force
Response Force: Nouns
19000
18500
Response
Amplitude
(kPa)
18000
17500
High
Lo w
No
17000
16500
16000
15500
15000
Force
Manner of Action

Sustain vs. Release
He gripped the hammer.
 He dropped the hammer.


Predictions
Sustain faster than Release (ACE).
 Sustain longer than Release (ACE).

Sustain vs. Release
Response Speed
1600
1500
1400
Sustain
Release
1300
1200
1100
1000
Sustain
Release
Sustain vs. Release
Force (kPa)
Response Duration
45000
40000
35000
30000
25000
20000
15000
10000
5000
0
10%
Time
Sustain vs. Release
320
Response Duration >10%
300
280
Sustain
Release
260
240
220
200
Sustain
Release
Motor Resonance

What is the role of motor representations
in language comprehension?
Instrumental (Glenberg)
 Ornamental (many skeptics)
 Useful/necessary in specific cases
(Jackendoff)

Motor Resonance

What is the role of motor representations
in language comprehension?

Useful/necessary when actions are
Concrete
 Punctate
 Simple


Let’s first ask: How is motor resonance
modulated by linguistic input?
How is motor resonance
modulated by linguistic
input?
(Zwaan & Taylor, 2006, JEP General)

Reading by rotating.



Every 4 deg. new frame.
He /realized /that /the music /was /too
loud/so he /turned down/the /volume.
60 psychology undergraduates.
Reading-by-Rotation
380
370
Reading Time
360
350
Match
Mismatch
340
330
320
310
300
He...he
turned down
Sentence Region
the
volume
Is Motor Resonance
Short-lived or Focus-bound?
(Taylor & Zwaan, in prep.)

Perspective change.


Maintaining focus on the action.


After/lighting/the candles/for
the/romantic/evening,/he/dimmed/the/lights.
He lit/the candles/for the/romantic/evening./He
noticed/the bright/lights,/which he/dimmed/slowly.
Reading-by-rotating.
Motor Resonance is Focus-bound
580
560
540
Mismatch
Match
520
500
480
460
volume w hich turned
he
down
slow ly

Strong Test
of Linguistic Focus
Hypothesis
Action-modifying vs. subject-modifying
adverbs
… which he turned down slowly.
 …which he turned down obediently.


Stay tuned.
Conclusions & Outlook

Language is embedded in systems of
perception and action.


Rapid exchange of information.
But it needs to be determined to what
extent it is in bed with these systems.

Perceptual & motor resonance ornamental or
instrumental?
Acknowledgements


Grants from the National Science Foundation (NSF) and
the National Institute of Mental Health (NIMH)
Graduate students (past & present)


Postdoctoral Fellows (past & present)


Rob Stanfield, Carol Madden, Rich Yaxley, Mark Aveyard, Larry
Taylor
Barbara Kaup, Dave Therriault, Johanna Kaakinen, Tobias
Richter
Faculty Collaborators

Mike Kaschak, Ralph Radach, Katinka Dijkstra