Brain, Psycholinguistics, & Cognitive Science

Outline

• How does psycholinguistics fit within the umbrella of cognitive science?

• What do we know about language and the brain?

Inter-relationships

Cognitive Psychology Linguistics Psycholinguistics Cognitive Science Philosophy Computer Science/AI Natural Language Processing Cognitive Neuroscience Neurolinguistics

What do cognitive psychologists do?

Construct theories that describe how mental processing works in real time: – Within some domain, what are the mental representations and how are they manipulated?

– How does information flow through the mind? – How is processing impacted by memory constraints, stimulus quality, mode of input, tasks, etc?

Levels of processing Serial vs. parallel processing Top-down/bottom-up Automatic vs. strategic processing

What do

(formal theoretical)

linguists do?

• Construct formal theories of our linguistic knowledge – Sets of rules/principles/assumptions for generating utterances • Criteria for a good theory – The rules generate all and only grammatical outputs (intuitions) – In the simplest way

Derivational Theory of Complexity • • Miller’s clause-processing model of syntactic processing, motivated by Chomsky’s Transformational Grammar.

Example of taking a linguistic theory and trying to implement it directly as a psychological theory of processing.

Transformational Grammar

• Two levels of syntactic structure, related by transformational rules; accounts for similarity in meaning across different strings.

– Surface structure

((John) (picked __ (the box) up)).

– Transformations

Particle Movement

– Deep Structure

((John) (picked up (the box))).

• Phrase structure rules and lexical insertion rules used to construct DS

Diff SS, Same DS

Transformations can move, delete, or add words. Complex SS ’s require a sequence of transformations – Wh-movement & Subj/Aux Inversion SS:

Which book did Jim buy __?

SS

Jim did buy a book.

DS:

Jim did buy a/which book

– Passivization SS:

The beer was drunk by Jim

.

SS:

Jim drank the beer

.

DS:

Jim drank the beer

.

DTC

1.Determine the surface structure of the sentence

2.Reverse the transformations one by one

3.Recover the deep structure 4.Map DS to meaning

Processing Unit is entire Sentence • Wh-movement & Subj/Aux Inversion SS:

Which book did Jim buy __?

SS:

Jim did buy a book.

DS:

Jim did buy a/which book

• Passivization SS:

The beer was drunk by Jim

.

SS:

Jim drunk the beer

.

DS:

Jim drunk the beer

.

How would you test the DTC?

• Early evidence seemed to support it.

• Later evidence was problematic • No longer a viable theory of sentence comprehension.

Memory & Language

• Semantic Memory: LTM storehouse of conceptual knowledge –What is a cow? What is truth? • Lexicon: LTM storehouse of knowledge about words –Lexical Entry: For each word/morpheme, Spelling, pronunciation, syntactic category, pointer to semantic memory • Episodic Memory: LTM storehouse of our experiences

Spreading Activation Model

(Collins & Loftus, 1975) • Length codes typicality • Link codes type of relation • Some redundancy uses is is a

Commmon Current Assumptions about Semantic Memory & the Lexicon •

Semantic Memory

is a network of concepts, organized by semantic similarity •

Lexicon

is a network of words, organized by phonological similarity • Interconnections link meanings to words

Cognitive Neuroscience

• Cognitive psychologists tend to talk about the architecture of the mind in terms of functionality – E.g., what is the input to word recognition? What is the output?

• Ultimately, the mental operations described by cognitive psychology occur in the brain • In some cases, neuroscience can inform cognitive psychology – E.g., we may be able to learn about how words/concepts are represented by investigating activation patterns in response to different classes of words: action verbs activate motor cortex; perception verbs activate visual cortex

Brain Anatomy (& Language)

– Is language localized in the brain?

– Is language lateralized?

LH lobes

Frontal lobe Temporal lobe Parietal lobe Occipital lobe Cerebellum

Some Lg-relevant areas

Video Clip (15 min)

Brain Story: First among Equals – The first segment has been comparing human and chimp abilities to plan. They conclude that chimps can make and execute plans, but not as far into the future as humans.

– We’ll watch a segment on LG • Aphasia patient • Mapping the brain prior to surgery

What did you learn from the clip?

• What does it mean to have aphasia?

• Do aphasics recover language function? How?

• What is the current view on the role of Broca’s and Wernicke’s areas?

Schiff et al. (2005)

• Do minimally conscious patients process speech?

Yellow= forward; Blue = backward Red = both

Figure 1. Functional maps obtained during listening to narratives

Averaged healthy data

Schiff, N. D. et al. Neurology 2005;64:514-523

Figure 2. Volumes of activation during the passive listening tasks. The 2 patients are in blue and red; Averaged healthy activation in black.

Schiff, N. D. et al. Neurology 2005;64:514-523

Schiff et al.

– paper was published shortly after the Terry Schiavo media/political frenzy.

– How do we decide whether a minimally conscious patient is experiencing a life worth living?

• Is language comprehension relevant?

• Why don’t we care as much about how their brain responds to smells?

Speech & Spoken Word Recognition

Outline

• Why is speech perception difficult for computers to do?

– Problem of Invariance • How do humans do it so easily?

– Bottom-up information (acoustic signal) – Top-down information (higher level context)

Automatic Speech Recognition Follies (David Pogue, NYTimes, 8/15/02) phonemes form different words What I said bookmark it Motorolla modem port a procedure Different phonemes, at underlying and then stick it in the mail movie clips I might add Inscrutable the right or left What was transcribed book market motor roll a mode import upper seizure and dense thicket in the mail move eclipse I my dad in screw double the writer left

The Problem of Invariance

• Individual phonemes do not have invariant acoustic cues.

• There’s a lot of variability in the acoustic signal!

• Variance in the acoustic signal has many sources: – Coarticulation – Differences among speakers – Differences within speakers: yelling/whispering, phone/in-person, etc.

 If the bottom-up acoustic signal doesn’t provide consistent cues, how do we recognize phonemes/words?

Reading a spectrogram formants Can you see invariant cues associated with /i/ ?

If we can’t/don’t rely solely on bottom-up input, how do we recognize speech?

• Perceive speech as (intended?) articulatory gestures, not as acoustic signal?

– McGurk Effect – Sine Wave Speech: There are no essential acoustic properties that enable speech perception. Rather second-order changes in frequency and amplitude over time are responsible. sine wave speech demos • Use top-down information (word and sentence context) to complement bottom-up information – If so, when and how?

Is Speech Special?

Modularity Thesis (Fodor, 1983) – The mind is not a unified whole. In addition to central processes, there are specialized input output modules • Central: decision making • Input: color perception, voice recognition • Output: throwing, touch-typing, articulation – Modules are fast, informationally-encapsulated, mandatory, exhibit characteristic breakdowns, and have shallow outputs.

– Speech perception may be handled by a specialized input module

The Essence of Motor Theory • Speech perception is grounded in our knowledge of speech production. We recognize phonemes by covertly re-creating the articulatory gestures. (Lieberman et al., 1967) • Consistent with philosophy that performance & perception are inextricably linked.

• Assume innate, encapsulated phonetic module

Spoken Word Recognition

• Overcoming the problem of invariance in speech perception – Motor Theory – Top-Down Feedback: Word to Phoneme • TRACE • Cohort theory of Spoken Word Recognition

Evidence for Top-Down influence on speech perception • Phoneme Restoration Effect (Warren, 1970) • Lexical bias in categorical perception task, e.g.

dype

vs.

type

(Clifton & Connine, 1987)

• • • • •

TRACE

(McClelland & Elman, 1986) Interactive connectionist model Nodes in network represent phonetic features, segments, & words Feature nodes activated by consistent input Activation spreads up through network & back down again Predicts top-down effects

Example: initial phoneme in “pick” is ambiguous betwn /b/ & /p/.

(Lexical Bias)

Is word recognition Automatic & Modular?

Automatic Processes – Fast – Do not require attention – Feed-forward (can’t be guided, controlled, or stopped midstream) – Not subject to top-down feedback (informational encapsulation)

Stroop Effect

Name font color

RED GREEN BLUE YELLOW GREEN

What happens if you have to name word?

Differences between spoken and written word recognition • For relatively short words, letters in a written word are processed in parallel – Eye movement data – Word superiority effect – Letter-Search Task • Spoken word unfolds across time – Can recognize some words before they are completely pronounced.

Research on the Lexicon (Outline) • How are lexical entries accessed? (Word Recognition) What is the input?

– Speaking (Ashcraft) – Reading – Listening • How is lexical ambiguity resolved?