Introduction to Natural Language Processing (aka, Computational Linguistics)

Slides by me, Martha Palmer, Eleni Miltsakaki, Dan Jurafsky, Tarkan Kacmaz, and others 1

Overview

• NLP without linguistics (4-5 weeks) – Information Retrieval (search) – Text Classification – Pattern Matching and Information Extraction • NLP with sequence structure (~3 weeks) – HMMs, CRFs – Sequence labeling tasks • NLP with more structure (~3 weeks) – Grammars and parsing – Learning grammars – Semantic role labeling • Selected topics (~2 weeks) – Learning representations and domain adaptation – Knowledge-based language processing 2

Practical Matters

• Prereqs:

General understanding of probability and statistics

• Grading: • 20% quizzes and in-class participation • 25% Midterm • 20% Project • 35% Final • I will supply some ideas for projects later – Projects to start after the midterm.

– You’re welcome and encouraged to suggest your own project ideas.

3

WHAT IS LANGUAGE?

When we study human language, we are approaching what some might call the “human essence”, the distinctive qualities of mind that are, so far as we know, unique to man.

Noam Chomsky

WHAT IS LANGUAGE?

• Definition with respect to form:

Language is a system of speech symbols. It is realized acoustically (sound waves), visually-spatially (sign language) and in written form.

• Definition with respect to function:

Language is the most important means of human communication. It is used to convey and exchange information (informative function)

• Multiplicity of languages: We know of about 7000 languages, which is estimated to be about 1% of all the languages that ever existed.

LANGUAGE AND THE BRAIN

LANGUAGE AND THE BRAIN

THEORIES OF LANGUAGE

•

Noam Chomsky claims that language is innate.

•

B. F. Skinner claims that language is learned; it is basically a stimulus-response mechanism.

WHAT IS

GRAMMAR

?

• When we learn a language we also learn the rules that govern how language elements, such as words, are combined to produce meaningful language.

• These elements and rules constitute the

Grammar

a language.

of • The

Grammar

is “what we know” •

Grammar

represents our linguistic competence.

DESCRIPTIVE vs PRESCRIPTIVE GRAMMAR

Prescriptive

Descriptive

(should be)

(is)

Areas of Linguistics

• • • •

phonetics phonology

- the study of speech sounds

morphology syntax

- the study of sound systems - the rules of word formation - the rules of sentence formation • •

semantics pragmatics

- the study of word meanings – the study of discourse meanings •

sociolinguistics

- the study of language in society •

applied linguistics

–the application of the methods and results of linguistics to such areas as language teaching, national language policies, lexicography, translation, language in politics etc.

What is phonetics?

• Phonetics is the science of speech.

• We all speak.

• But how many of us know how we speak? • Or what speech is like?

• Phonetics seeks to answer those questions.

Orthography and Sounds

• The English language is not

phonetic

.

• Words are not spelled as they are pronounced • There is no one-to-one correspondence between the

letters

and the

sounds

or

phonemes

.

Orthography and Sounds

• Did h

e

bel

ie

ve that C

ae

sar could s

ee

the p

eo

ple s

ei

ze the s

ea

s.

• The sill

y

mach

i

ne am

oe

ba stole the k

ey

to the

Articulatory Phonetics

• The production of any speech sound involves the movement of an air stream.

• Most speech sounds are produced by pushing the air out of the lungs through the mouth (oral) and sometimes through the nose (nasal).

SPEECH ORGANS

Phonology

•

Phonology

deals with the system and pattern of speech sounds in a language.

•

Phonology of a language

is the system and pattern of speech sounds.

Phonology

Phonological knowledge permits us to: • produce sounds which form meaningful utterances, • to recognize a “foreign” accent, • to make up new words, • To know what is or is not a sound in one’s language • to know what different sound strings may represent

Phonetics vs Phonology

Phonetics The study of speech sounds. Phonology The study of the way speech sounds form patterns.

Sequences of Phonemes

b l ı k k l ı b b ı l k k ı l b k b

possible

l ı I b k ı ı l b k b k ı l ı b l k •“I just bought a beautiful new

blick

” What is a blick?

•“I just bought a beautiful new

bkli

” WHAT!!

impossible

Sequences of Phonemes

• Your knowledge of English “tells” you that certain strings of phonemes are permissible and others are not.

• That’s why /bkli/ does not sound like an English word.

• It violates the restrictions on the sequencing of phonemes; i.e. it violates the phonological rules of English.

Rules of Phonology

• Delete a word-final /b/ when it occurs after a /m/ as in: But not! bomb crumb lamb tomb bombard crumble limber tumble

Morphology & Syntax

•

Morphology

deals with the combination of morphemes into words.

•

Syntax

deals with the combination of words into sentences.

What is the meaning of ‘meaning’?

• Learning a language includes learning the “agreed upon” meanings of certain strings of sounds and, • Learning how to combine these meaningful units into larger units which also convey meaning.

Morphemes

•

Morpheme

is the smallest linguistic unit that has meaning.

•

Morpheme

is a grammatical unit in which there is an arbitrary union of sound and a meaning and, • which cannot be further analysed (broken down into parts that have meaning).

Morphemes

• A morpheme may be represented by a single sound: • e.g. the plural morpheme [s] in cat+s • A morpheme may be represented by a syllable (monosyllabic): • e.g. child+ish

Morphemes

A morpheme may be represented by more than one syllable (polysyllabic): • e.g. lady, water or three syllables: • e.g. crocodile or four syllables: • e.g. salamander

Words

• Two basic ways to form words – Inflectional (e.g. English verbs) • Open + ed = opened • Open + ing = opening – Derivational (e.g. adverbs from adjectives, nouns from adjectives) • Happy  • Happy  happily happiness (nouns from adjectives) 32

Syntax

The study of

classes of words

and the

rules

that govern how the words can combine to make phrases and sentences.

33

Basic classes of words

• Classes of words aka

parts of speech

– Nouns – Verbs – Adjectives – Adverbs (POS) • The above classes of word belong to the type

open class words

• We also have – Articles, pronouns, prepositions, particles, quantifiers, conjunctions

closed class words

34

Basic phrases

• A word from an open class can be used to form the basis of a phrase • The basis of a phrase is called the head 35

Examples of phrases

• Noun phrases – The manager of the institute – Her worry to pass the exams – Several students from the English Department • Adjective phrases – easy to understand – mad as a dog – glad that he passed the exam 36

Examples of phrases

• Adverb phrases – fast like the wind – outside the building • Verb phrases – ate her sandwich – went to the doctor – believed what I told him 37

“Complements”

• Notice that to be meaningful the verb “go”, for example requires a phrase for “location” – *John went – John went home • Such phrases “complete” the meaning of the verb (or other type of head) and are called

complements

38

Inside the noun phrase

• NPs are used to refer to things: objects, places, concepts, events, qualities, etc • NPs may consist of: – A single pronoun (he, she, etc) – A name or proper noun (John, Athens, etc) – A specifier and a noun – A qualifier and a noun – A specifier and a qualifier and a noun (e.g., the first three winners) 39

Specifiers

• Specifiers indicate

how many objects are described

and also

how these objects relate to the speaker

• Basis types of specifiers – Ordinals (e.g., first, second) – Cardinals (e.g., one, two) – Determiners (see next slide) 40

Determiners

• Basic types of determiners – Articles (the, a, an) – Demonstratives (this, that, these, those) – Possessives (‘s, her, my, whose, etc) – Wh-determiners (which, what –in questions) – Quantifying determiners (some, every, most, no, any, etc.) 41

Qualifiers

• Basic types of qualifiers – Adjectives • Happy cat • Angry feelings – Noun modifiers • Cook book • University hospitals 42

Inside the verb phrase

• A simple VP – Adverbial modifier + head verb + complements • Types of verbs – Auxiliary (be, do, have) – Modal (will, can, could) – Main (eat, work, think) 43

Types of verb complements

• Intransitive verbs do not require complements • Transitive verbs require an object as a complement (e.g. find a key) • Transitive verbs allow passive found) forms (e.g. a key was • Ditransitive verbs require one direct object (e.g. give Mary a book) and on indirect 44

Other verb complements

• Clausal complements – Some verbs require clausal complements • Mary knows that John left • Prepositional phrase complements – Some verbs requires specific PP complements • Mary gave the book to John – Others require any PP complement • John put the book on the shelf/in the room/under the table 45

Adjective phrases

• Simple – Angry, easy, etc • Complex – Pleased with the prize – Angry at the committee – Willing to read the book • Complex AdjP normally do not precede nouns, they are used as complements of verbs such as

be

or

seem

46

Adverbial phrases

• Indicators of – Degree – Location – Manner – The time of something (now, yesterday, etc) – Frequency – Duration • Location in the sentence – Initial – Medial – Final 47

Grammars and parsing

• What is syntactic parsing – Determining the syntactic structure of a sentence • Basic steps – Identify sentence boundaries – Identify what part of speech is each word – Identify syntactic relations 48

Context Free Grammar

• S -> NP VP • NP -> det (adj) N • NP -> Proper N • NP -> N • VP -> V, VP -> V PP • VP -> V NP • VP -> V NP PP, PP -> Prep NP • VP -> V NP NP 49

Parses

Det the S The cat sat on the mat NP N cat VP V sat Prep on PP Det the NP N mat

50

Parses

NP N time S Time flies like an arrow.

VP V flies PP Prep like Det an NP N arrow

51

Parses

S NP N time N flies V like VP Det an Time flies like an arrow.

NP N arrow

52

Features

• C for Case, Subjective/Objective –

She visited her.

• P for Person agreement, (1 st , 2 nd , 3 rd ) –

I like him, You like him, He likes him,

• N for Number agreement, Subject/Verb –

He likes him, They like him.

• G for Gender agreement, Subject/Verb – English, reflexive pronouns • T for Tense, – *

will finished

is bad

He washed himself.

– Romance languages, det/noun – auxiliaries, sentential complements, etc. 53

Semantics and Pragmatics

High-level Linguistics (the good stuff!)

Semantics: the study of meaning that can be determined from a sentence, phrase or word.

Pragmatics: the study of meaning, as it depends on context (speaker, situation) 54

Language to Logic

• •

John went to the book store.

 John  store1 , go(John, store1)

John bought a book.

buy(John,book1) •

John gave the book to Mary.

give(John,book1,Mary) •

Mary put the book on the table.

put(Mary,book1,table1) 55

Semantics Same event - different sentences

John broke the window with a hammer.

John broke the window with the crack.

The hammer broke the window.

The window broke.

56

Same event - different syntactic frames

John broke the window with a hammer.

SUBJ VERB OBJ MODIFIER

John broke the window with the crack.

SUBJ VERB OBJ MODIFIER

The hammer broke the window.

SUBJ VERB OBJ

The window broke.

SUBJ VERB

57

Semantics -predicate arguments

break(AGENT, INSTRUMENT, PATIENT) AGENT PATIENT INSTRUMENT John broke the window with a hammer.

INSTRUMENT PATIENT The hammer broke the window.

PATIENT The window broke.

Fillmore 68 - The case for case

58

AGENT PATIENT INSTRUMENT John broke the window with a hammer.

SUBJ OBJ MODIFIER

INSTRUMENT PATIENT The hammer broke the window.

SUBJ OBJ

PATIENT The window broke.

SUBJ

59

Headlines

• Police Begin Campaign To Run Down Jaywalkers • Iraqi Head Seeks Arms • Teacher Strikes Idle Kids • Miners Refuse To Work After Death • Juvenile Court To Try Shooting Defendant 60

Language Families

NLP tends to focus on:

• Syntax – Grammars, parsers, parse trees, dependency structures • Semantics – Subcategorization frames, semantic classes, ontologies, formal semantics • Pragmatics – Pronouns, reference resolution, discourse models 62

Natural Language Processing Applications and Tasks

• • • Machine Translation • Question-Answering

Information Retrieval Information Extraction

63

Machine Translation • One of the first applications for computers – bilingual dictionary > word-word translation • Good translation requires

understanding!

–

War and Peace, The Sound and The Fury

?

• What can we do? Sublanguages.

– technical domains, static vocabulary – Meteo in Canada, Caterpillar Tractor Manuals, Botanical descriptions, Military Messages 64

Example translation

65

Machine Translation

• Chinese gloss : Dai-yu alone on bed top think-of-with-gratitude Bao chai again listen to window outside bamboo tip plantain leaf of on top rain sound sigh drop clear cold penetrate curtain not feeling again fall down tears come • Hawkes translation : As she lay there alone, Dai yu’s thoughts turned to Bao chai… Then she listened to the insistent rustle of the rain on the bamboos and plantains outside her window. The coldness penetrated the curtains of her bed. Almost without noticing it she had begun to cry.

Machine Translation

• The Story of the Stone – =The Dream of the Red Chamber (Cao Xueqin 1792) • Issues: (“Language Divergences”) – Sentence segmentation – Zero-anaphora – Coding of tense/aspect – Penetrate -> penetrated – Stylistic differences across languages • Bamboo tip plaintain leaf -> bamboos and plantains – Cultural knowledge • Curtain -> curtains of her bed

Question Answering

• What does “archaeopteryx” mean?

• What year was Abraham Lincoln born?

• How many states were in the United States when Lincoln was born?

• Was there a military draft during the Hoover administration?

• What do philosophers think about whether human cloning should be legal?

Modern QA systems

• Still in infancy • Simple factoid questions beginning to work OK • Annual government-sponsored “bakeoff” called TREC

Issues in NLP

• Ambiguity • World Knowledge – it’s needed for understanding, but computers don’t have it 70

Ambiguity

• Computational linguists are obsessed with ambiguity • Ambiguity is a fundamental problem of computational linguistics • Resolving ambiguity is a crucial goal

Ambiguity

• Find at least 5 meanings of this sentence: – I made her duck

Ambiguity

• Find at least 5 meanings of this sentence: – I made her duck • I cooked waterfowl for her benefit (to eat) • I cooked waterfowl belonging to her • I created the (plaster?) duck she owns • I caused her to quickly lower her head or body • I waved my magic wand and turned her into undifferentiated waterfowl • At least one other meaning that’s inappropriate for gentle company.

Ambiguity is Pervasive

• I caused her to quickly lower her head or body –

Lexical category

: “duck” can be a N or V • I cooked waterfowl belonging to her.

–

Lexical category :

“her” can be a possessive (“of her”) or dative (“for her”) pronoun • I made the (plaster) duck statue she owns –

Lexical Semantics :

“make” can mean “create” or “cook”

Ambiguity is Pervasive

•

Grammar

: Make can be: –

Transitive: (verb has a noun direct object)

• I cooked [waterfowl belonging to her] –

Ditransitive: (verb has 2 noun objects)

• I made [her] (into) [undifferentiated waterfowl] –

Action-transitive (verb has a direct object and another verb)

– I caused [her] [to move her body]

Ambiguity is Pervasive

•

Phonetics!

– I mate or duck – I’m eight or duck – Eye maid; her duck – Aye mate, her duck – I maid her duck – I’m aid her duck – I mate her duck – I’m ate her duck – I’m ate or duck – I mate or duck

Kinds of knowledge needed?

• Consider the following interaction with HAL the computer from

2001: A Space Odyssey

• Dave: Open the pod bay doors, Hal.

• HAL: I’m sorry Dave, I’m afraid I can’t do that.

Knowledge needed to build HAL?

• Speech recognition and synthesis – Dictionaries (how words are pronounced) – Phonetics (how to recognize/produce each sound of English) • Natural language understanding – Knowledge of the English words involved • What they mean • How they combine (what is a `pod bay door’?) – Knowledge of syntactic structure • I’m I do, Sorry that afraid Dave I’m can’t

What’s needed?

• Dialog and pragmatic knowledge – “open the door” is a REQUEST (as opposed to a STATEMENT or information-question) – It is polite to respond, even if you’re planning to kill someone.

– It is polite to pretend to want to be cooperative ( I’m afraid I can’t… ) – What is `that’ in `I can’t do that’ ?

• Even a system to book airline flights needs much of this kind of knowledge

Modern conversational agents

• Switchboard Receptionists – Implications: need to know how a native or non-native speaker of English is likely to pronounce a random American name. Eek.

• Airline and other simple dialogue systems – Lots of fabulously fun pragmatic issues • Grounding • Error handling • In general – Now that there are commercial installations – Big incentive to use linguistic or any other kind of knowledge if it actually improves performance

Eliza: Weizenbaum (1966)

• User: You are like my father in some ways.

• ELIZA: What resemblance do you see • User: You are not very aggressive but I think you don’t want me to notice that.

• ELIZA: What makes you think I am not aggressive • User: You don’t argue with me • ELIZA: Why do you think I don’t argue with you • User: You are afraid of me • ELIZA: Does it please you to believe I am afraid of you