CSC 9010 Natural Language Processing Lecture 6: Context-Free Grammars Paula Matuszek Mary-Angela Papalaskari Presentation slides adapted from: Martin: http://www.cs.colorado.edu/~martin/csci5832.html McCoy: http://www.cis.udel.edu/~mccoy/courses/cisc882.03f (after Owen Rambow) 11/6/2015 CSC 9010- NLP -Context-Free Grammars.
Download ReportTranscript CSC 9010 Natural Language Processing Lecture 6: Context-Free Grammars Paula Matuszek Mary-Angela Papalaskari Presentation slides adapted from: Martin: http://www.cs.colorado.edu/~martin/csci5832.html McCoy: http://www.cis.udel.edu/~mccoy/courses/cisc882.03f (after Owen Rambow) 11/6/2015 CSC 9010- NLP -Context-Free Grammars.
CSC 9010 Natural Language Processing Lecture 6: Context-Free Grammars Paula Matuszek Mary-Angela Papalaskari Presentation slides adapted from: Martin: http://www.cs.colorado.edu/~martin/csci5832.html McCoy: http://www.cis.udel.edu/~mccoy/courses/cisc882.03f (after Owen Rambow) 11/6/2015 CSC 9010- NLP -Context-Free Grammars 1 Grammaticality Does NOT depend on • Having heard the sentence before • The sentence being true – Julia Roberts wears green pyjamas • The sentence being meaningful – Colorless green ideas sleep furiously – *Furiously sleep ideas green colorless – My groklar is sklivier than your bosser Grammatically is a formal property that we can investigate and describe 11/6/2015 CSC 9010- NLP -Context-Free Grammars 2 Syntax Words are strung together to form components of sentences which are in turn strung together to form other components or sentences • New Concept: Constituency • Constituent: group of words that behave as a single unit • E.g., noun phrase (NP) 11/6/2015 CSC 9010- NLP -Context-Free Grammars 3 Evidence • Whole group appears in similar syntactic environment (eg before a verb) • Preposed/postposed constructions • Note: notions of meaning play no role in syntax (sort-of) 11/6/2015 CSC 9010- NLP -Context-Free Grammars 4 What is Syntax? • Study of structure of language • Goal: relate surface form to semantics • Morphology, phonology, semantics farmed out (mainly), issue is word order and structure • Representational device is tree structure 11/6/2015 CSC 9010- NLP -Context-Free Grammars 5 What About Chomsky? • At birth of formal language theory (comp sci) and formal linguistics • Major contribution: syntax is cognitive reality • Humans able to learn languages quickly, but not all languages universal grammar is biological • Goal of syntactic study: find universal principles and language-specific parameters • Specific Chomskyan theories change regularly • These ideas adopted by almost all contemporary syntactic theories (“principles-and-parameters-type theories”) 11/6/2015 CSC 9010- NLP -Context-Free 6 Grammars Types of Linguistics • Descriptive: account of syntax of a language; often good enough for NLP engineering work • Explanatory: principles-and-parameters style account of syntax of (preferably) several languages • Prescriptive: “prescriptive linguistics” not very useful in any way “We don’t need no education…” 11/6/2015 CSC 9010- NLP -Context-Free Grammars 7 Syntax • Why should you care? – Grammar checkers – Question answering – Information extraction – Machine translation 11/6/2015 CSC 9010- NLP -Context-Free Grammars 8 Context-Free Grammar Example: • S -> NP VP • NP -> Det NOMINAL • NOMINAL -> Noun • VP -> Verb • Det -> a • Noun -> flight • Verb -> left 11/6/2015 CSC 9010- NLP -Context-Free Grammars Productions 9 Earlier examples S A A → → → b a a A a A ! S → NP VP NP → PrNoun NP → Det Noun Det → a | the Noun → cat | dog| book PrNoun → samantha |elmer | fido VP → IVerb | TVerb NP IVerb → ran |slept | ate TVerb → hit | kissed | ate Regular language Regular? 11/6/2015 CSC 9010- NLP -Context-Free Grammars 10 CFGs • S -> NP VP – This says that there are units called S, NP, and VP in this language – That an S consists of an NP followed immediately by a VP – Doesn’t say that that’s the only kind of S – Nor does it say that this is the only place that NPs and VPs occur 11/6/2015 CSC 9010- NLP -Context-Free Grammars 11 Generativity • As with FSAs and FSTs you can view these rules as either analysis or synthesis machines – Generate strings in the language – Reject strings not in the language – Impose structures (trees) on strings in the language 11/6/2015 CSC 9010- NLP -Context-Free Grammars 12 Derivations • A derivation is a sequence of rules applied to a string that accounts for that string – Covers all the elements in the string – Covers only the elements in the string 11/6/2015 CSC 9010- NLP -Context-Free Grammars 13 Context-Free Grammars • Defined in formal language theory (comp sci) • Terminals, nonterminals, start symbol, rules • String-rewriting system • Start with start symbol, rewrite using rules, done when only terminals left • NOT A LINGUISTIC THEORY, just a formal device 11/6/2015 CSC 9010- NLP -Context-Free Grammars 14 Derivations as Trees Phrase structure tree 11/6/2015 CSC 9010- NLP -Context-Free Grammars 15 Another example - Types of Nodes • (((the/Det) boy/N) likes/V ((a/Det) girl/N)) nonterminal symbols (constituents) S NP DetP the 11/6/2015 boy likes NP DetP a girl terminal symbols = words CSC 9010- NLP -Context-Free Grammars 16 CFG: Example • Many possible CFGs for English, here is an example (fragment): – – – – – – – – – S NP VP VP V NP NP DetP N | AdjP NP AdjP Adj | Adv AdjP N boy | girl V sees | likes Adj big | small Adv very DetP a | the 11/6/2015 the very small likes a girl CSC 9010NLPboy -Context-Free Grammars 17 Derivations in a CFG S S NP VP VP V NP NP DetP N | AdjP NP AdjP Adj | Adv AdjP N boy | girl V sees | likes Adj big | small Adv very DetP a | the 11/6/2015 CSC 9010- NLP -Context-Free Grammars S 18 Derivations in a CFG NP VP S NP VP VP V NP NP DetP N | AdjP NP NP AdjP Adj | Adv AdjP N boy | girl V sees | likes Adj big | small Adv very DetP a | the 11/6/2015 CSC 9010- NLP -Context-Free Grammars S VP 19 Derivations in a CFG DetP N VP S NP VP VP V NP NP DetP N | AdjP NP NP AdjP Adj | Adv AdjP DetP N N boy | girl V sees | likes Adj big | small Adv very DetP a | the 11/6/2015 CSC 9010- NLP -Context-Free Grammars S VP 20 Derivations in a CFG the boy VP S NP VP S VP V NP NP DetP N | AdjP NP NP AdjP Adj | Adv AdjP DetP N N boy | girl V sees | likes the boy Adj big | small Adv very DetP a | the 11/6/2015 CSC 9010- NLP -Context-Free Grammars VP 21 Derivations in a CFG the boy likes NP S NP VP S VP V NP NP DetP N | AdjP NP NP VP AdjP Adj | Adv AdjP DetP N N boy | girl V V sees | likes the boy likes Adj big | small Adv very DetP a | the 11/6/2015 CSC 9010- NLP -Context-Free Grammars NP 22 Derivations in a CFG the boy likes a girl S NP VP S VP V NP NP DetP N | AdjP NP NP VP AdjP Adj | Adv AdjP DetP N N boy | girl V NP V sees | likes the boy likes DetP N Adj big | small Adv very a girl DetP a | the 11/6/2015 CSC 9010- NLP -Context-Free 23 Grammars Derivations in a CFG; Order of Derivation Irrelevant NP likes DetP girl S NP VP VP V NP NP DetP N | AdjP NP NP AdjP Adj | Adv AdjP N boy | girl V sees | likes Adj big | small Adv very DetP a | the 11/6/2015 CSC 9010- NLP -Context-Free Grammars S VP V likes NP DetP N girl 24 Derivations of CFGs • String rewriting system: we derive a string (=derived structure) • But derivation history represented by phrase-structure tree (=derivation structure)! 11/6/2015 CSC 9010- NLP -Context-Free Grammars 25 Grammar Equivalence • Can have different grammars that generate same set of strings (weak equivalence) – Grammar 1: NP DetP N and DetP a | the – Grammar 2: NP a N | NP the N • Can have different grammars that have same set of derivation trees (strong equivalence) – With CFGs, possible only with useless rules – Grammar 2’: DetP many • Strong equivalence implies weak equivalence 11/6/2015 CSC 9010- NLP -Context-Free Grammars 26 Normal Forms &c • There are weakly equivalent normal forms (Chomsky Normal Form, Greibach Normal Form) • There are ways to eliminate useless productions and so on 11/6/2015 CSC 9010- NLP -Context-Free Grammars 27 Nobody Uses CFGs Only (Except Intro NLP Courses) • All major syntactic theories (Chomsky, LFG, HPSG) represent both phrase structure and dependency, in one way or another • All successful parsers currently use statistics about phrase structure and about dependency • Derive dependency through “head percolation”: for each rule, say which daughter is head 11/6/2015 CSC 9010- NLP -Context-Free Grammars 28 What about Computational Complexity – Options to CFG – Regular Grammars – generally claimed to be too weak to capture linguistic generalizations – Context Sentsitive Grammars – generally regarded as too strong – Recursively Enumerable (Type 0) Grammars – generally regarded as way too strong • Approaches that are TOO STRONG have the power to predict/describe/capture syntactic structures that don’t exist in human languages. (But CFG probably not enough) 11/6/2015 CSC 9010- NLP -Context-Free • Computational processes Grammars associated with 29 Massive Ambiguity of Syntax • For a standard sentence, and a grammar with wide coverage, there are 1000s of derivations! • Example: – The large head painter told the delegation that he gave money orders and shares in a letter on Wednesday 11/6/2015 CSC 9010- NLP -Context-Free Grammars 30 Penn Treebank, Again • Syntactically annotated corpus (phrase structure) • PTB is not naturally occurring data! • Represents a particular linguistic theory (but a fairly “vanilla” one) • Particularities – Very indirect representation of grammatical relations (need for head percolation tables) – Completely flat structure in NP (brown bag lunch, pink-and-yellow child seat ) – Has flat Ss, flat 11/6/2015 CSCVPs 9010- NLP -Context-Free Grammars 31 Parsing • Parsing is the process of taking a string and a grammar and returning a (many?) parse tree(s) for that string • It is completely analogous to running a finite-state transducer with a tape – It’s just more powerful • Remember this means that there are languages we can capture with CFGs that we can’t capture with finite-state methods 11/6/2015 CSC 9010- NLP -Context-Free Grammars 32 Other Options • Regular languages (expressions) – Too weak • Context-sensitive or Turing equivalent – Too powerful 11/6/2015 CSC 9010- NLP -Context-Free Grammars 33 Context? • The notion of context in CFGs has nothing to do with the ordinary meaning of the word context in language. • All it really means is that the non-terminal on the left-hand side of a rule is out there all by itself (free of context) A -> B C Means that I can rewrite an A as a B followed by a C regardless of the context in which A is found 11/6/2015 CSC 9010- NLP -Context-Free Grammars 34 Key Constituents (English) • • • • Sentences Noun phrases Verb phrases Prepositional phrases 11/6/2015 CSC 9010- NLP -Context-Free Grammars 35 Sentence-Types • Declaratives: The cat ate my homework S -> NP VP • Imperatives: Fetch! S -> VP • Yes-No Questions: Do you love me? S -> Aux NP VP • WH Questions: Where did that book go? S -> WH Aux NP VP 11/6/2015 CSC 9010- NLP -Context-Free Grammars 36 Recursion • We’ll have to deal with rules such as the following where the non-terminal on the left also appears somewhere on the right (directly). NP -> NP PP [[The flight] [to Boston]] VP -> VP PP [[departed Miami] [at noon]] 11/6/2015 CSC 9010- NLP -Context-Free Grammars 37 Recursion • Of course, this is what makes syntax interesting flights from Denver Flights from Denver to Miami Flights from Denver to Miami in February Flights from Denver to Miami in February on a Friday Flights from Denver to Miami in February on a Friday under $300 Flights from Denver to Miami in February on a Friday under $300 with lunch 11/6/2015 CSC 9010- NLP -Context-Free Grammars 38 Recursion • Of course, this is what makes syntax interesting [[flights] [from Denver]] [[[Flights] [from Denver]] [to Miami]] [[[[Flights] [from Denver]] [to Miami]] [in February]] [[[[[Flights] [from Denver]] [to Miami]] [in February]] [on a Friday]] Etc. 11/6/2015 CSC 9010- NLP -Context-Free Grammars 39 The Point • If you have a rule like – VP -> V NP – It only cares that the thing after the verb is an NP. It doesn’t have to know about the internal affairs of that NP 11/6/2015 CSC 9010- NLP -Context-Free Grammars 40 The Point • VP -> V NP • I hate flights from Denver Flights from Denver to Miami Flights from Denver to Miami in February Flights from Denver to Miami in February on a Friday Flights from Denver to Miami in February on a Friday under $300 Flights from Denver to Miami in February on a Friday under $300 with lunch 11/6/2015 CSC 9010- NLP -Context-Free Grammars 41 Conjunctive Constructions • S -> S and S – John went to NY and Mary followed him • • • • NP -> NP and NP VP -> VP and VP … In fact the right rule for English is X -> X and X 11/6/2015 CSC 9010- NLP -Context-Free Grammars 42 Problems • Agreement • Subcategorization • Movement (for want of a better term) 11/6/2015 CSC 9010- NLP -Context-Free Grammars 43 Agreement • This dog • Those dogs • *This dogs • *Those dog • This dog eats • Those dogs eat • *This dog eat • *Those dogs eats 11/6/2015 CSC 9010- NLP -Context-Free Grammars 44 Handing Number Agreement in CFGs • To handle, would need to expand the grammar with multiple sets of rules – but it gets rather messy quickly. • NP_sg Det_sg N_sg • NP_pl Det_pl N_pl • ….. • VP_sg V_sg NP_sg • VP_sg V_sg NP_pl • VP_pl V_pl NP_sg 11/6/2015 9010- NLP -Context-Free 45 • VP_pl V_plCSC NP_pl Grammars Subcategorization • • • • • • • Sneeze: John sneezed Find: Please find [a flight to NY]NP Give: Give [me]NP[a cheaper fare]NP Help: Can you help [me]NP[with a flight]PP Prefer: I prefer [to leave earlier]TO-VP Told: I was told [United has a flight]S … 11/6/2015 CSC 9010- NLP -Context-Free Grammars 46 Subcategorization • *John sneezed the book • *I prefer United has a flight • *Give with a flight • Subcat expresses the constraints that a predicate (verb for now) places on the number and syntactic types of arguments it wants to take (occur with). 11/6/2015 CSC 9010- NLP -Context-Free Grammars 47 So? • So the various rules for VPs overgenerate. – They permit the presence of strings containing verbs and arguments that don’t go together – For example – VP -> V NP therefore Sneezed the book is a VP since “sneeze” is a verb and “the book” is a valid NP 11/6/2015 CSC 9010- NLP -Context-Free Grammars 48 The Point • CFGs appear to be just about what we need to account for a lot of basic syntactic structure in English. • But there are problems – That can be dealt with adequately, although not elegantly, by staying within the CFG framework. • There are simpler, more elegant, solutions that take us out of the CFG framework (beyond its formal power) 11/6/2015 CSC 9010- NLP -Context-Free Grammars 49