Transcript Natural Logic for Textual Inference

Natural Logic and Natural Language
Inference
Bill MacCartney
Stanford University / Google, Inc.
8 April 2011
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
Two disclaimers
• The work I present today isn’t exactly fresh
• Essentially, it’s my dissertation work from 2009
• I hope it can usefully provide context for more recent work
• I’m a computer scientist, not a semanticist or a logician
• Consequently, I emphasize pragmatism over rigor
2
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
Natural language inference (NLI)
• Aka recognizing textual ‘entailment’ (RTE)
• Does premise P justify an inference to hypothesis H?
• An informal, intuitive notion of inference: not strict logic
• Emphasis on variability of linguistic expression
P
H
Some
Every firm polled saw costs grow more than expected,
even after adjusting for inflation.
Some
Every big company in the poll reported cost increases.
yes
no
• Necessary to goal of natural language understanding
(NLU)
• Can also enable semantic search, question answering,
…
3
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
NLI: a spectrum of approaches
Solution?
natural
logic
(this work)
robust,
but shallow
lexical/
semantic
overlap
deep,
but brittle
Jijkoun & de Rijke 2005
Romano et al. 2006
patterned
relation
extraction semantic
graph
matching
MacCartney et al. 2006
Hickl et al. 2006
Problem:
imprecise  easily confounded by
negation, quantifiers, conditionals,
factive & implicative verbs, etc.
FOL &
theorem
proving
Bos & Markert 2006
Problem:
hard to translate NL to
FOL
idioms, anaphora, ellipsis,
intensionality, tense, aspect,
vagueness, modals, indexicals,
reciprocals, propositional attitudes, scope
ambiguities, anaphoric adjectives, nonintersective adjectives, temporal & causal relations,
unselective quantifiers, adverbs of quantification, donkey
sentences, generic determiners, comparatives, phrasal
4
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
What is natural logic?
( natural deduction)
• Characterizes valid patterns of inference via surface
forms
• precise, yet sidesteps difficulties of translating to FOL
• A long history
• traditional logic: Aristotle’s syllogisms, scholastics, Leibniz, …
• the term “natural logic” was introduced by Lakoff (1970)
• van Benthem & Sánchez Valencia (1986-91): monotonicity
calculus
• Nairn et al. (2006): an account of implicatives & factives
• We introduce a new theory of natural logic
• extends monotonicity calculus to account for negation &
exclusion
• incorporates elements of Nairn et al.’s model of implicatives
5
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
6
‘Entailment’ relations in past work
2-way
RTE1,2,3
X is a couch
X is a crow
X is a fish
X is a hippo
X is a man
X is a sofa
X is a bird
X is a carp
X is hungry
X is a woman
Yes
No
entailment
non-entailment
3-way
Yes
Unknown
No
FraCaS,
PARC, RTE4
entailment
compatibility
contradiction
containment
Sánchez-Valencia
P=Q
P<Q
P>Q
P#Q
equivalence
forward
entailment
reverse
entailment
non-entailment
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
16 elementary set relations
Assign sets x, y to one of
16 relations, depending on
emptiness or nonemptiness of each of four
partitions
y
y
x
?
?
x
?
?
empty
non-empty
7
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
8
16 elementary set relations
But 9 of 16 are
degenerate: either x or y is
either empty or universal.
I.e., they correspond to
semantically vacuous
expressions, which are
rare outside logic
textbooks.
We therefore focus on the
remaining seven relations.
x^y
x⊐y
xy
x⊏y
x y
x|y
x#y
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
The set of 7 basic entailment
relations
Venn
symbo
l
name
example
xy
equivalence
couch  sofa
x⊏y
forward entailment
crow ⊏ bird
x⊐y
reverse entailment
x^y
x|y
x y
x#y
(strict)
(strict)
negation
(exhaustive exclusion)
alternation
(non-exhaustive exclusion)
cover
(exhaustive non-exclusion)
independence
European ⊐ French
human ^ nonhuman
cat | dog
animal  nonhuman
hungry # hippo
Relations are defined for all semantic types: tiny ⊏ small, hover ⊏ fly, kick ⊏
strike,
this morning ⊏ today, in Beijing ⊏ in China, everyone ⊏ someone, all ⊏ most ⊏
9
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
10
Joining entailment relations
x
R
?
y
S
z
⊏
?
fish | human ^ nonhuman
 ⋈ 


⊏ ⋈ ⊏

⊏
⊐ ⋈ ⊐

⊐
^ ⋈ ^


R ⋈ 

R
 ⋈ R

R
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
Some joins yield unions of relations!
What is | ⋈ | ?
x | y
y | z
couch | table
table | sofa
couch  sofa
pistol | knife
knife | gun
pistol ⊏ gun
dog | cat
rose | orchid
woman | frog
cat | terrier
orchid | daisy
frog | Eskimo
x ? z
dog ⊐ terrier
rose | daisy
woman # Eskimo
| ⋈|  {, ⊏, ⊐, |, #}
11
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
The complete join table
Of 49 join pairs, 32 yield relations in
; 17 yield unions
Larger unions convey less information — limits power of
inference
In practice, any union which contains # can be approximated by #
12
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
Lexical entailment relations
( x, e(x will depend on:
) lexical entailment relation generated by e: (e)
) the
1.
atomic edit: DEL, INS, SUB
2. other properties of the context x in which e is applied
compound expression
entailment relation
Example: suppose x is red car
If e is SUB(car, convertible), then (e) is ⊐
If e is DEL(red), then (e) is ⊏
Crucially, (e) depends solely on lexical items in e,
independent of context x
But how are lexical entailment relations determined?
13
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
Lexical entailment relations: SUBs
(SUB(x, y)) = (x, y)
For open-class terms, use lexical resource (e.g.
WordNet)
 for synonyms: sofa  couch, forbid  prohibit
⊏ for hypo-/hypernyms: crow ⊏ bird, frigid ⊏ cold, soar ⊏ rise
| for antonyms and coordinate terms: hot | cold, cat | dog
 or | for proper nouns: USA  United States, JFK | FDR
# for most other pairs: hungry # hippo
Closed-class terms may require special handling
Quantifiers: all ⊏ some, some ^ no, no | all, at least 4  at most 6
See paper for discussion of pronouns, prepositions, …
14
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
Lexical entailment relations: DEL &
INS
Generic (default) case: (DEL(•)) = ⊏, (INS(•)) = ⊐
• Examples: red car ⊏ car, sing ⊐ sing off-key
• Even quite long phrases: car parked outside since last week ⊏ car
• Applies to intersective modifiers, conjuncts, independent
clauses, …
• This heuristic underlies most approaches to RTE!
• Does P subsume H? Deletions OK; insertions penalized.
Special cases
• Negation: didn’t sleep ^ did sleep
• Implicatives & factives (e.g. refuse to, admit that): discussed
later
• Non-intersective adjectives: former spy | spy, alleged spy # spy
• Auxiliaries etc.: is sleeping  sleeps, did sleep  slept
15
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
16
The impact of semantic composition
How are entailment relations affected by semantic
How is (x, y) projected by f?
composition?
?
@
f

x
]
[
@ means
fn application
@
y
f
The monotonicity calculus provides a partial answer
If f has monotonicity…

⊏
⊐
#
UP





⊏
⊐
#
DOWN

⊏
⊐
#





⊐
⊏
#
But how are other relations (|, ^,  ) projected?

⊏
⊐
#
NON





#
#
#
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
A typology of projectivity
Projectivity signatures: a generalization of monotonicity
classes
Each
projectivity signature is a map↦
In principle, 77 possible signatures, but few actually
realized
negation
  
⊏  ⊐
⊐  ⊏
^  ^
|  
  |
#  #
not happy
didn’t kiss
not ill
not human
not French
not more than 4
isn’t swimming

⊐
⊏
^

|
#
not glad
didn’t touch
not seasick
not nonhuman
not German
not less than 6
isn’t hungry
17
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
A typology of projectivity
Projectivity signatures: a generalization of monotonicity
classes
↦
Each projectivity signature is a map
In principle, 77 possible
signatures, but few actually
intersective
negation
modification
realized

⊏
⊐
^
|

#








⊐
⊏
^

|
#

⊏
⊐
^
|

#








⊏
⊐
|
|
#
#
live human | live nonhuman
French wine | Spanish wine
metallic pipe # nonferrous pipe
See my disseration for projectivity of various quantifiers,
verbs
18
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
19
Projecting through multiple levels
Propagate entailment relation between atoms upward,
according to projectivity class of each node on path to
root
nobody can enter without a shirt ⊏ nobody can enter without clothes
⊏
⊐
@
@
⊐
@
@
@
nobody can without a shirt enter
@
⊐
@
⊏
@
nobody can without clothes enter
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
Implicatives & factives
20
[Nairn et al. 06]
9 signatures, per implications (+, –, or o) in positive and negative contexts
signatur
e
implicative
s
implicative
s
factives
example
+/–
he managed to escape
+/o
he was forced to sell
o/–
he was permitted to live
–/+
he forgot to pay
–/o
he refused to fight
o/+
he hesitated to ask
+/+
he admitted that he knew
–/–
he pretended he was sick
o/o
he wanted to fly
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
21
Implicatives & factives
We can specify relation generated by DEL or INS of each signature
signatur
e
implicative
s
implicative
s
factives
example
(DEL
)
(INS)
+/–
he managed to escape

he escaped


+/o
he was forced to sell
⊏
he sold
⊏
⊐
o/–
he was permitted to live
⊐
he lived
⊐
⊏
–/+
he forgot to pay
^
he paid
^
^
–/o
he refused to fight
|
he fought
|
|
o/+
he hesitated to ask

he asked


+/+
he admitted that he knew
⊏
he knew
⊏
⊐
–/–
he pretended he was sick
|
he was sick
|
|
Factives
fully
explained:
he didn’t
admit
that
he knew
| he#|didn’t
know
Room more
fornot
variation
w.r.t.
infinitives,
complementizers,
etc.
Some
intuitive
when
negated:
he
didn’t
hesitate
topassivation,
ask
he didn’t
ask
he wanted to fly # he flew
o/o
#
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
Putting it all together
1. Find a sequence of edits e1, …, en which transforms p into h.
Define x0 = p, xn = h, and xi = ei(xi–1) for i  [1, n].
2. For each atomic edit ei:
a. Determine the lexical entailment relation (ei).
b. Project (ei) upward through the semantic composition tree of
expression xi–1 to find the atomic entailment relation (xi–1, xi)
3. Join atomic entailment relations across the sequence of edits:
(p, h) = (x0, xn) = (x0, x1) ⋈ … ⋈ (xi–1, xi) ⋈ … ⋈ (xn–1, xn)
Limitations: need to find appropriate edit sequence connecting p
and h;
tendency of ⋈ operation toward less-informative entailment
relations; lack of general mechanism for combining multiple
premises
Less deductive power than FOL. Can’t handle e.g. de Morgan’s
22
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
23
An example
P
H
i
The doctor didn’t hesitate to recommend Prozac.
The doctor recommended medication.
yes
ei
xi
lex atom join
The doctor didn’t hesitate to recommend Prozac.
1
DEL(hesitate to)

|
|
^
^
⊏
⊏
⊏
⊏ yes
The doctor didn’t recommend Prozac.
2
DEL(didn’t)
The doctor recommended Prozac.
3
SUB(Prozac, medication)
The doctor recommended medication.
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
24
Different edit orders?
i
ei
lex atom join
i
ei
1
DEL(hesitate to)

|
2
DEL(didn’t)
^
3
SUB(Prozac, medication)
⊏
i
ei
lex atom join
1
DEL(didn’t)
^
^
2
DEL(hesitate to)

3
SUB(Prozac, medication)
⊏
i
ei
1
lex atom join
|
1
DEL(hesitate to)

|
|
^
⊏
2
SUB(Prozac, medication)
⊏
⊐
|
⊏
⊏
3
DEL(didn’t)
^
^
⊏
i
ei
^
1
DEL(didn’t)
^
^
^

⊏
2
SUB(Prozac, medication)
⊏
⊐
|
⊏
⊏
3
DEL(hesitate to)


⊏
lex atom join
i
ei
lex atom join
SUB(Prozac, medication)
⊏
⊏
⊏
1
SUB(Prozac, medication)
⊏
⊏
⊏
2
DEL(hesitate to)

|
|
2
DEL(didn’t)
^
^
|
3
DEL(didn’t)
^
^
⊏
3
DEL(hesitate to)


⊏
lex atom join
Intermediate steps may vary; final result is typically (though not necessarily) the sam
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
Implementation & evaluation
The NatLog system: an implementation of this model in
code
For implementation details, see [MacCartney & Manning 2008]
Evaluation on FraCaS test suite
183 NLI problems, nine sections, three-way classification
Accuracy 70% overall; 87% on “relevant” sections (60%
coverage)
Precision 89% overall: rarely predicts entailment wrongly
Evaluation on RTE3 test suite
Longer, more natural premises; greater diversity of inference
types
NatLog alone has mediocre accuracy (59%) but good precision
Hybridization with broad-coverage RTE system yields gains of
25
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
Conclusions
Natural logic is not a universal solution for NLI
Many types of inference not amenable to natural logic
approach
Our inference method faces many limitations on deductive
power
More work to be done in fleshing out our account
Establishing projectivity signatures for more quantifiers, verbs,
etc.
Better incorporating presuppositions
But, our model of natural logic fills an important niche
:-)
Precise reasoning on negation, antonymy, quantifiers,
implicatives, …
Sidesteps the myriad difficulties of full semantic interpretation
Practical value demonstrated on FraCaS and RTE3 test suites
Thanks! Questions?
26
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
Backup slides follow
27
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
28
An example involving exclusion
P
H
i
Stimpy is a cat.
Stimpy is not a poodle.
ei
xi
yes
lex atom join
Stimpy is a cat.
1
SUB(cat, dog)
|
|
|
^
^
⊏
⊐
⊏
⊏ yes
Stimpy is a dog.
2
INS(not)
Stimpy is not a dog.
3
SUB(dog, poodle)
Stimpy is not a poodle.
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
29
An example involving an implicative
P
H
i
We were not permitted to smoke.
We smoked Cuban cigars.
ei
xi
no
lex atom join
We were not permitted to smoke.
1
DEL(permitted to)
⊐
⊏
⊏
^
^
|
⊐
⊐
|
We did not smoke.
2
DEL(not)
We smoked.
3
INS(Cuban cigars)
We smoked Cuban cigars.
no
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
30
de Morgan’s Laws for quantifiers
P
H
i
Not all birds fly.
Some birds do not fly.
ei
xi
yes
lex atom join
Not all birds fly.
1
DEL(not)
^
^
^
⊏
⊏

^

All birds fly.
2
SUB(all, some)
Some birds fly.
3
INS(not)
Some birds do not fly.
⊏⊐ #
wtf??
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
31
de Morgan’s Laws for quantifiers (2)
P
H
i
Not all birds fly.
Some birds do not fly.
ei
xi
yes
lex atom join
Not all birds fly.
1
DEL(not)
^
^
^
^
|
⊐
⊏
⊏
All birds fly.
2
INS(not)
All birds do not fly.
3
SUB(all, some)
Some birds do not fly.
⊏⊐ #
wtf??
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
32
A more complex example
P
H
i
Jimmy Dean refused to move without blue jeans.
James Dean didn’t dance without pants.
yes
ei
lex atom join
1
SUB(Jimmy Dean, James Dean)



2
DEL(refuse to)
|
|
|
3
INS(did)


|
4
INS(n’t)
^
^
⊏
5
SUB(move, dance)
⊐
⊏
⊏
6
DEL(blue)
⊏
⊏
⊏
7
SUB(jeans, pants)
⊏
⊏
⊏
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
33
A more complex example (2)
P
H
i
Jimmy Dean refused to move without blue jeans.
James Dean didn’t dance without pants.
yes
ei
lex atom join
1
INS(did)



2
INS(n’t)
^
^
^
3
DEL(blue)
⊏
⊐
|
4
SUB(jeans, pants)
⊏
⊐
|
5
SUB(move, dance)
⊐
⊐
|
6
DEL(refuse to)
|

⊏
7
SUB(Jimmy Dean, James Dean)


⊏
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
34
A more complex example (3)
P
H
i
Jimmy Dean refused to move without blue jeans.
James Dean didn’t dance without pants.
yes
ei
lex atom join
1
INS(did)



2
INS(n’t)
^
|
|
6
DEL(refuse to)
|
|
⊏⊐|
#
3
DEL(blue)
⊏
⊏
•
4
SUB(jeans, pants)
⊏
⊏
•
5
SUB(move, dance)
⊐
⊐
•
7
SUB(Jimmy Dean, James Dean)


•