Transcript ICoS-4

Question Answering
• Lecture 1 (two weeks ago):
Introduction; History of QA; Architecture of a QA system;
Evaluation.
© Johan Bos
November 2005
• Lecture 2 (last week):
Question Classification; NLP techniques for question
analysis; Tokenisation; Lemmatisation; POS-tagging;
Parsing; WordNet.
• Lecture 3 (today):
Named Entity Recognition;
Anaphora Resolution;
Matching;
Reranking;
Answer Validation.
© Johan Bos
November 2005
The Panda
A panda…
© Johan Bos
November 2005
A panda walks into a cafe.
He orders a sandwich, eats it, then draws
a gun and fires two shots in the air.
A panda…
© Johan Bos
November 2005
“Why?” asks the confused waiter, as the
panda makes towards the exit.
The panda produces a dictionary and
tosses it over his shoulder.
“I am a panda,” he says. “Look it up.”
The panda’s dictionary
© Johan Bos
November 2005
Panda. Large black-and-white bear-like
mammal, native to China.
Eats, shoots and leaves.
Ambiguities
© Johan Bos
November 2005
Eats, shoots and leaves.
VBZ VBZ CC VBZ
Ambiguities
© Johan Bos
November 2005
Eats shoots and leaves.
VBZ NNS CC NNS
Question Answering
• Lecture 1 (two weeks ago):
Introduction; History of QA; Architecture of a QA system;
Evaluation.
© Johan Bos
November 2005
• Lecture 2 (last week):
Question Classification; NLP techniques for question analysis;
Tokenisation; Lemmatisation; POS-tagging; Parsing; WordNet.
• Lecture 3 (today):
Named Entity Recognition;
Anaphora Resolution;
Matching;
Reranking;
Answer Validation.
Architecture of a QA system
corpus
question
Question
Analysis
query
documents/passages
answer-type
question
representation
© Johan Bos
November 2005
answers
IR
Answer
Extraction
Document
Analysis
passage
representation
Architecture of a QA system
corpus
question
Question
Analysis
query
documents/passages
answer-type
question
representation
© Johan Bos
November 2005
answers
IR
Answer
Extraction
Document
Analysis
passage
representation
Recall the Answer-Type Taxonomy
© Johan Bos
November 2005
• We divided questions according to their
expected answer type
• Simple Answer-Type Typology
PERSON
NUMERAL
DATE
MEASURE
LOCATION
ORGANISATION
ENTITY
Named Entity Recognition
© Johan Bos
November 2005
• In order to make use of the answer types, we
need to be able to recognise named
entities of the same types in the corpus
PERSON
NUMERAL
DATE
MEASURE
LOCATION
ORGANISATION
ENTITY
Example Text
© Johan Bos
November 2005
Italy’s business world was rocked by the
announcement last Thursday that Mr. Verdi
would leave his job as vice-president of Music
Masters of Milan, Inc to become operations
director of Arthur Andersen.
Named Entity Recognition
© Johan Bos
November 2005
<ENAMEX TYPE=„LOCATION“>Italy</ENAME>‘s
business world was rocked by the
announcement <TIMEX TYPE=„DATE“>last
Thursday</TIMEX> that Mr. <ENAMEX
TYPE=„PERSON“>Verdi</ENAMEX> would leave
his job as vice-president of <ENAMEX
TYPE=„ORGANIZATION“>Music Masters of Milan,
Inc</ENAMEX> to become operations director
of <ENAMEX TYPE=„ORGANIZATION“>Arthur
Andersen</ENAMEX>.
NER difficulties
• Several types of entities are too
numerous to include in dictionaries
• New names turn up every day
• Different forms of same entities
in same text
– Brian Jones … Mr. Jones
© Johan Bos
November 2005
• Capitalisation
NER approaches
• Rule-based approach
– Hand-crafted rules
– Help from databases of known named
entities
© Johan Bos
November 2005
• Statistical approach
– Features
– Machine learning
© Johan Bos
November 2005
Anaphora
What is anaphora?
• Relation between a pronoun and
another element in the same or earlier
sentence
• Anaphoric pronouns:
– he, she, it, they
© Johan Bos
November 2005
• Anaphoric noun phrases:
– the country,
– that idiot,
– his hat, her dress
Anaphora (pronouns)
• Question:
What is the biggest sector in Andorra’s
economy?
• Corpus:
© Johan Bos
November 2005
Andorra is a tiny land-locked country in
southwestern Europe, between France and Spain.
Tourism, the largest sector of its tiny, well-to-do
economy, accounts for roughly 80% of the GDP.
• Answer: ?
Anaphora (definite descriptions)
• Question:
What is the biggest sector in Andorra’s
economy?
• Corpus:
© Johan Bos
November 2005
Andorra is a tiny land-locked country in
southwestern Europe, between France and Spain.
Tourism, the largest sector of the country’s tiny,
well-to-do economy, accounts for roughly 80% of the
GDP.
• Answer: ?
Anaphora Resolution
• Anaphora Resolution is the task of
finding the antecedents of anaphoric
expressions
• Example system:
© Johan Bos
November 2005
– Mitkov, Evans & Orasan (2002)
– http://clg.wlv.ac.uk/MARS/
Anaphora (pronouns)
• Question:
What is the biggest sector in Andorra’s
economy?
• Corpus:
© Johan Bos
November 2005
Andorra is a tiny land-locked country in
southwestern Europe, between France and Spain.
Tourism, the largest sector of Andorra’s tiny, well-todo economy, accounts for roughly 80% of the GDP.
• Answer: Tourism
Architecture of a QA system
corpus
question
Question
Analysis
query
documents/passages
answer-type
question
representation
© Johan Bos
November 2005
answers
IR
Answer
Extraction
Document
Analysis
passage
representation
Matching
© Johan Bos
November 2005
• Given a question and an expression
with a potential answer, calculate a
matching score
S = match(Q,A)
that indicates how well Q matches A
• Example
– Q: When was Franz Kafka born?
– A1: Franz Kafka died in 1924.
– A2: Kafka was born in 1883.
Semantic Matching
Q: answer(X)
© Johan Bos
November 2005
franz(Y)
kafka(Y)
born(E)
patient(E,Y)
temp(E,X)
A1: franz(x1)
kafka(x1)
die(x3)
agent(x3,x1)
in(x3,x2)
1924(x2)
Semantic Matching
Q: answer(X)
© Johan Bos
November 2005
franz(Y)
kafka(Y)
born(E)
patient(E,Y)
temp(E,X)
A1: franz(x1)
kafka(x1)
die(x3)
agent(x3,x1)
in(x3,x2)
1924(x2)
X=x2
Semantic Matching
Q: answer(x2)
© Johan Bos
November 2005
franz(Y)
kafka(Y)
born(E)
patient(E,Y)
temp(E,x2)
A1: franz(x1)
kafka(x1)
die(x3)
agent(x3,x1)
in(x3,x2)
1924(x2)
Y=x1
Semantic Matching
Q: answer(x2)
© Johan Bos
November 2005
franz(x1)
kafka(x1)
born(E)
patient(E,Y)
temp(E,x2)
A1: franz(x1)
kafka(x1)
die(x3)
agent(x3,x1)
in(x3,x2)
1924(x2)
Y=x1
Semantic Matching
Q: answer(x2)
© Johan Bos
November 2005
franz(x1)
kafka(x1)
born(E)
patient(E,Y)
temp(E,x2)
A1: franz(x1)
kafka(x1)
die(x3)
agent(x3,x1)
in(x3,x2)
1924(x2)
Match score = 3/6 = 0.50
Semantic Matching
Q: answer(X)
© Johan Bos
November 2005
franz(Y)
kafka(Y)
born(E)
patient(E,Y)
temp(E,X)
A2: kafka(x1)
born(x3)
patient(x3,x1)
in(x3,x2)
1883(x2)
Semantic Matching
Q: answer(X)
© Johan Bos
November 2005
franz(Y)
kafka(Y)
born(E)
patient(E,Y)
temp(E,X)
A2: kafka(x1)
born(x3)
patient(x3,x1)
in(x3,x2)
1883(x2)
X=x2
Semantic Matching
Q: answer(x2)
© Johan Bos
November 2005
franz(Y)
kafka(Y)
born(E)
patient(E,Y)
temp(E,x2)
A2: kafka(x1)
born(x3)
patient(x3,x1)
in(x3,x2)
1883(x2)
Y=x1
Semantic Matching
Q: answer(x2)
© Johan Bos
November 2005
franz(x1)
kafka(x1)
born(E)
patient(E,x1)
temp(E,x2)
A2: kafka(x1)
born(x3)
patient(x3,x1)
in(x3,x2)
1883(x2)
E=x3
Semantic Matching
Q: answer(x2)
© Johan Bos
November 2005
franz(x1)
kafka(x1)
born(x3)
patient(x3,x1)
temp(x3,x2)
A2: kafka(x1)
born(x3)
patient(x3,x1)
in(x3,x2)
1883(x2)
E=x3
Semantic Matching
Q: answer(x2)
© Johan Bos
November 2005
franz(x1)
kafka(x1)
born(x3)
patient(x3,x1)
temp(x3,x2)
A2: kafka(x1)
born(x3)
patient(x3,x1)
in(x3,x2)
1883(x2)
Match score = 4/6 = 0.67
Matching Techniques
• Weighted matching
– Higher weight for named entities
• WordNet
– Hyponyms
• Inferences rules
– Example:
© Johan Bos
November 2005
BORN(E) & IN(E,Y) & DATE(Y)  TEMP(E,Y)
© Johan Bos
November 2005
Reranking
Reranking
© Johan Bos
November 2005
• Most QA systems first produce a list of
possible answers…
• This is usually followed by a process
called reranking
• Reranking promotes correct answers to
a higher rank
Factors in reranking
• Matching score
– The better the match with the question, the
more likely the answers
• Frequency
© Johan Bos
November 2005
– If the same answer occurs many times,
it is likely to be correct
Sanity Checking
Answer should be informative
Q: Who is Tom Cruise married to?
A: Tom Cruise
© Johan Bos
November 2005
Q: Where was Florence Nightingale born?
A: Florence
Answer Validation
• Given a ranked list of answers, some of
these might not make sense at all
• Promote answers that make sense
© Johan Bos
November 2005
• How?
• Use even a larger corpus!
– “Sloppy” approach
– “Strict” approach
© Johan Bos
November 2005
The World Wide Web
Answer validation (sloppy)
© Johan Bos
November 2005
• Given a question Q and a set of
answers A1…An
• For each i, generate query Q Ai
• Count the number of hits for each i
• Choose Ai with most number of hits
• Use existing search engines
– Google, AltaVista
– Magnini et al. 2002 (CCP)
Corrected Conditional Probability
• Treat Q and A as a bag of words
– Q = content words question
– A = answer
hits(A NEAR Q)
• CCP(Qsp,Asp) = ------------------------------
© Johan Bos
November 2005
hits(A) x hits(Q)
• Accept answers above a certain CCP
threshold
Answer validation (strict)
• Given a question Q and a set of
answers A1…An
• Create a declarative sentence with the
focus of the question replaced by Ai
• Use the strict search option in Google
© Johan Bos
November 2005
– High precision
– Low recall
• Any terms of the target not in the
sentence as added to the query
Example
© Johan Bos
November 2005
• TREC 99.3
Target: Woody Guthrie.
Question: Where was Guthrie born?
• Top-5 Answers:
1) Britain
* 2) Okemah, Okla.
3) Newport
* 4) Oklahoma
5) New York
Example: generate queries
© Johan Bos
November 2005
• TREC 99.3
Target: Woody Guthrie.
Question: Where was Guthrie born?
• Generated queries:
1) “Guthrie was born in Britain”
2) “Guthrie was born in Okemah, Okla.”
3) “Guthrie was born in Newport”
4) “Guthrie was born in Oklahoma”
5) “Guthrie was born in New York”
Example: add target words
• TREC 99.3
Target: Woody Guthrie.
Question: Where was Guthrie born?
• Generated queries:
© Johan Bos
November 2005
1) “Guthrie was born in Britain” Woody
2) “Guthrie was born in Okemah, Okla.” Woody
3) “Guthrie was born in Newport” Woody
4) “Guthrie was born in Oklahoma” Woody
5) “Guthrie was born in New York” Woody
Example: morphological variants
TREC 99.3
Target: Woody Guthrie.
Question: Where was Guthrie born?
Generated queries:
© Johan Bos
November 2005
“Guthrie is OR was OR are OR were
“Guthrie is OR was OR are OR were
“Guthrie is OR was OR are OR were
“Guthrie is OR was OR are OR were
“Guthrie is OR was OR are OR were
born in Britain” Woody
born in Okemah, Okla.” Woody
born in Newport” Woody
born in Oklahoma” Woody
born in New York” Woody
Example: google hits
TREC 99.3
Target: Woody Guthrie.
Question: Where was Guthrie born?
Generated queries:
© Johan Bos
November 2005
“Guthrie is OR was OR are OR were
“Guthrie is OR was OR are OR were
“Guthrie is OR was OR are OR were
“Guthrie is OR was OR are OR were
“Guthrie is OR was OR are OR were
born in Britain” Woody 0
born in Okemah, Okla.” Woody 10
born in Newport” Woody 0
born in Oklahoma” Woody 42
born in New York” Woody 2
Example: reranked answers
© Johan Bos
November 2005
TREC 99.3
Target: Woody Guthrie.
Question: Where was Guthrie born?
Original answers
Reranked answers
1) Britain
* 2) Okemah, Okla.
3) Newport
* 4) Oklahoma
5) New York
* 4) Oklahoma
* 2) Okemah, Okla.
5) New York
1) Britain
3) Newport
Summary
• Introduction to QA
– Typical Architecture, Evaluation
– Types of Questions and Answers
• Use of general NLP techniques
– Tokenisation, POS tagging, Parsing
– NER, Anaphora Resolution
© Johan Bos
November 2005
• QA Techniques
– Matching
– Reranking
– Answer Validation
© Johan Bos
November 2005
Where to go from here
•
•
•
•
•
•
•
Producing answers in real-time
Improve accuracy
Answer explanation
User modelling
Speech interfaces
Dialogue (interactive QA)
Multi-lingual QA
© Johan Bos
November 2005
Video (Robot)