Transcript Document

Event Extraction Error Analysis and
Event-Event Relations
Heng Ji
[email protected]
Outline
•
•
•
•
Challenges of Event Extraction 2pm-2:30pm
Event-event Relations 2:30pm-2:50pm
Student Presentations [70mins] 2:50pm-4pm
Causal Relation Exercise 4pm-4:50pm
2
From Entity-Centric to Event-Centric
• State-of-the-art Information Extraction and Knowledge
Discovery Techniques are Entity-Centric
• Event-centric Knowledge Discovery from unstructured
data to organize knowledge in a network structure
• Construct High-quality Event Networks
– Node: an event (a cluster of coreferential event mentions)
• Event Mention Extraction based on Joint Modeling with Structured Prediction
• Event Coreference Resolution based on semantic attributes
– Edge: relation between events
• Some are horizontal and some are vertical, forming an event cone
3
From Entity-Centric to Event-Centric
COREFERENCE
CAUSAL
SUB-EVENT
Barack Obama
US congress
Obama
(approver)
[Title: President
Employer: US]
(agent)
(attendee)
CLOSE-ORG
SIGN
COREFERENCE
Obamacare
(act)
2010/03/03
Doc1
Healthcare US
reform government
MEET
John Boehner
NSF
(organization)
CLOSE-ORG
DARPA
2013/10/02
Doc3
2013/10/02
Doc4
(Employer)
LEAVE
(attendee)
(organization)
[Title: House Speaker
(cause) (organization)
Member of: Republican
Residence: Ohio]
2013/10/01
Doc2
Lockheed
Martin
3000
Workers
(Employee)
2013/10/03
Doc5
4
What is an event?
– An Event is a specific occurrence involving participants.
– An Event is something that happens.
– An Event can frequently be described as a change of state.
Most of current NLP work
focused on this
Chart from (Dölling, 2011)
5
What is an event?
Topic Detection and Tracking (TDT)
Automatic Content Extraction
(ACE) & Entities, Relations
and Events (ERE)
Abstract Meaning
Representation (AMR)
Propbank,
Nombank
Timebank,
Discourse
Treebank
Temporal order of coarse-grained
groups of events (“topics”)
Defined 33 types of events,
each event mention
includes a trigger word and
arguments with roles
Rich semantic roles on concepts,
no typing on predicates
Each predicate is an event type, no arguments
Some nominals and adjectives allowed
Chart adapted from (Strassel, 2012)
6
Limitations of Previous Event Representations
• Coarse-grained predicate-argument representations defined in Propbank (Palmer
et al., 2005, Xue and Palmer, 2009) and FrameNet (Baker et al., 1998)
– No attempt to cluster the predicates
– Assign coarse-grained roles to arguments
• Fine-grained event trigger and argument representations defined in ACE and ERE
– Only covers 33 subtypes; e.g., Some typical types of emergent events such as
“donation” and “evacuation” in response to a natural disaster are missing
• Both representations are missing:
– Event hierarchies, Relations among events
– Measurement to quantify uncertainty, Implicit argument roles
– The narrator’s role, profile and behavior intent and temporal effects
– Hard to construct a timeline of events across documents
– Efforts are being made by the DEFT event working group (NAACL13/ACL14
Event Workshops and this DEFT PI meeting Event Workshop)
7
Limitations of Previous Event Representations
• Coarse-grained predicate-argument representations defined in Propbank (Palmer
et al., 2005, Xue and Palmer, 2009) and FrameNet (Baker et al., 1998)
– No attempt to cluster the predicates
– Assign coarse-grained roles to arguments
• Fine-grained event trigger and argument representations defined in ACE and ERE
– Only covers 33 subtypes; e.g., Some typical types of emergent events such as
“donation” and “evacuation” in response to a natural disaster are missing
• Both representations are missing:
– Event hierarchies, Relations among events
– Measurement to quantify uncertainty, Implicit argument roles
– The narrator’s role, profile and behavior intent and temporal effects
– Hard to construct a timeline of events across documents
– Efforts are being made by the DEFT event working group (NAACL13/ACL14
Event Workshops and this DEFT PI meeting Event Workshop)
8
A New Representation for Events
Current NLP’s
Focus
9
Typical Event Mention Extraction Features

Trigger Labeling

Lexical










Entity


the entity type of the syntactically nearest
entity to the trigger in the parse tree
the entity type of the physically nearest
entity to the trigger in the sentence

Entity type and subtype
Head word of the entity mention
Context


Trigger tokens
Event type and subtype
Entity

Trigger list, synonym gazetteers
the depth of the trigger in the parse tree
the path from the node of the trigger to
the root in the parse tree
the phrase structure expanded by the
parent node of the trigger
the phrase type of the trigger
Event type and trigger

Syntactic


Argument Labeling
Tokens and POS tags of candidate
trigger and context words
Dictionaries


Context words of the argument
candidate
Syntactic




the phrase structure expanding the
parent of the trigger
the relative position of the entity
regarding to the trigger (before or after)
the minimal path from the entity to the
trigger
the shortest length from the entity to
the trigger in the parse tree
(Chen and Ji, 2009)
Event Trigger Identification Missing Errors
• OOV
• this morning in Michigan, a second straight night and into this morning, hundreds
of people have been rioting [attack] in Benton harbor.
• Scene understanding
• This was the Italian ship that was taken -- that was captured [transfer-ownership]
by Palestinian terrorists back in 1985 and some may remember the story of Leon
clinghover, he was in a cheal chair and the terrorists shot him and pushed him
over the side of the ship into the Mediterranean where he obviously, died.
• The gloved one claims the label has been releasing new albums and Jackson five
merchandise without giving [transfer-money] him "a single dollar."
• Airlines are getting [transport] flyers to destinations ontime more often.
• Three Young Boys,ages 2, 5 and 10 survived and are in critical [injure] condition
after spending in 18 hours in the cold.
11
Event Trigger Identification Spurious Errors
• Lara hijacking in 1995 has been captured [arrest-jail] by U.S. forces in or near
Baghdad.
• I want to take [transport] this opportunity to stand behind the Mimi and proclaim
my solidarity.
• But when we moved [transport] in closer and started to scan the area, then I saw
the 2-year-old waving from next to the airplane.
• He's left [transport] a lot on the table.
• Barbara, Bill Bennet's Glam gambling loss [die] changed my opinion.
• Stewart has found the road to fortune wherever she has traveled [transport].
• And it's hard to win back that sort of brand equity that she's lost [end-position].
• Police measure the spot where ten months old Miana Williams landed after she
was thrown [transport] from a seventh floor window at this apartment building.
• The baby fell [die] 80 feet.
• These tree branches cushioned her fall [die] and saved her life.
12
Event Trigger Identification Spurious Errors
(Con’t)
• He -- After landing and taking a brief tour of part of the ship, he also went
[transport] out on to the flight deck and observed some of the flight operations
of some of the f-18 aircraft that are now leaving these aircraft…
• still hurts [attack] me to read this.
– Metaphor features: semantic categories (noun.animal, noun.cognition) or
degree of abstractness
• Any farm kid knows that certain animals cannot see for a few days after they're
born [born], including puppies.
• We happen to be at a very nice spot by the beach where this is a chance for
people to get [transport] away from CNN coverage, everything, and kind of relax .
• He bought the machinery, moved [transport] to a new factory, rehired some of
the old workers and started heritage programs.
• Under the reported plans, Blackstone Group would buy Vivendi's theme park
division, including Universal Studios Hollywood, Universal Orlando in Florida and
Universal's ownership interests in parks in Spain and Japan, a source close to the
negotiations [meet] told the paper.
13
Event Trigger Classification Errors
• a man in New York is facing attempted murder charges after
allegedly throwing [transportattack] his baby seven stories to the
ground below.
• He's also being fined [transfer-money  fine] $3 million and was
ordered to pay $1.2 million in restitution to the New York State Tax
Commission.
• ``They admitted to having nuclear capability and weapons at this
moment,'' said Rep. Curt Weldon, who headed a delegation of U.S.
lawmakers that visited [meettransport] Pyongyang for three days
ending Sunday.
• Blasphemy is punishable by death [die  execute] under the
Pakistan Penal Code.
• We discussed [phone-write  meet] the Middle East peace
process.
14
Event Argument Identification Missing Errors
• Scene understanding:
• This was the Italian ship that was taken -- that was captured by Palestinian
terrorists back in 1985 [die_time] and some may remember the story of Leon
clinghover, he was in a cheal chair and the terrorists shot him and pushed him
over the side of the ship into the Mediterranean [die_place] where he
[die_victim] obviously, died.
• She [die_victim] was going to fight to the death.
• Then police say the baby's mother pulled out a kitchen knife [die_instrument]
opinion on the 911 tape you can hear Williams tape say "go ahead kill me."
• Coreference:
• another story out of Belgrade [attack_place], violence at the highest form.
• Unusual syntactic structure:
• Two f-14 Tomcats struck the targets, the same area was a site [attack_place] of
heavy bombing yesterday.
• The EU foreign ministers met hours after U.S. President George W. Bush gave
Saddam [attack_target] 48 hours to leave Iraq or face invasion.
15
Event Argument Identification Missing Errors
(Cont’)
• OOV:
• We've seen in the past in Bosnia for example, you held elections and all of the old
ethnic thugs [election_person] get into power because they have organization
and they have money and they stop the process of genuine building of
democracy.
• There are charges, U.S. charges which have expired but could, I am told, possibly
be re -- restarted for piracy, hostage taking and conspiracy [charge_crime].
• Stewart and former imclone CEO Sam Waksal shared a broker who now also faces
charges of obstruction of justice and perjury [charge_crime].
• I was born four years too late to have slept with JFK [born_time]?
• He called the case and I'm quoting now, a judge's worst nightmare, but he noted
that Maryland Parole Boards and the facility where Michael Serious was held, the
institution made what the judge called the final decision on whether to release
serious [release-parole_person].
• Last week Williamson, a mother of four, was found stabbed to death at a
condominium [die_place] in Greenbelt, Maryland.
• A source tell US Enron is considering suing its own investment bankers for giving
16
Event Argument Identification Spurious Errors
• He's been in Libya and he's been living under the protection of Saddam Hussein
in Baghdad [die_place], but he is wanted for murder in Italy Italy.
• The EU foreign ministers met hours after U.S. President George W. Bush
[meet_entity] gave Saddam 48 hours to leave Iraq or face invasion.
17
Event Argument Role Classification Errors
• His private plane [vehicle  artifact] arrived at Heathrow Airport.
• a cruise ship [instrument  target] is being searched off the Hawaii coast after
two notes threatening terrorist attack were found on board.
18
Event-Event Relation Taxonomy
• Preliminary version, under refinement with guidance from Ed Hovy
• Focus: subevent, confirmation, causality and temporality
Inheritance
Expansion
Reemergence
Subevent
Variation
Reference/Target
Ancestor/Descendant
Semina/Variance
Conjunction
Disjunction
Confirmation
Competition
Negation
Opposite
Concession
Generation/Neighborship
Generation/Neighborship
Generation/Instantiation
Contingency
Causality
Conditionality
Superior/Inferior/parallelism
Initiator/Negator
N/A
Prerequisite/Opposite to Expectation
Cause/Result
Condition/Emergence
Temporality
Asynchronism
Synchronism
Subtype
Priority/Posteriority
N/A
Role
Comparison
Main Type
19
Event-Event Relation: Inheritance
Definition: if X clones or succeeds Y, or serves as a revolutionary change of Y
Inheritance
Reemergence
Subevent
Variation
Reference/Target
Ancestor/Descendant
Semina/Variance
Eruption of Mount
Vesuvius in 1944.
Inheritance.
Reemergence
Inheritance.
Variation
Inheritance.Subevent
Eruption of Mount
Vesuvius in 1906.
Mount Vesuvius keeps
silent recently.
The eruption covered
most of north area
with ash.
Role=Reference
Role=Descendant
Role=Variance
20
Event-Event Relation: Expansion
Definition: if X and Y jointly constitute a larger event or share the same
subevent, or X serves as the fine-grain instance or evidence of Y,
Expansion
Conjunction
Disjunction
Confirmation
Generation/Neighborship
Generation/Neighborship
Generation/Instantiation
Expansion.Disjunction
X and Y share the same subevent
Try the
suspect
Arrest the
suspect
Boston Marathon Bombing.
Dzhokhar was
suspected
Appear on
the scene.
Expansion.Confirmation
Second Explosion.
First Explosion.
Expansion.Conjunction
X and Y jointly constitute a main event
X serves as the fine-grain
instance or evidence of Y
Made bombs
at home.
21
Event-Event Relation: Contingency
Definition: if X creates a condition for the emergence of Y, or results in Y
Primacy Effect
Contingency
Causality
Conditionality
Cause/Result
Condition/Emergence
Role=Cause
Contingency.Conditionality
X creates a condition for the emergence of Y
A violent storm would hit Barents sea.
Contingency.Causality
X results in the
emergence of Y
The special hoisting equipment
had been properly installed in
the massive pick-up boat.
Role=Condition
The operation to raise the Kursk
submarine was scheduled to begin
in about two weeks.
Role=Emergence
Salvage plan was
postponed indefinitely.
Role=Result
22
Event-Event Relation: Temporality
Original TimeBank**
There were four or five people
inside, and they just started firing
TimeBank-Dense
There were four or five people
inside, and they just started firing
Ms. Sanders was hit several times
and was pronounced dead at the
scene.
Ms. Sanders was hit several times
and was pronounced dead at the
scene.
The other customers fled, and the
police said it did not appear that
anyone else was injured.
The other customers fled, and the
police said it did not appear that
anyone else was injured.
New relation
From TimeBank to TimeBank-Dense
• Relations: BEFORE, AFTER, INCLUDES, INCLUDED_IN, SIMULTANEOUS, VAGUE
• Annotators must label all pairs within one sentence window
• Annotation guidelines address resulting difficult cases
• Evaluate and develop systems to decide whether a pair can be related
(Cassidy et al., 2014); ** (Pustejovsky et al., 2003)
23
An Example: Input
[Oct.25 News]
WNYC: The Office of Emergency Management situation room
is now open, and Mayor Michael Bloomberg said they are
monitoring the progress of the storm. "We saw that
hurricanes like Irene can really do damage and we have to
take them seriously, but we don't expect based on current
forecasts to have anything like that," Bloomberg said. "But
we're going to make sure we're prepared." He advised
residents in flood-prone areas be prepared to evacuate.
[Oct.29 Tweets]
OMG!!! NYC IS GETTING THE EYE OF HURRICANE
SANDY NEXT WEEK....SCARY!!! I HAVE TO EVACUATE
MY NEIGHBORHOOD!
24
Representing Event-Event Relations in KB
ID = E1 Name = Office of Emergency Management
ID = E2 Name = Michael Bloomberg
Title = Mayor
Employer = E5
Entity ID = E3 Name = Residents in flood-phone areas
KB
ID = E4 Name = WNYC Property = News
ID = E5 Name = New York City
ID = E6 Name =
Tammi
Delancy
#Follower s= #Friends = Role =
211
770
Participant
&Commenter
City_of_reside
nce = E5
25
Representing Event-Event Relations in KB
ID =
EV1
Event
KB
ID =
EV2
Properties
Type=
Warning
Novelty=
1.0
Modality =
Asserted
Tense =
Past
Polarity =
Positive
Genericity =
Specific
Trigger
advised
Arguments
Advisor =
E1, E2
Advisees =
E3
Advice =
Evacuation
Narrator = E4
Time = 10/25/2012
Place = E5
Result Event
EV2
Children Event
EV
Properties
Type=
Evacuation
Novelty=
1.0
Modality =
Asserted
Tense=
Future
Polarity =
Positive
Genericity =
Specific
Time = 10/29/2012
Place = E5
Trigger
evacuate
Arguments
Narrator = E4
Cause Event
EV1
Parent Event
EV4
26
Pilot System
• We have a reasonable understanding on how to detect temporal
relations (Ji et al., 2013; Cassidy and Ji, 2014) but not others
• Main Challenges
– Clustering coreferential events - what is event coreference anyway (Hovy
et al., 2013)?
– Event-event relations are often logic relations without explicit linguistic
indicators
– Fine-grained distinction / Implicit logic relations (Hong et al., 2012)
• Deep background and commonsense knowledge acquisition
– FrameNet/Google ngram / Concept Net /Background news
– Retrospective Historical Event Detection  precursor prediction (Hong and
Ji, 2014submission)
– Take advantage of rich knowledge representation such as AMR (Banarescu
et al., 2013)
27