Transcript Exploiting Timelines to Enhance Multi

Exploiting Timelines to
Enhance Multi-document
Summarization
Jun-Ping Ng, Yan Chen, Min-Yen Kan and
Zhoujun Li
National University of Singapore
Beihang University
24 Jun 2014
ACL 2014 - Timelines in Summarization
2
Cyclone
Sidr
2007, JTWC designation: 06B
“A fierce cyclone
packing extreme
winds and torrential
rain smashed into
Bangladesh’s
southwestern coast
Thursday, …”
Image Courtesy: Univ. WisconsinMadison
24 Jun 2014
ACL 2014 - Timelines in Summarization
3
“… wiping out
homes and trees in
what officials
described as the
worst storm in
years.”
Image Courtesy: US Navy /
Wikipedia
24 Jun 2014
ACL 2014 - Timelines in Summarization
4
“More than 100,000
coastal villagers
have been
evacuated before
the cyclone made
landfall.”
Image Courtesy: US State Department /
Wikipedia
24 Jun 2014
ACL 2014 - Timelines in Summarization
1991 Bangladesh
Cyclone
Image Courtesy: US Navy /
Wikipedia
“The storm
matched one in
1991 that
sparked a tidal
wave that killed
an estimated
138,000 people,
Karmakar told
AFP.”
5
24 Jun 2014
ACL 2014 - Timelines in Summarization
[1] “A fierce cyclone
packing extreme
winds and torrential
rain smashed into
Bangladesh’s
southwestern coast
Thursday, wiping
out homes and
trees in what
officials described
as the worst storm
in years.”
6
[2] “More than 100,000
coastal villagers have
been evacuated before the
cyclone made landfall.”
[3] “The storm matched
one in 1991 that
sparked a tidal wave
that killed an estimated
138,000 people,
Karmakar told AFP.”
24 Jun 2014
ACL 2014 - Timelines in Summarization
[1] “A fierce cyclone
packing extreme
winds and torrential
rain smashed into
Bangladesh’s
southwestern coast
Thursday, wiping
out homes and
trees in what
officials described
as the worst storm
in years.”
7
[2] “More than 100,000
coastal villagers have
been evacuated before the
cyclone made landfall.”
[3] “The storm matched
one in 1991 that
sparked a tidal wave
that killed an estimated
138,000 people,
Karmakar told AFP.”
24 Jun 2014
ACL 2014 - Timelines in Summarization
Timelines from Text
[3] “The storm matched one in 1991
that sparked a tidal wave that killed
an estimated 138,000 people,
Karmakar told AFP.”
[1] “A fierce cyclone packing extreme
winds and torrential rain smashed into
Bangladesh’s southwestern coast
Thursday, wiping out homes and trees
in what officials described as the worst
storm in years.”
[2] “More than 100,000 coastal villagers
have been evacuated before the
cyclone made landfall.”
8
24 Jun 2014
ACL 2014 - Timelines in Summarization
Key time spans are summary
worthy
[3] “The storm matched one in 1991
that sparked a tidal wave that killed
an estimated 138,000 people,
Karmakar told AFP.”
[1] “A fierce cyclone packing extreme
winds and torrential rain smashed into
Bangladesh’s southwestern coast
Thursday, wiping out homes and trees
in what officials described as the worst
storm in years.”
[2] “More than 100,000 coastal villagers
have been evacuated before the
cyclone made landfall.”
9
24 Jun 2014
ACL 2014 - Timelines in Summarization
10
Timelines + Summarization
Lexical and
positional
features
events
Timelinederived
features
Timelines
(per input document)
Summarization
System
Summary
24 Jun 2014
ACL 2014 - Timelines in Summarization
Outline
• Goal and Motivation
• Timeline Generation
• Integrating Timelines
– In Scoring: (Contextual) Importance, Density
– In Re-ordering: TimeMMR
• Experiments
• Discussion
11
24 Jun 2014
ACL 2014 - Timelines in Summarization
12
Timeline Generation
Temporal Processing
Input
Document
Event and Timex
Extraction
E-E
Temporal
Classification
E-T
Temporal
Classification
Timeline
Generation
Timex
Normalization
Timeline
24 Jun 2014
ACL 2014 - Timelines in Summarization
1. Event-Event Temporal
Classification
(Ng et al., 2013; EMNLP)
13
24 Jun 2014
ACL 2014 - Timelines in Summarization
2. Event-Timex Temporal
Classification
(Ng and Kan, 2012; COLING)
14
24 Jun 2014
ACL 2014 - Timelines in Summarization
3. Timex Normalization
(HeidelTime; Strötgen and Gertz, 2013)
“Today”  June 6, 2014
15
24 Jun 2014
ACL 2014 - Timelines in Summarization
Timeline Construction
1. Map normalized timexes to timeline
2. Place events which OVERLAP with timexes onto
timeline
3. Place events which OVERLAP with other events onto
the timeline
4. Insert rest of events based on BEFORE/AFTER
ordering
1999
16
24 Jun 2014
ACL 2014 - Timelines in Summarization
17
Integrating Timelines into
SWING
Timelines
Timelines
Temporal Processing
SWING
Summarization Pipeline
(Ng et al., COLING 2012,
TAC 2011)
State-of-the-art open-source
extractive
summarizerhttps://github.c
Pre-processing
Sentence
Scoring
Sentence
Re-ordering
om/WING-NUS/SWING
Basic, k of n
sentence
summaries
Time Span Importance
Contextual
Time Span Importance
Sentence
Temporal Coverage
Density
Time MMR
Summary
24 Jun 2014
ACL 2014 - Timelines in Summarization
18
1. Time Span Importance (TSI)
right peak of e
biggest cluster
left peak of e
e
Time Span A
Time Span A+4
• Time spans which contain
many events are more
salient
• Sentences which
references events in these
time spans are thus better
candidates for a summary
24 Jun 2014
ACL 2014 - Timelines in Summarization
19
2. Contextual Time Span Importance (CTSI)
right peak of e
biggest cluster
left peak of e
e
Time Span A
Time Span A+4
• Time spans near to
important time spans are
important
• Search left and right for
local peaks
, where
24 Jun 2014
ACL 2014 - Timelines in Summarization
20
3. Sentence Temporal Coverage Density
(TCD)
left peak of e
right peak of e
biggest cluster
• Favour sentences which
– contain more events
– covering a wide variety
of time spans
e
Time Span A
Time Span A+4
24 Jun 2014
ACL 2014 - Timelines in Summarization
Identifying Redundancies
• SWING makes use of the Maximal Marginal
Relevance (MMR) algorithm to identify redundancies
in selected sentences
• MMR is based largely on surface lexical similarities
Idea: Let’s use time as a basis to penalize the selection
of sentences from redundant time periods.
21
24 Jun 2014
ACL 2014 - Timelines in Summarization
22
TimeMMR
• Beyond lexical similarities, identify sentences which
contain substantial time span overlap.
• Candidate sentences which share many time spans
with selected sentences are penalized.
Lexically dissimilar but redundant
Proportion of
overlap
(1)
(2)
(3)
An official in Barisal, 120 kilometres south of Dhaka, spoke of severe
destruction as the 500 kilometre-wide mass of cloud passed overhead.
“Many trees have been uprooted and houses and schools blown away,”
Mostofa Kamal, a district relief and rehabilitation officer, told AFP by
telephone.
“Mud huts have been damaged and the roofs of several houses blown off,”
said the state’s relief minister, Mortaza Hossain.
24 Jun 2014
ACL 2014 - Timelines in Summarization
Experiments
• Data
– TAC 2010 dataset for training
– TAC 2011 dataset for testing
• Temporal Processing Systems
– HeidelTime (Strötgen and Gertz, 2013)
– E-T temporal classification (Ng and Kan, 2012)
– E-E temporal classification (Ng et al., 2013)
• Summarization baseline
– SWING (Ng et al., 2012)
23
24 Jun 2014
ACL 2014 - Timelines in Summarization
24
Results
* = p < 0.1, ** = p < 0.05, against R
row
#
Configuration
R-2
R
SWING
0.1339
B1
CLASSY
0.1278
1
SWING + Timeline Features
0.1394*
2
SWING + Timeline Features + TimeMMR
0.1389
Doesn’t seem
very effective!
24 Jun 2014
ACL 2014 - Timelines in Summarization
Analysis: Timelines contain
errors
• Errors from underlying temporal processing systems
• Simplifying assumptions made in timeline construction
• Lack of consistency checking and validation
For effective use, we must identify good timelines
• Identify timelines which potentially contain more errors
• Exclude these when performing summarization
25
24 Jun 2014
ACL 2014 - Timelines in Summarization
26
Reliability Filtering
• Short timelines can result when the system fails to
extract or relate events and timexes
• Features derived from short timelines are prone to
have extreme values
• Use the length of a timeline as a gauge of its accuracy
• Don’t use timelines shorter than average
(as computed over the whole collection)
24 Jun 2014
ACL 2014 - Timelines in Summarization
27
With Reliability Filtering
* = p < 0.1, ** = p < 0.05, against R
row
#
Configuration
R-2
R
SWING
0.1339
B1
CLASSY
0.1278
1
SWING + Timeline Features
0.1394*
2
SWING + Timeline Features + TimeMMR
0.1389
3
SWING + Timeline Features [Filtered]
0.1418**
4
SWING + Timeline Features + TimeMMR [Filtered]
0.1402**
TimeMMR
doesn’t seem
effective! Why?
24 Jun 2014
ACL 2014 - Timelines in Summarization
28
Does TimeMMR actually help?
L1
An Iraqi reporter threw his shoes at visiting U.S. President George W. Bush and called him a ”dog” in
Arabic during a news conference with Iraqi Prime Minister Nuri al-Maliki in Baghdad
R1
L2
”All I can report is it is a size 10,.
R2
L3
Muntadhar al-Zaidi, reporter of Baghdadiya
television jumped and threw his two shoes one
by one at the president, who ducked and thus
narrowly missed being struck, raising chaos in the
hall in Baghdad’s heavily fortified green Zone.
The incident occurred as Bush was appearing
with Iraqi Prime Minister Nouri al-Maliki.
L4
The president lowered his head and the first
shoe hit the American and Iraqi flags behind the
two leaders.
Muntadhar al-Zaidi, reporter of Baghdadiya
television jumped and threw his two shoes one
by one at the president, who ducked and thus
narrowly missed being struck, raising chaos in the
hall in Baghdad’s heavily fortified green Zone.
R4
L5
The
The president lowered his head and the
R5
R3
Possibly Redundant?
worse by R-2
R-2: 0.2772, better by R-2
CouldR-2:
an0.2643,
(automated)
evaluation
metric cater for
time?
24 Jun 2014
ACL 2014 - Timelines in Summarization
Conclusion
• Use of automatic timeline generation
• Integration of timelines into summarization
– Sentence scoring via timeline features
– Sentence re-ordering via TimeMMR
– Length based timeline filtering helps to ameliorate
errors
For details on temporal processing, see:
Jun Ping’s work at COLING 2012, EMNLP
2013 and his doctoral thesis (2014)
Questions? If not, ask for more detailed
analysis!
29
24 Jun 2014
ACL 2014 - Timelines in Summarization
ADDITIONAL SLIDES
30
24 Jun 2014
ACL 2014 - Timelines in Summarization
31
Related Work
• For Sentence Reordering
– Barzilay et al., 1999
• Recency as an indicator of salience
– Goldstein et al., 2000;Wan, 2007;
Demartini et al., 2010
– Liu et al., 2009 (“Temporal Graph”)
– Wu, 2008 (“Largest Cluster”)
Close to our TSI
• TREC Temporal Summarization Track
– Not as relevant; about monitoring an event over time
24 Jun 2014
ACL 2014 - Timelines in Summarization
With time features;
better
32
Baseline; worse
24 Jun 2014
ACL 2014 - Timelines in Summarization
33
TSI: A crane accident
With TSI; better
Without TSI;
worse
With TSI, the cause of the accident in this
summary is included; the alternative R1
sentence is background information and
does not occur at any key time span.
24 Jun 2014
ACL 2014 - Timelines in Summarization
34
CTSI: Coral Reef Preservation
Without CTSI;
With CTSI; better
With CTSI, the “warn”
and “disappear”
events were
promoted in
importance due to
their proximity with
peak P
worse
24 Jun 2014
ACL 2014 - Timelines in Summarization
Timeline Caveats
• Some events span a long period of time (i.e., “1999”)
• Events are ordered based on the start of the duration
• Timeline captures relative order
• Construction algorithm does not attempt to reconcile
contradictions
35
24 Jun 2014
ACL 2014 - Timelines in Summarization
Timex Normalization
Source:Bethard, 2013
36
24 Jun 2014
ACL 2014 - Timelines in Summarization
37
References
• Jun-Ping Ng, Interpreting Text with Time, Doctoral
Thesis, National University of Singapore, 2014
• Jun-Ping Ng, Min-Yen Kan, Ziheng Lin, Wei Feng, Bin
Chen, Jian Su, Chew-Lim Tan, Exploiting Discourse
Analysis for Article-Wide Temporal Classification,
EMNLP 2013
• Jun-Ping Ng, Praveen Bysani, Ziheng Lin, Min-Yen Kan,
Chew-Lim Tan, Exploiting Category-Specific
Information for Multi-Document Summarization,
COLING 2012
• Jun-Ping Ng, Min-Yen Kan, Improved Temporal Relation
Classification using Dependency Parses and