An Unsupervised Approach to Biography Production using Wikipedia Fadi Biadsy, Julia Hirschberg, Elena Filatova Columbia University ACL 2008

Download Report

Transcript An Unsupervised Approach to Biography Production using Wikipedia Fadi Biadsy, Julia Hirschberg, Elena Filatova Columbia University ACL 2008

An Unsupervised Approach to Biography Production using Wikipedia Fadi Biadsy, Julia Hirschberg, Elena Filatova

Columbia University

ACL 2008

Motivation



Biographies

identify important information about individuals  Manually producing biographies is labor intensive – usually produced only for famous individuals  Can we produce biography automatically, about a wider variety of people, using the web for example?

Overview



Multi-document summarization (MDS) approach based on extractive techniques to produce biographies



Automatic approach to collect training data from Wikipedia

System Overview Documents with NE tags and resolved coreferences Input Sentence Selection Hypothesis sentences Sentence ordering Non-redundant biographical sentences Biographical sentence classifier Biographical sentences Redundancy Removal Reference Rewriting and Trimming

Biography

System Overview Documents with NE tags and resolved coreferences Input Sentence Selection Sentence ordering Hypothesis sentences Non-redundant biographical sentences Biographical sentence classifier Biographical sentences Redundancy Removal Reference Rewriting and Trimming

Biography

Sentence Selection



Given N documents, find the sentences that contain a reference to the target person



hypothesis sentences

Biography

Biographical-Sentence Classifier

 Train a binary classifier to identify biographical sentences  Manually annotating a large corpus of biographical and non-biographical information (e.g., Zhou et al., 2004) is labor intensive  Our approach: collect biographical and non biographical corpora automatically 8

Training Data – Biographical Corpus from Wikipedia

 Utilize Wikipedia biographies  Extract 16,906 biographies from the xml version of Wikipedia  Apply simple text processing techniques to clean up the text 9

Constructing the Biographical Corpus

Identify the subject of each biography 1.

Run NYU’s ACE system to tag NEs and do coreference resolution (Grishman et al., 2005) 10

Constructing the Biographical Corpus

Replace each

by its tag type and subtype In

September 1951

King

began his doctoral studies In theology at

Boston University.

[TIMEX]

, [

PER_ Individual]

began [TARGET_HIS] doctoral studies 11 In theology at

[ORG_Educational]

Constructing the Biographical Corpus

Replace each NE by its tag type and subtype 4.

Non-pronominal referring expression that is coreferential with the target person is replaced by

[TARGET_PER]

In September 1951,

King

began his doctoral studies In theology at Boston University.

[TIMEX]

[TARGET_PER]

began

[TARGET_HIS]

doctoral studies 12 In theology at

[ORG_Educational]

Constructing the Biographical Corpus

Replace each NE by its tag type and subtype 4.

Non-pronominal referring expression that is coreferential with the target person is replaced by [TARGET_PER] 5.

Every pronoun

that refers to the target person is replaced by

[TARGET_P]

, where P is the pronoun replaced In September 1951, King began his doctoral studies In theology at Boston University.

[TIMEX]

[TARGET_PER]

began

[TARGET_HIS]

doctoral studies 13 In theology at

[ORG_Educational]

Constructing the Biographical Corpus

Replace each NE by its tag type and subtype 4.

Non-pronominal referring expressions that are coreferential with the target person are replaced by [TARGET_PER] 5.

Every pronoun

that refers to the target person is replaced by [TARGET_

], where P is the pronoun replaced 6.

Sentences containing no reference to the target person are removed In September 1951, King began his doctoral studies In theology at Boston University.

[TIMEX]

[TARGET_PER]

began

[TARGET_HIS]

doctoral studies 14 In theology at

[ORG_Educational]

Constructing the Non-Biographical Corpus

 English newswire articles in TDT4 used to represent non-biographical sentences 1.

Run NYU’s ACE system on each article 2.

Select a PERSON NE mention at

random

article to represent the target person from all NEs in 3.

Exclude sentences with no reference to this target 4.

Replace referring expressions and NEs as in biography corpus 15

Biographical-Sentence Classifier

 Train a classifier on the biographical and non biographical corpora  Biographical corpus:   30,002 sentences from Wikipedia 2,108 sentences held out for testing  Non-Biographical corpus:   23,424 sentences from TDT4 2,108 sentences held out for testing 16

Biographical-Sentence Classifier

 Features :  Frequency of 1-2-3 grams of class-based/lexical , e.g.:    [TARGET_PER] was born [TARGET_HER] husband was [TARGET_PER] said  Frequency of 1-2 grams of POS  Chi-square for feature selection 17

Classification Results

 Experimented with three types of classifiers:

Classifier

SVM M.

Naïve Bayes (MNB) C4.5

Accuracy

87.6% 84.1% 81.8%

F-Meassure

0.87

0.84

0.82

 Note: Classifiers provide a confidence score for each classified sample 18

System Overview Documents with NE tags and resolved coreferences Input Sentence Selection Sentence ordering Hypothesis sentences Biographical sentence classifier Biographical sentences Non-redundant biographical sentences Redundancy Removal Reference Rewriting and Trimming

Biography

Redundancy Removal

  Some of the sentences we select may contain the same information How can we minimize redundancy among the selected sentences?

Cluster biographical sentences using

single-link nearest neighbor clustering technique based on stem-overlap

(Blair-Goldensohn et al., 2004) 2.

Select the sentence from each cluster that maximizes classifier confidence score 20

Biography

Sentence Ordering: Two Methods



Method I:

Use classifier confidence score to order the biographical sentences 

Method II:

 Learn from the presentation order of Wikipedia biographies  Represent sentence positions as integers  SVM regression with RBF kernel using features: class-based/lexical unigrams and bigrams 22

Biography

Reference Rewriting

 News articles typically provide important biographical information of individuals when they are first mentioned  NYU’s ACE system tags full noun phrases including appositives as part of NEs Example:

Brian Jones

, the co-pilot on the Breitling mission, ...

Reference Rewriting

Search for sentence containing longest NE that includes target person’s full name and is coreferential with the target person (

LONG-NE)

Example: Brian Jones’ biography

 The following sentence contains LONG-NE:

Brian Jones, the co-pilot on the Breitling mission

remained a quiet but crucial part of his team's ballooning effort if it had not been for a series of lucky breaks.

, would have 26

Reference Rewriting

Search for sentence containing longest NE that includes target person’s full name and is coreferential with the target person (

LONG-NE)

If this sentence was classified as biographical  boost rank in the biography to first 27

Example: Brian Jones’ biography

 The following sentence contains LONG-NE:

Brian Jones, the co-pilot on the Breitling mission

remained a quiet but crucial part of his team's ballooning effort if it had not been for a series of lucky breaks.

, would have This sentence was

NOT

classified as biographical 28

Reference Rewriting

Search for sentence containing longest NE that includes target person’s full name and is coreferential with the target person (

LONG-NE)

If this sentence was classified as biographical  boost rank in summary to first 3.

Otherwise, replace reference to target person in first sentence of biography by

LONG-NE

Example: Brian Jones’ biography

 The following sentence contains LONG-NE:

Brian Jones, the co-pilot on the Breitling mission

remained a quiet but crucial part of his team's ballooning effort if it had not been for a series of lucky breaks.

, would have 

First Sentence:

Born in Bristol in 1947,

Jones

learned to fly at 16, dropping out of school a year later to join the Royal Air Force.

Replace target NE by NE-LONG  Born in Bristol in 1947,

Brian Jones, the co-pilot on the Breitling mission

, learned to fly at 16, dropping out of school a year later to join the Royal Air Force. 30

Evaluation – Task

 Evaluate using DUC (Document Understanding Conferences) 2004, Task 5  Task: Given document cluster and person’s name X  Answer the question:

Who is X?

 Summary should be no longer than 665 bytes 31

Evaluation – Corpus

 50 clusters of TREC documents  each cluster contains 10 documents on average  We have the output summaries of the 22 systems that participated in original competition  NIST had 4 human summaries written for each cluster  used ROUGE to evaluate  Our system evaluated against

top-DUC2004:

the best performing of the 22 systems (according to ROUGE-L metric) 32

Comparing our best system to the DUC2004 systems and humans using ROUGE-L score 0,45 0,4 0,35

ROUGE-L Average_F

0,3 0,25 0,2 B E H G F A D C * .

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 BL 17 18 19 21 22 33

Comparing our systems to top DUC2004

ROUGE-L Average_F

0,35 0,325 0,3 0,275 0,25 0 1 2 3 4 5 6 7 8 9 10 top DUC2004 C4.5 + Order I C4.5 + baseline order C4.5 + Order II SVM + Order I SVM + baseline order SVM + Order II

MNB + Order I

MNB + baseline order MNB + Order II 11 Order II Only 12   Order I: Classifier confidence score order Order II: SVM regression order 34

Automatic Evaluation using ROUGE-L

ROUGE-L Average_F

0,35 0,325 0,3 0,275 0,25 0 1 2 3 4 5 6 7 8 9 10 top DUC2004

C4.5 + Order I

C4.5 + baseline order

C4.5 + Order II

SVM + Order I SVM + baseline order SVM + Order II MNB + Order I MNB + baseline order MNB + Order II 11 Order II Only 12   Order I: Classifier confidence score order Order II: SVM regression order 35

Automatic Evaluation using ROUGE-L

ROUGE-L Average_F

0,35 0,325 0,3 0,275 0,25 0 1 2 3 4 5 6 7 8 9 10 top DUC2004 C4.5 + Order I C4.5 + baseline order C4.5 + Order II

SVM + Order I

SVM + baseline order

SVM + Order II

MNB + Order I MNB + baseline order MNB + Order II 11 Order II Only 12   Order I: Classifier confidence score order Order II: SVM regression order 36

Automatic Evaluation using ROUGE-L

ROUGE-L Average_F

0,35 0,325 0,3 0,275 0,25 0 1 2 3 4 5 6 7 8 9 10 top DUC2004 C4.5 + Order I C4.5 + baseline order C4.5 + Order II SVM + Order I SVM + baseline order SVM + Order II

MNB + Order I

MNB + baseline order

MNB + Order II

11 Order II only 12   Order I: Classifier confidence score order Order II: SVM regression order 37

Automatic Evaluation using ROUGE-L

ROUGE-L Average_F

0,35 0,325 0,3 0,275 0,25 0 1 2 3 4 5 6 7 8 9 10 top DUC2004 C4.5 + Order I C4.5 + baseline order C4.5 + Order II SVM + Order I SVM + baseline order SVM + Order II MNB + Order I MNB + baseline order MNB + Order II 11

Order II only

12   Order I: Classifier confidence score order Order II: SVM regression order 38

Manual Evaluation (I) MNB vs. Top-DUC2004

 Subjects: 3 native American English speakers  They were presented with pairs of summaries 1.

TopDUC-2004 2.

MNB + Order I 

Task:

Decide which summary more responsive in

form

and

content

to the question -- or whether both equally responsive 39

Manual Evaluation (I) MNB vs. Top-DUC2004

 85.3% (128/150) of judgments preferred one summary over the other 

78.1%

of these (100/128) preferred the summaries produced by our system.

 Majority vote: in 42/50 summaries at least two subjects made same choice. Of these,

88.1%

(37/42) preferred our summaries

= 4.4

7)  kappa statistic = 0.441

Manual Evaluation (II) MNB + Order I vs. MNB + Order II

  Subjects: 3 (different) native American English speakers They were presented with pairs of summaries: 1.

MNB + Order I MNB + Order II   

Task:

Which summary has better presentation order?

kappa = 0

362 Majority vote:

61.7%

(29/47) preferred summaries with order II (not sign.) 41

Conclusion

 Described a system for producing biographies from training corpora collected automatically from Wikipedia and TDT4 –

NO manual annotations whatsoever

 Embed ACE markups and coreferences in the feature space to train our models  Sentence ordering for biographies modeled in two ways  Rewriting heuristic used to create the final biography  Our system significantly outperforms all systems that participated in DUC-2004 according to ROUGE-L scores  Sentence order of SVM regression was preferred by our human subjects 42

Future Work

 Sometimes only part of the sentence is biographical so first,

simplify hypothesis sentences

before classifying them 

Cluster the sentences

model before learning the SVM regression  Experiment with more

linguistically-informed features

 Use our approach on

other query-focused summarization tasks

, e.g.

What is X?

where X is organization’s name or historical event. Wikipedia contains organizational profiles as well as lists of events and their descriptions 43

Thank You!

Special thanks to

Kathy McKeown

and the Speech and NLP groups at Columbia for useful discussions Example of an output of our system: Who is Brian Jones?

Born in Bristol in 1947, Brian Jones, the co-pilot on the Breitling mission, learned to fly at 16, dropping out of school a year later to join the Royal Air Force. After earning his commercial balloon flying license, Jones became a ballooning instructor in 1989 and was certified as an examiner for balloon flight licenses by the British Civil Aviation Authority. He helped organize Breitling's most recent around-the-world attempts, in 1997 and 1998. Jones, who is to turn 52 next week, is actually the team's third co-pilot. After 13 years of service, he joined a catering business and, in the 1980s,… 44

[

TARGET_PER]

(

[TIMEX]

–

[TIMEX]

) was one of the main leaders of

[ORG_Commercial]

 1 46

A Baptist minister,

[TARGET_PER]

became a civil rights activist early in

[TARGET_HIS]

career 

2 SVM Regression – select important biographical features

 where,   

is the set of ~17K Wikipedia biographies

n(t) d

is the number of occurrences of

d d

 Examples:

Unigrams

born became [ORG_Educational]

Bigrams

was born [TARGET_PER] died [TARGET_PER] joined [TARGET_his] family 48

Example: Who is Brian Jones?

 Born in Bristol in 1947, Jones learned to fly at 16, dropping out of school a year later to join the Royal Air Force.

(-1.6)

 After earning his commercial balloon flying license, Jones became a ballooning instructor in 1989 and was certified as an examiner for balloon flight licenses by the British Civil Aviation Authority.

(4.6)

 He helped organize Breitling's most recent around-the-world attempts, in 1997 and 1998.

(6.9)

 Jones, who is to turn 52 next week, is actually the team's third co pilot.

(8.2)

 After 13 years of service, he joined a catering business and, in the 1980s, set about becoming a professional balloonist.

(9.3)

Born on June 18, 1942, JOH South African Deputy President Thabo Mbeki grew up in poverty in Mbewuleni village in the historic land of the Xhosa.

He was in the liberation struggle virtually from the age of 10, when he and a cousin sold Coke bottles to raise money to pay their African National Congress membership fees. Thabo Mbeki, who was elected president by the National Assembly on Monday, was inaugurated in an auspicious ceremony with a distinctly African flavor at the Union Buildings in Pretoria. The father, Govan Mbeki, spent much of his life in jail and sent his children to relatives very young so they couldn't come to depend on parents who 50

Example: 3 sentences from a Wikipedia biography



Martin Luther King, Jr., was born on January 15, 1929, in Atlanta, Georgia. He was the son of Reverend Martin Luther King, Sr. and Alberta Williams King. He had an older sister, Willie Christine (September 11, 1927) and a younger brother, Albert Daniel.

Example: class-based/lexical sentences



[TARGET_PER]

, was born on [TIMEX] , in [GPE_Population-Center] .



[TARGET_HE]

was the son of [PER_Individual] and [PER_Individual] .



[TARGET_HE]

had an older sister , [PER_Individual] ( [TIMEX] ) and a younger brother , [PER_Individual] .

Example: Who is Brian Jones?

            After weeks of frustrating delays, Piccard and

Jones

set off from the Swiss Alps on March 1.

Balloonists

Brian Jones

and Bertrand Piccard tore their billowy craft to keep it from dragging the gondola across the sand.

`` Bertrand and I were furiously trying to slash it with our knives, just to get the extra helium out,''

Jones

said Wednesday after a ceremony to donate the torn and tattered balloon to the Smithsonian Institution's National Air and Space Museum.

The prospect that the Breitling's crew, Bertrand Piccard and

Brian Jones

, may succeed where all others have failed seemed more promising with each passing hour.

Jones

, who is to turn 52 next week, is actually the team's third co-pilot.

After 13 years of service,

professional balloonist.

joined a catering business and, in the 1980s, set about becoming a Born in Bristol in 1947,

Jones

Royal Air Force.

learned to fly at 16, dropping out of school a year later to join the Until he moved from his position as back-up pilot to pilot, its flight systems.

was an on-the-ground man, responsible as the team's project manager for the construction of both the balloon's gondola and

Brian Jones

, the co-pilot on the Breitling mission, would have remained a quiet but crucial part of his team's ballooning effort if it had not been for a series of lucky breaks.

After earning 1989 and was certified as an examiner for balloon flight licenses by the British Civil Aviation Authority.

his

commercial balloon flying license,

Jones

became a ballooning instructor in

helped organize Breitling's most recent around-the-world attempts, in 1997 and 1998.

…….

Example: Who is Brian Jones?

           

Jones, who is to turn 52 next week, is actually the team's third co-pilot.

After weeks of frustrating delays, Piccard and Jones set off from the Swiss Alps on March 1.

After 13 years of service, he joined a catering business and, in the 1980s, set about becoming a professional balloonist.

`` Bertrand and I were furiously trying to slash it with our knives, just to get the extra helium out,'' Jones said Wednesday after a ceremony to donate the torn and tattered balloon to the Smithsonian Institution's National Air and Space Museum.

Born in Bristol in 1947, Jones learned to fly at 16, dropping out of school a year later to join the Royal Air Force.

Brian Jones, the co-pilot on the Breitling mission, would have remained a quiet but crucial part of his team's ballooning effort if it had not been for a series of lucky breaks.

Until he moved from his position as back-up pilot to pilot, he was an on-the-ground man, responsible as the team's project manager for the construction of both the balloon's gondola and its flight systems.

Balloonists Brian Jones and Bertrand Piccard tore their billowy craft to keep it from dragging the gondola across the sand.

After earning his commercial balloon flying license, Jones became a ballooning instructor in 1989 and was certified as an examiner for balloon flight licenses by the British Civil Aviation Authority.

The prospect that the Breitling's crew, Bertrand Piccard and Brian Jones, may succeed where all others have failed seemed more promising with each passing hour.

He helped organize Breitling's most recent around-the-world attempts, in 1997 and 1998.

54 …….

Related Work

    Blair-Goldensohn et al. (2004a) ’s DefScriber, which treats “Who is X?” as a definition question and targets definitional themes (e.g. genus, species) found in the input document collections which include references to the target person. Fadi: Therfore biographical fact are not considered (such as birth and death) Duboue et al. (2003) also address the problem of learning content selection rules for biographies. These rules are learned from two corpora, the first is a semi-structured corpus with lists of biographical facts about show business celebrities, and the other corpus contains free-text biographies about the same celebrities. Weischedel et al. (2004) focuses on modeling kernel-fact features typical for biographies using linguistic and semantic processing. Linguistic features are derived from predicate argument structures deduced from parse trees, and semantic features are the set of biography-related relations and events defined in the ACE guidelines. Sentences containing these kernel facts are ranked according to the probability of their relevance to biographies, using probabilities estimated from a corpus of manually created biographies, including Wikipedia, to estimate the conditional distribution of relevant material given a kernel fact and a background corpus to learn the feature distribution.

Barzilay and Lee (2004) used a novel adaptation of algorithms for Hidden Markov Models to learn how to structure summaries in a specific domain by analyzing topic shifts within a corpus in that domain.

An Unsupervised Approach to Biography Production using Wikipedia Fadi Biadsy, Julia Hirschberg, Elena Filatova Columbia University ACL 2008

Transcript An Unsupervised Approach to Biography Production using Wikipedia Fadi Biadsy, Julia Hirschberg, Elena Filatova Columbia University ACL 2008