Transcript [Poster]

Classifying Entities into an Incomplete Ontology
Bhavana Dalvi , William W. Cohen , Jamie Callan
School Of Computer Science, Carnegie Mellon University
Experimental Results
Motivation
 Existing Techniques
Ontology 1
 Semi-supervised Hierarchical Classification: Carlson WSDM’10
 Extending knowledge bases: Finding new relations or attributes of existing concepts
Mohamed et al. EMNLP’11, Reisinger et al. ACL’09
 Unsupervised Ontology Discovery:
Adams et al. NIPS’10, Blei et al. JACM’10, Reisinger et al. ACL’09
Ontology 2
 Evolving Web-scale datasets
 Billions of entities and hundreds of thousands of concepts
 Difficult to create a complete ontology
 Hierarchical classification of entities into incomplete ontologies is needed
 Our technique is in between Semi-supervised Hierarchical Classification and
Unsupervised Ontology Discovery
 Class hierarchy is a tree.
Assumptions   Classes at any one level are mutually exclusive.
Hierarchical DAC Exploratory EM
 Initialize the model with a few seeds per class
Entity
 Iterate till convergence (Data likelihood and number of classes)
 E Step: Predict labels for unlabeled points
For i = 1 : n
Features
Example of features extracted from Clueweb09
Pittsburgh
Lives in _ : 200 , City of _ : 156, was born in _ : 250
Washington DC City of _ : 230, was born in _ : 150, _ is capital of : 50
Spinach
_ is a green vegetable : 150, _ contains iron : 100
For l = 1 : NumLevels
𝐶𝑝𝑎𝑟𝑒𝑛𝑡 = 𝑙𝑎𝑏𝑒𝑙(𝑋𝑖 , 𝑙 − 1)
Example of DAC
Exploratory EM
Coke
1.0
Root
𝐶𝑐𝑎𝑛𝑑𝑖𝑑𝑎𝑡𝑒 = 𝐶ℎ𝑖𝑙𝑑𝑟𝑒𝑛 𝐶𝑝𝑎𝑟𝑒𝑛𝑡
If 𝑷 𝑪𝒄𝒂𝒏𝒅𝒊𝒅𝒂𝒕𝒆
𝑿𝒊 ) is nearly-uniform
Location
Create a new class 𝑪𝒏𝒆𝒘 ; assign Xi to it
0.1
Adds to class
constraints
Food
0.9
𝑷𝒂𝒓𝒆𝒏𝒕 𝑪𝒏𝒆𝒘 = 𝑪𝒑𝒂𝒓𝒆𝒏𝒕 (Update class constraints)
else
Assign 𝑋𝑖 to 𝐴𝑟𝑔𝑚𝑎𝑥 𝐶𝑗 𝑃 𝐶𝑐𝑎𝑛𝑑𝑖𝑑𝑎𝑡𝑒
𝑋𝑖 )
0.55
end
C8
Condiment
Vegetable
Country
State
0.45
A
Dataset
end
#Classes #Levels
#NELL
Entities
 M step: Re-compute model parameters using predictions in E step.
Number of classes might increase in each iteration.
 Check if model selection criterion is satisfied
If not, revert to model in Iteration `t-1’
Methodology Details
 Class Creation Criterion:
1
1
given 𝑃 𝐶𝑗 𝑋𝑖 ) , 𝑗 = 1 … 𝑘 𝑎𝑛𝑑 𝑃𝑢𝑛𝑖𝑓𝑜𝑟𝑚 = [ … ]
𝑘
DS-1
DS-2
Jensen-Shannon divergence: JS-Div(𝑃 𝐶𝑗 𝑋𝑖 ), 𝑃𝑢𝑛𝑖𝑓𝑜𝑟𝑚 ) <
 Objective function:
Maximize { Log Data Likelihood – Model Penalty }
m: #clusters,
Params{𝐶1 … 𝐶𝑚 }
subject to
Class constraints: 𝒁𝒎
 Model Selection: Extended Akaike Information Criterion
AICc(g) = -2*L(g) + 2*v + 2 * v * (v+1) / (n – v -1)
Here g: model being evaluated, L(g): log-likelihood of data given g,
v: number of free parameters of the model, n: number of data-points.
 Class Constraints:
 Inclusion: If an entity is of type ``Vegetable’’ then it is also of type ``Food’’.
 Mutual Exclusion: If an entity is of type ``Mammal’’ then it is not of type
``Reptile’’ and vise a versa.
#Entity,
#NELL
Context
Entity Labels
Pairs
3.4M
8.8M
6.7K
6.7M
25.8M
42.2K
2.5K
12.9K
Macro-averaged Seed Class F1
FLAT
Semisup
EM
MinMax ratio: 𝑚𝑎𝑥(𝑃 𝐶𝑗 𝑋𝑖 )) / 𝑚𝑖𝑛(𝑃 𝐶𝑗 𝑋𝑖 )) < 2
1
𝑘
3
4
Dataset #Train Level #Seed/
/Test
#Ideal
Points
Classes
DS-1
𝑘
11
39
#Contexts
DS-2
335/
2.2K
DAC
ExploreEM Semisup
EM
ExploreEM
2
2/3
43.2
78.7 *
69.5
77.2 *
3
4/7
34.4
42.6 *
31.3
44.4 *
3.9/4
64.3
53.4 *
65.4
68.9 *
9.4/24
31.3
33.7 *
34.9
41.7 *
2.4/10
27.5
38.9 *
43.2
42.4 *
1.5K/1 2
1.4K
3
4
Conclusions
 Hierarchical Exploratory EM works with incomplete class hierarchy and few
seed instances to extend the existing knowledge base in terms of adding new
instances of known classes and discovering new classes and adding them at
right places in the hierarchy.
 Hierarchical classification methods performed better than Flat classification.
 Exploratory learning methods performed better than or comparable to their
Semi-supervised counterparts.
 Future work:
Incorporate arbitrary class constraints
Evaluate the newly added clusters
Acknowledgements : This work is supported by Google and the Intelligence Advanced Research Projects Activity (IARPA) via Air Force Research Laboratory (AFRL)
contract number FA8650-10-C-7058.