Transcript Why Can’t We All Get Along? (Structured Data and

Why Can’t We All Get Along?
(Structured Data and Information Retrieval)
Bruce Croft
Computer Science Department
University of Massachusetts Amherst
Overview
•
•
•
•
•
History of structured data in IR
Conceptual similarities and differences
What is the goal?
The Indri System
Examples using IR for structured data
– XML retrieval
– Relevance models
– Entity retrieval
History
• IR systems have had Boolean field restrictions
since 1970s
– metadata: date, type, source, keywords
– content structure: title, body
• Implementing IR systems using a relational
DBMS first done in the 70s
– Crawford and McCleod, 1978-1983
– Efficiency issues with this approach persisted until
90s (e.g. DeFazio et al, SIGIR 95)
– Inquery IR system successfully used object
management system (Brown, SIGIR 95)
History
• Modifying DBMS model to incorporate
probabilities to integrate DB/IR
– e.g. probabilistic relational algebra (Fuhr and
Rolleke, ACM TOIS 1994)
– e.g. probabilistic datalog (Fuhr, SIGIR 95)
• Text retrieval as a SQL function in commercial
DBMSs
– e.g. Oracle, early 90s
History
• Ranked retrieval of “complex” documents
– e.g. office documents with structure and significant
text content (Croft, Krovetz and Turtle, IPM 1990)
– Bayesian inference net model to combine evidence
from different parts of document structure (Croft
and Turtle, EDT 1992)
– e.g. marked-up documents (Croft, Smith, and Turtle,
SIGIR 1992)
• XML retrieval
– INEX (2002)
Similarities and Differences
• Common interest in providing efficient access
to information on a very large scale
– indexing and optimization key topics
• Until recently, concern about effectiveness
(accuracy) of access was domain of IR
• Focus on structured vs. unstructured data is
historically true but less relevant today
• Statistical inference and ranking are central to
IR, becoming more important in DB
Similarities and Differences
• IR systems have focused on providing access to
information rather than answers
– e.g. Web search
– evaluation typically based on topical relevance and
user relevance rather than correctness (except QA)
• IR works with multiple databases but not
multiple relations
• IR query languages more like calculus than
algebra
• Integrity, security, concurrency are central for
DB, less so in IR
What is the Goal?
• One unified information system?
– i.e. a single conceptual and formal framework to
support the entire range of information needs
– at least a grand challenge
– or is it the Web?
• An integrated DB/IR system?
– i.e. extend database model to fully support
statistical inference and ranking
– a major challenge given established systems and
models
What is the Goal?
• An IR system with extended capability for
structured data
– i.e. extend IR model to include combination of
evidence from structured and unstructured
components of complex objects (documents)
– backend database system used to store objects (cf.
“one hand clapping”)
– many applications look like this (e.g. desktop
search, web shopping)
– users seem to prefer this approach (simple queries
or forms and ranking)
What is the Goal?
• What about important database functionality?
– Source data can be stored in databases
– Extended IR system will construct separate indexes
• What about optimization?
– Search engines worry about optimization!
– Can incorporate ideas from DB optimization
• What about updates?
– Search engines worry about updates!
– Backend database system still available
• What about joins?
– Interesting. Treat IR objects as a view?
Indri – A Candidate IR System
• Indri is a separate, downloadable component of the
Lemur Toolkit
• Influences
– INQUERY [Callan, et. al. ’92]
• Inference network framework
• Query language
– Lemur [http://www.lemurproject.org]
• Language modeling (LM) toolkit
– Lucene [http://jakarta.apache.org/lucene/docs/index.html]
• Popular off the shelf Java-based IR system
• Based on heuristic retrieval models
• Designed for new retrieval environments
– i.e. GALE, CALO, AQUAINT, Web retrieval, and XML retrieval
Zoology 101
• The indri is the largest
type of lemur
• When first spotted the
natives yelled “Indri!
Indri!”
• Malagasy for "Look! Over
there!"
Design Goals
• Off the shelf (Windows, *NIX, Mac platforms)
– Simple to set up and use
– Fully functional API w/ language wrappers for Java, etc…
• Robust retrieval model
– Inference net + language modeling [Metzler and Croft ’04]
• Powerful query language
– Designed to be simple to use, yet support complex information
needs
– Provides “adaptable, customizable scoring”
• Scalable
– Highly efficient code
– Distributed retrieval
– Incremental update
Model
• Based on original inference network retrieval
framework [Turtle and Croft ’91]
• Casts retrieval as inference in simple graphical model
• Extensions made to original model
– Incorporation of probabilities based on language modeling
rather than tf.idf
– Multiple language models allowed in the network (one per
indexed context)
Model
Model hyperparameters (observed)
Document node (observed)
α,βh1
α,βtitle
Context language models
θtitle
r1
α,βbody
D
…
θbody
rN
Representation nodes
(terms, phrases, etc…)
r1
…
q1
Information need node
(belief node)
θh1
rN
r1
…
rN
q2
I
Belief nodes
(#combine, #not, #max)
Model
α,βbody
D
α,βh1
α,βtitle
θtitle
r1
…
θbody
rN
r1
…
q1
θh1
rN
r1
q2
I
…
rN
P( r | θ )
• Probability of observing a term, phrase, or
feature given a context language model
– ri nodes are binary
• Assume r ~ Bernoulli( θ )
– “Model B” – [Metzler, Lavrenko, Croft ’04]
Model
α,βbody
D
α,βh1
α,βtitle
θtitle
r1
…
θbody
rN
r1
…
q1
θh1
rN
r1
q2
I
…
rN
P( θ | α, β, D )
• Prior over context language model determined
by α, β
• Assume P( θ | α, β ) ~ Beta( α, β )
–
–
–
–
Bernoulli’s conjugate prior
αr = μP( r | C ) + 1
βr = μP( ¬ r | C ) + 1
μ is a free parameter
P(ri |  ,  , D)   P(ri |  ) P( |  ,  , D) 

tf r , D  P(r | C )
| D | 
Model
α,βbody
D
α,βh1
α,βtitle
θtitle
r1
…
θbody
rN
r1
…
q1
θh1
rN
r1
q2
I
…
rN
P( q | r ) and P( I | r )
• Belief nodes are created dynamically based on
query
• Belief node estimates are derived from standard
link matrices
– Combine evidence from parents in various ways
– Allows fast inference by making marginalization
computationally tractable
• Information need node is simply a belief node that
combines all network evidence into a single value
• Documents are ranked according to P( I | α, β, D)
Example: #AND
P(Q=true|a,b)
A
B
0
false
false
0
false
true
0
true
false
1
true
true
A
B
Q
P#and (Q  true)   P(Q  true | A  a, B  b) P( A  a) P( B  b)
a ,b
 P(t | f , f )(1  p A )(1  p B )  P(t | f , t )(1  p A ) pB  P(t | t , f ) p A (1  pB )  P(t | t , t ) p A pB
 0(1  p A )(1  p B )  0(1  p A ) pB  0 p A (1  pB )  1 p A pB
 p A pB
Document Representation
<html>
<head>
<title>Department Descriptions</title>
</head>
<body>
The following list describes …
<h1>Agriculture</h1> …
<h1>Chemistry</h1> …
<h1>Computer Science</h1> …
<h1>Electrical Engineering</h1> …
…
<h1>Zoology</h1>
</body>
</html>
<title>
context
<title>department
descriptions</title>
<title>
extents
<body>
context
<body>the following
list describes …
<h1>agriculture</h1>
… </body>
<body>
extents
<h1>
context
<h1>agriculture</h1>
<h1>chemistry</h1>
…
<h1>zoology</h1>
<h1>
extents
.
.
.
1. department
descriptions
1. the following
list describes
<h1>agriculture
</h1> …
1. agriculture
2. chemistry
…
36. zoology
Terms
Type
Example
Matches
Stemmed term
dog
All occurrences of dog (and its
stems)
Surface term
“dogs”
Exact occurrences of dogs
(without stemming)
Term group (synonym group)
<”dogs” canine>
All occurrences of dogs (without
stemming) or canine (and its
stems)
POS qualified term
<”dogs” canine>.NNS
Same as previous, except
matches must also be tagged
with the NNS POS tag
Proximity
Type
Example
Matches
#odN(e1 … em) or
#N(e1 … em)
#od5(dog cat) or
#5(dog cat)
All occurrences of dog and cat
appearing ordered within a
window of 5 words
#uwN(e1 … em)
#uw5(dog cat)
All occurrences of dog and cat
that appear in any order within a
window of 5 words
#phrase(e1 … em)
#phrase(#1(willy wonka)
#uw3(chocolate factory))
System dependent
implementation (defaults to
#odm)
#syntax:xx(e1 … em)
#syntax:np(fresh powder)
System dependent
implementation
Context Restriction
Example
Matches
dog.title
All occurrences of dog appearing in the title context
dog.title,paragraph
All occurrences of dog appearing in both a title and paragraph
contexts (may not be possible)
<dog.title dog.paragraph>
All occurrences of dog appearing in either a title context or a
paragraph context
#5(dog cat).head
All matching windows contained within a head context
Context Evaluation
Example
Evaluated
dog.(title)
The term dog evaluated using the title context as the document
dog.(title, paragraph)
The term dog evaluated using the concatenation of the title and
paragraph contexts as the document
dog.figure(paragraph)
The term dog restricted to figure tags within the paragraph
context.
Belief Operators
INQUERY
#sum / #and
#wsum*
INDRI
#combine
#weight
#or
#not
#max
#or
#not
#max
* #wsum is still available in INDRI, but should be used with discretion
Extent Retrieval
Example
Evaluated
#combine[section](dog canine)
Evaluates #combine(dog canine) for each extent
associated with the section context
#combine[title, section](dog canine)
Same as previous, except is evaluated for each
extent associated with either the title context or
the section context
#sum(#sum[section](dog))
Returns a single score that is the #sum of the scores
returned from #sum(dog) evaluated for each
section extent
#max(#sum[section](dog))
Same as previous, except returns the maximum
score
Extent Retrieval Example
<document>
<section><head>Introduction</head>
Statistical language modeling allows formal
methods to be applied to information retrieval.
...
</section>
<section><head>Multinomial Model</head>
Here we provide a quick review of multinomial
language models.
...
</section>
<section><head>Multiple-Bernoulli Model</head>
We now examine two formal methods for
statistically modeling documents and queries
based on the multiple-Bernoulli distribution.
...
</section>
…
</document>
Query:
#combine[section]( dirichlet smoothing )
0.15
1. Treat each section
extent as a
“document”
0.50
2. Score each
“document”
according to
#combine( … )
0.05
SCORE
0.50
0.35
0.15
…
3. Return a ranked
list of extents.
DOCID
IR-352
IR-352
IR-352
…
BEGIN
51
405
0
…
END
205
548
50
…
Indri Examples
• “Where was George Washington born?”
#combine[sentence]( #1( george washington )
born #any:place )
• Paragraphs from news feed articles published between
1991 and 2000 that mention a person, a monetary
amount, and the company InfoCom
#filreq(#band( NewsFeed.doctype
#date:between(1991 2000) )
#combine[paragraph]( #any:person
#any:money InfoCom ) )
Example Indri Web Query
#weight(
0.1 #weight(
1.0 #prior(pagerank) 0.75 #prior(inlinks) )
1.0 #weight(
0.9 #combine(
#wsum( 1 stellwagen.(inlink)
1 stellwagen.(title)
3 stellwagen.(mainbody)
1 stellwagen.(heading) )
#wsum( 1 bank.(inlink)
1 bank.(title)
3 bank.(mainbody)
1 bank.(heading) ) )
0.1 #combine(
#wsum( 1 #uw8( stellwagen bank ).(inlink)
1 #uw8( stellwagen bank ).(title)
3 #uw8( stellwagen bank ).(mainbody)
1 #uw8( stellwagen bank ).(heading) ) ) ) )
Examples of Using IR for Structured Data
• XML search
• Relevance models for incomplete data
• Extracted entity retrieval
XML Search
• INEX workshop is similar to TREC but
focused on XML documents
• Queries contain varying degrees of
structural specification
– Not clear that these queries are realistic
• earlier study showed that people are not good about
remembering structure
– document structure can provide valuable
evidence for content representation
Example INEX Query
“NEXI”
Hierarchical Language Models
• Estimate a language model for each component
of a document tree (Ogilvie 2004, 2005)
• Smooth using a weighted mixture of a
background model, a document model, a
parent model, and a mixture of the children
models
Hierarchical Language Models
document
P(w|θdoc)
…
title
P(w|θtitle)
body
P(w|θbody)
section 1
section title
section 2
paragraph 1
…
…
…
bibliography
P(w|θbib)
section n
paragraph n
Does it work?
1
Content Only Topics
INEX 2003
0.9
Precision
0.8
Single Corpus Model
0.7
Shrinkage
0.6
Article Retrieval
0.5
0.4
0.3
0.2
0.1
0
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
Recall
Results from
Ogilvie, 2003
Does it work?
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Single Corpus Model
Results from
Ogilvie, 2003
Shrinkage
Recall
0.97
0.89
0.81
0.73
0.65
0.57
0.49
0.41
0.33
0.25
0.17
0.09
Element Based Retrieval
0.01
Precision
Content and Structure Topics
INEX 2003
Indri INEX extensions
• Indri incorporates hierarchical language models
• Allows weights to be set for different language
models and component type
• Query language extended to reference parent
and child extents
–
–
–
–
use the .\field operator to access a child reference
use the ./field operator to access a parent reference
use the .//field operator to access an ancestor reference
e.g. #combine[section]( bootstrap #combine[./title]( methodology ) )
Relevance Models for Incomplete Data
• Relevance models (Lavrenko, 2001) are used for
query expansion in IR based on generative LMs
• Estimates dependencies between words based
on training set or initial ranking
• Recently extended to semi-structured data for
applications where records are missing data
(Lavrenko, Yi, Allan, 2006)
– e.g. NSDL collection with fields title, description,
subject, content, audience
– 24% of 650,000 records have no subject field, 30% no
author, 96% no audience
Relevance Models for Incomplete Data
• Basic process is to estimate relevance models
for each field based on training data for a
query, then rank test records based on
comparison to relevance models
• Relevance model estimates how likely it is that
a word occurs in a field of a record, given that
a record matches the specified query fields
• Ranking is done using a weighted cross-entropy
– weights reflect importance of field
Relevance Models for Incomplete Data
• In NSDL experiment, 127 queries of form
{subject=’philosophy’ AND audience=‘high school’}
• In test collection, all records had subject and audience
field values removed
• Retrieved records had precision of 30% in top 10,
compared to 15% for a baseline that ranked text
records containing all fields
• Shows potential of probabilistic models for this type of
application
– can also generate structured queries (Calado et al, CIKM 02)
Extracted Entity Retrieval
• Information extraction extracts structure from
text
– e.g. names, addresses, email addresses, CVs,
publications, tables
• Creates semi-structured (and noisy) data rather
than databases
– Table extraction can be the basis for question
answering (Wei, Croft and McCallum, 2006)
– Publication extraction is the basis of CITESEER-like
systems (e.g. REXA, McCallum, 2005)
– Person extraction can be the basis for “expert
finding”
Expert Finding
• Evaluated in TREC Enterprise Track
• People are represented by text that co-occurs
with names
– which names? what text?
• People are ranked for a query using the text
“profile”
• Relevance model approach is effective
Conclusion
• For many applications involving retrieval of
semi-structured data, the right approach is an
IR system based on a probabilistic retrieval
model as a front-end, and a database system as
the back-end
– but IR system is not implemented using database
system
• “Right” means gives effective results and
supports users’ world view
• IR systems based on language models (e.g.
Indri) are a good candidate