Transcript A Risk Minimization Framework for Information Retrieval

Information Retrieval Models:
Vector Space Models
ChengXiang Zhai
Department of Computer Science
University of Illinois, Urbana-Champaign
Empirical IR vs. Model-based IR
•
Empirical IR:
– heuristic approaches
– solely rely on empirical evaluation
– assumptions not always clearly stated
– findings: empirical observations; may or may not generalize well
•
Model-based IR:
– theoretical approaches
– rely more on mathematics
– assumptions are explicitly stated
– findings: principles, models that may work well or not work well;
generalize better
•
Boundary may not be clear and a combination is
generally necessary
2
History of Research on IR Models
• 1960: First probabilistic model
• 1970s: Active research on retrieval models
[Maron & Kuhns 60]
started
– Vector-space model [Salton et al. 75]
– Classic probabilistic model [Robertson & Sparck Jones 76]
– Probability Ranking Principle [Robertson 77]
• 1980s: Further development of different models
– Non-classic logic model [Rijsbergen 86]
– Extended Boolean [Salton et al. 83]
– Early work on learning to rank [Fuhr 89]
3
History of Research on IR Models (cont.)
•
1990s: retrieval model research driven by TREC
– Inference network [Turtle & Croft 91]
– BM25/Okapi [Robertson et al. 94]
– Pivoted length normalization [Singhal et al. 96]
– Language model [Ponte & Croft 98]
•
2000s-present: retrieval model influenced by machine
learning and Web search
– Further development of language models [Zhai & Lafferty 01, Lavrenko &
Croft 01]
– Divergence from randomness [Amati et al. 02]
– Axiomatic model [Fang et al. 04]
– Markov Random Field [Metzler & Croft 05]
– Further development of Learning to rank [Joachimes 02, Burges et al. 05]
4
Modeling Relevance:
Raodmap for Retrieval Models
Relevance constraints
[Fang et al. 04]
Relevance
(Rep(q), Rep(d))
Similarity
Different
rep & similarity
…
P(d q) or P(q d)
Probabilistic inference
P(r=1|q,d) r {0,1}
Probability of Relevance
Regression
Model (Fuhr 89)
Generative
Model
Learn. To Rank Doc
(Joachims 02,
generation
Berges et al. 05)
Vector space
Prob. distr.
model
model
(Salton et al., 75) (Wong & Yao, 89)
Div. from Randomness
(Amati & Rijsbergen 02)
Classical
prob. Model
(Robertson &
Sparck Jones, 76)
Query
generation
Different
inference system
Prob. concept Inference
network
space model
(Wong & Yao, 95) model
(Turtle & Croft, 91)
LM
approach
(Ponte & Croft, 98)
(Lafferty & Zhai, 01a)
5
1. Vector Space Models
The Basic Question
Given a query, how do we know if
document A is more relevant than B?
One Possible Answer
If document A uses more query words
than document B
(Word usage in document A is more
similar to that in query)
Relevance = Similarity
• Assumptions
– Query and document are represented similarly
– A query can be regarded as a “document”
– Relevance(d,q)  similarity(d,q)
• R(q) = {dC|f(d,q)>}, f(q,d)=(Rep(q), Rep(d))
• Key issues
– How to represent query/document?
– How to define the similarity measure ?
8
Vector Space Model
• Represent a doc/query by a term vector
– Term: basic concept, e.g., word or phrase
– Each term defines one dimension
– N terms define a high-dimensional space
– Element of vector corresponds to term weight
– E.g., d=(x1,…,xN), xi is “importance” of term i
• Measure relevance by the distance between the
query vector and document vector in the vector
space
9
VS Model: illustration
Starbucks
D2
D9
D11
??
??
D5
D3
D10
D4 D6
Query
D7
D8
Microsoft
Java
D1
??
10
What the VS model doesn’t say
• How to define/select the “basic concept”
– Concepts are assumed to be orthogonal
• How to assign weights
– Weight in query indicates importance of term
– Weight in doc indicates how well the term
characterizes the doc
• How to define the similarity/distance measure
11
What’s a good “basic concept”?
• Orthogonal
– Linearly independent basis vectors
– “Non-overlapping” in meaning
• No ambiguity
• Weights can be assigned automatically and
hopefully accurately
• Many possibilities: Words, stemmed words,
phrases, “latent concept”, …
• “Bag of words” representation works
“surprisingly” well!
12
How to Assign Weights?
• Very very important!
• Why weighting
– Query side: Not all terms are equally important
– Doc side: Some terms carry more information about contents
• How?
– Two basic heuristics
• TF (Term Frequency) = Within-doc-frequency
• IDF (Inverse Document Frequency)
– Document length normalization
13
TF Weighting
• Idea: A term is more important if it occurs more
frequently in a document
• Formulas: Let f(t,d) be the frequency count of term
t in doc d
– Raw TF: TF(t,d) = f(t,d)
– Log TF: TF(t,d)=log ( f(t,d) +1)
– Maximum frequency normalization:
TF(t,d) = 0.5 +0.5*f(t,d)/MaxFreq(d)
– “Okapi/BM25 TF”:
TF(t,d) = k f(t,d)/(f(t,d)+k(1-b+b*doclen/avgdoclen))
• Normalization of TF is very important!
14
TF Normalization
• Why?
– Document length variation
– “Repeated occurrences” are less informative than
the “first occurrence”
• Two views of document length
– A doc is long because it uses more words
– A doc is long because it has more contents
• Generally penalize long doc, but avoid overpenalizing (e.g., pivoted normalization)
15
TF Normalization (cont.)
Norm. TF
Raw TF
“Pivoted normalization”: Using avg. doc length to regularize normalization
1-b+b*doclen/avgdoclen
b varies from 0 to 1
Normalization interacts with the similarity measure
16
IDF Weighting
• Idea: A term is more discriminative/important if it
occurs only in fewer documents
• Formula:
IDF(t) = 1+ log(n/k)
n – total number of docs
k -- # docs with term t (doc freq)
• Other variants:
– IDF(t) = log((n+1)/k)
– IDF(t)=log ((n+1)/(k+0.5))
• What are the maximum and minimum values of
IDF?
17
Non-Linear Transformation in IDF
IDF(t)
IDF(t) = 1+ log(n/k)
1+log(n)
Linear penalization
1
k (doc freq)
N
=totoal number of docs in collection
Is this transformation optimal?
18
TF-IDF Weighting
• TF-IDF weighting : weight(t,d)=TF(t,d)*IDF(t)
– Common in doc  high tf  high weight
– Rare in collection high idf high weight
• Imagine a word count profile, what kind of terms
would have high weights?
19
Empirical distribution of words
• There are stable language-independent patterns
in how people use natural languages
• A few words occur very frequently; most occur
rarely. E.g., in news articles,
– Top 4 words: 10~15% word occurrences
– Top 50 words: 35~40% word occurrences
• The most frequent word in one corpus may be
rare in another
20
Zipf’s Law
• rank * frequency  constant
Word
Freq.
F ( w) 
C
r ( w)
  1, C  0.1
High entropy words
Word Rank (by Freq)
Generalized Zipf’s law:
C
F ( w) 
[r ( w)  B]
Applicable in many domains
21
How to Measure Similarity?

Di  ( w i 1 ,...,w iN )

Q  ( wq1 ,...,wqN )
w  0 if a te rmis abse n t
N
 
Dot produ ctsim ilarity
: sim (Q , Di )   wqj  w ij
j 1
N
C osin e:
 
sim (Q , Di ) 
 wqj  wij
j 1
N
 ( wqj )
2

j 1
(  n orm aliz e dot
d produ ct)
How about Euclidean?
 
sim(Q, Di ) 
N
 ( wij )2
j 1
N
2
(
w

w
)
 qj ij
j 1
22
What Works the Best?
Error
[
]
•Use single words
•Use stat. phrases
•Remove stop words
•Stemming
•Others(?)
(Singhal 2001; Singhal et al. 1996)
23
Relevance Feedback in VS
•
Basic setting: Learn from examples
– Positive examples: docs known to be relevant
– Negative examples: docs known to be non-relevant
– How do you learn from this to improve performance?
•
General method: Query modification
– Adding new (weighted) terms
– Adjusting weights of old terms
– Doing both
•
The most well-known and effective approach is Rocchio
[Rocchio 1971]
24
Rocchio Feedback: Illustration
Centroid of relevant documents
Centroid of
non-relevant documents
-- --+
+
++ +
--+ q
q
m
+
+ +
- - - ++ + +++
-+ + +
- -- --
25
Rocchio Feedback: Formula
Parameters
New query
Origial query
Rel docs
Non-rel docs
26
Rocchio in Practice
•
•
•
How can we optimize the parameters?
Can it be used for both relevance feedback and pseudo
feedback?
How does Rocchio feedback affect the efficiency of
scoring documents? How can we improve the
efficiency?
27
Advantages of VS Model
• Empirically effective! (Top TREC performance)
• Intuitive
• Easy to implement
• Well-studied/Most evaluated
• The Smart system
– Developed at Cornell: 1960-1999
– Still widely used
• Warning: Many variants of TF-IDF!
28
Disadvantages of VS Model
• Assume term independence
• Assume query and document to be the same
• Lack of “predictive adequacy”
– Arbitrary term weighting
– Arbitrary similarity measure
• Lots of parameter tuning!
29
What You Should Know
• Basic idea of the vector space model
• TF-IDF weighting
• Pivoted length normalization (read [Singhal et
al. 1996] to know more)
• BM25/Okapi retrieval function (particularly TF
weighting)
• How Rocchio feedback works
30