Transcript Learning to Rank from heuristics to theoretic approach

Learning to Rank
from heuristics to theoretic approaches
Guest Lecture by Hongning Wang
[email protected]
Congratulations
• Job Offer
– Design the ranking module for Bing.com
Guest Lecture for Learing to Rank
2
How should I rank documents?
Answer: Rank by relevance!
Guest Lecture for Learing to Rank
3
Relevance ?!
Guest Lecture for Learing to Rank
4
The Notion of Relevance
Relevance constraints
[Fang et al. 04]
Relevance
(Rep(q), Rep(d))
Similarity
Different
rep & similarity
…
P(d q) or P(q d)
Probabilistic inference
P(r=1|q,d) r {0,1}
Probability of Relevance
Regression
Model (Fuhr 89)
Generative
Model
Learn. To Rank Doc
(Joachims 02,
generation
Berges et al. 05)
Vector space
Prob. distr.
model
model
(Salton et al., 75) (Wong & Yao, 89)
Div. from Randomness
(Amati & Rijsbergen 02)
Classical
prob. Model
(Robertson &
Sparck Jones, 76)
Query
generation
Different
inference system
Prob. concept Inference
network
space model
(Wong & Yao, 95) model
(Turtle & Croft, 91)
LM
approach
(Ponte & Croft, 98)
(Lafferty & Zhai, 01a)
Guest Lecture for Learing to Rank
5
Lecture notes from CS598CXZ
Relevance Estimation
• Query matching
– Language model
– BM25
– Vector space cosine similarity
• Document importance
– PageRank
– HITS
Guest Lecture for Learing to Rank
6
Did I do a good job of ranking
documents?
• IR evaluations
metrics
PageRank
BM25
– Precision
– MAP
– NDCG
Guest Lecture for Learing to Rank
7
Take advantage of different relevance estimator?
• Ensemble the cues
– Linear?
•
𝑎1 × 𝐵𝑀25 + 𝛼2 × 𝐿𝑀 + 𝛼3 × 𝑃𝑎𝑔𝑒𝑅𝑎𝑛𝑘 + 𝛼4 × 𝐻𝐼𝑇𝑆
–{𝛼Non-linear?
1 = 0.4, 𝛼2 = 0.2, 𝛼3 = 0.1, 𝛼4 = 0.1} → {𝑀𝐴𝑃 = 0.20, 𝑁𝐷𝐶𝐺 = 0.6}
{𝛼1 =
𝛼2 = 0.2,
𝛼3 = 0.3, 𝛼4 = 0.1} → {𝑀𝐴𝑃 = 0.12, 𝑁𝐷𝐶𝐺 = 0.5 }
• 0.2,
Decision
tree-like
{𝛼1 = 0.1, 𝛼2 = 0.1, 𝛼3 = 0.5, 𝛼4 = 0.2} → {𝑀𝐴𝑃 = 0.18, 𝑁𝐷𝐶𝐺 = 0.7}
BM25>0.5
True
False
PageRank
> 0.3
LM > 0.1
True
False
True
False
r= 1.0
r= 0.7
r= 0.4
r= 0.1
Guest Lecture for Learing to Rank
8
What if we have thousands of features?
• Is there any way I can do better?
– Optimizing the metrics automatically!
Where to find those tree structures?
How to determine those 𝛼s?
Guest Lecture for Learing to Rank
9
Rethink the task
• Given: (query, document) pairs represented by
a set of relevance estimators, a.k.a., features
DocID
BM25
LM
PageRank
Label
0001
1.6
1.1
0.9
0
0002
2.7
1.9
0.2
1
• Needed: a way of combining the estimators
– 𝑓 𝑞, 𝑑
𝐷
𝑖=1
→ ordered 𝑑
𝐷
𝑖=1
• Criterion: optimize IR metrics
Key!
– P@k, MAP, NDCG, etc.
Guest Lecture for Learing to Rank
10
M
Machine Learning
• Input: { 𝑋1 , 𝑌1 , 𝑋2 , 𝑌2 , … , (𝑋𝑛 , 𝑌𝑛 )}, where
𝑋𝑖 ∈ 𝑅𝑁 , 𝑌𝑖 ∈ 𝑅𝑀
• Object function : 𝑂(𝑌 ′ , 𝑌)
We will only
• Output: 𝑓 𝑋 → 𝑌, such that 𝑓 = NOTE:
talk about supervised
learning.
argmax𝑓′⊂𝐹 𝑂(𝑓 ′ (𝑋), 𝑌)
Regression
Classification
𝑂 𝑌 ′ , 𝑌 = −||𝑌 ′ − 𝑌||
𝑂 𝑌 ′ , 𝑌 = 𝛿(𝑌 ′ = 𝑌)
http://en.wikipedia.org/wiki/Regress
ion_analysis
http://en.wikipedia.org/wiki/Stat
istical_classification
Guest Lecture for Learing to Rank
11
Learning to Rank
• General solution in optimization framework
– Input: (𝑞𝑖 , 𝑑1 ), 𝑦1 , (𝑞𝑖 , 𝑑2 ), 𝑦2 , … , (𝑞𝑖 , 𝑑𝑛 ), 𝑦𝑛 ,
where dn ∈ 𝑅𝑁 , 𝑦𝑖 ∈ {0, … , 𝐿}
– Object: O = {P@k, MAP, NDCG}
– Output: 𝑓 𝑞, 𝑑 → 𝑌, s.t., 𝑓 =
argmax𝑓′⊂𝐹 𝑂(𝑓 ′ (𝑞, 𝑑), 𝑌)
DocID
BM25
LM
PageRank
Label
0001
1.6
1.1
0.9
0
0002
2.7
1.9
0.2
1
Guest Lecture for Learing to Rank
12
Challenge: how to optimize?
• Evaluation metric recap
– Average Precision
•
– DCG
Not continuous with respect to f(X)!
•
• Order is essential!
– 𝑓 → order → metric
Guest Lecture for Learing to Rank
13
Approximating the Objects!
• Pointwise
– Fit the relevance labels individually
• Pairwise
– Fit the relative orders
• Listwise
– Fit the whole order
Guest Lecture for Learing to Rank
14
Pointwise Learning to Rank
• Ideally perfect relevance prediction leads to
perfect ranking
– 𝑓 → score → order → metric
• Reducing ranking problem to
– Regression
• 𝑂 𝑓 𝑄, 𝐷 , 𝑌 = − 𝑖 |𝑓 𝑞𝑖 , 𝑑𝑖 − 𝑦𝑖 |
• Subset Ranking using Regression, D.Cossock and T.Zhang,
COLT 2006
– (multi-)Classification
• 𝑂 𝑓 𝑄, 𝐷 , 𝑌 = 𝑖 𝛿(𝑓 𝑞𝑖 , 𝑑𝑖 = 𝑦𝑖 )
• Ranking with Large Margin Principles, A. Shashua and A.
Levin, NIPS 2002
Guest Lecture for Learing to Rank
15
Subset Ranking using Regression
D.Cossock and T.Zhang, COLT 2006
• Fit relevance labels via regression
–
– Emphasize more on relevant documents
•
Weights on each document
http://en.wikipedia.org/wiki/Regression_analysis
Guest Lecture for Learing to Rank
Most positive document
16
Ranking with Large Margin Principles
A. Shashua and A. Levin, NIPS 2002
• Goal: correctly placing the documents in the
corresponding category and maximize the
margin
Y=1
Reduce the
violations
Y=2
Y=0
𝛼𝑠
Guest Lecture for Learing to Rank
17
Ranking with Large Margin Principles
A. Shashua and A. Levin, NIPS 2002
• Maximizing the sum of margins
Y=0
Y=1
Y=2
𝛼𝑠
Guest Lecture for Learing to Rank
18
Ranking with Large Margin Principles
A. Shashua and A. Levin, NIPS 2002
• Ranking lost is consistently decreasing with
more training data
Guest Lecture for Learing to Rank
19
What did we learn
• Machine learning helps!
– Derive something optimizable
– More efficient and guided
Guest Lecture for Learing to Rank
20
There is always a catch
• Cannot directly optimize IR metrics
– (0 → 1, 2 → 0) worse than (0->-2, 2->4)
• Position of documents are ignored
– Penalty on documents at higher positions should
be larger
• Favor the queries with more documents
Guest Lecture for Learing to Rank
21
Pairwise Learning to Rank
• Ideally perfect partial order leads to perfect ranking
– 𝑓 → partial order → order → metric
• Ordinal regression
– 𝑂 𝑓 𝑄, 𝐷 , 𝑌 =
𝑖≠𝑗 𝛿(𝑦𝑖
> 𝑦𝑗 )𝛿(𝑓 𝑞𝑖 , 𝑑𝑖 > 𝑓(𝑞𝑖 , 𝑑𝑖 ))
• Relative ordering between different documents is
significant
• E.g., (0->-2, 2->4) is better than (0 → 1, 2 → 0)
– Large body of work
Guest Lecture for Learing to Rank
22
Optimizing Search Engines using Clickthrough Data
Thorsten Joachims, KDD’02
• Minimizing the number of mis-ordered pairs
linear combination of features
𝑦1 > 𝑦2
𝑓 𝑞, 𝑑 = 𝑤 𝑇 𝑋
𝑦1 > 𝑦2 , 𝑦2 > 𝑦3 , 𝑦1 > 𝑦4
𝑞,𝑑
1
0
Keep the relative orders
RankingSVM
Guest Lecture for Learing to Rank
23
Optimizing Search Engines using Clickthrough Data
Thorsten Joachims, KDD’02
• How to use it?
– 𝑓 → s𝐜𝐨𝐫𝐞 → order
𝑦1 > 𝑦2 > 𝑦3 > 𝑦4
Guest Lecture for Learing to Rank
24
Optimizing Search Engines using Clickthrough Data
Thorsten Joachims, KDD’02
• What did it learn from the data?
– Linear correlations
Positive
correlated
features
Negative
correlated
features
Guest Lecture for Learing to Rank
25
Optimizing Search Engines using Clickthrough Data
Thorsten Joachims, KDD’02
• How good is it?
– Test on real system
Guest Lecture for Learing to Rank
26
An Efficient Boosting Algorithm for Combining
Preferences
Y. Freund, R. Iyer, et al. JMLR 2003
• Smooth the loss on mis-ordered pairs
–
𝑦𝑖 >𝑦𝑗 𝑃𝑟
𝑑𝑖 , 𝑑𝑗 𝑒𝑥𝑝[𝑓 𝑞, 𝑑𝑗 − 𝑓 𝑞, 𝑑𝑖 ]
square loss
hinge loss (RankingSVM)
0/1 loss
exponential loss
Guest Lecture for Learing to Rank
from Pattern Recognition
27 and
Machine Learning, P337
An Efficient Boosting Algorithm for Combining
Preferences
Y. Freund, R. Iyer, et al. JMLR 2003
• RankBoost: optimize via boosting
– Vote by a committee
BM25
PageRank
Cosine
Updating
Pr(𝑑𝑖 , 𝑑𝑗 )
Credibility of each
committee member
(ranking feature)
Guest Lecture for Learing to Rank
from Pattern Recognition and
28
Machine Learning, P658
An Efficient Boosting Algorithm for Combining
Preferences
Y. Freund, R. Iyer, et al. JMLR 2003
• How good is it?
Guest Lecture for Learing to Rank
29
A Regression Framework for Learning Ranking
Functions Using Relative Relevance Judgments
Zheng et al. SIRIG’07
• Non-linear ensemble of features
– Object:
𝑦𝑖 >𝑦𝑗
max 0, 𝑓 𝑞, 𝑑𝑗 − 𝑓 𝑞, 𝑑𝑖
2
– Gradient descent boosting tree
• Boosting tree
– Using regression tree to minimize the residuals
– 𝑟 𝑡 (𝑞, 𝑑, 𝑦) = 𝑂𝑡 (𝑞, 𝑑, 𝑦) − 𝑓
𝑡−1
(𝑞, 𝑑, 𝑦)
BM25>0.5
True
LM > 0.1
True
False
r= 1.0
False
PageRank
> 0.3
True
False
Guest Lecture for Learing to Rank
r= 0.7
r= 0.4
r= 0.1
30
A Regression Framework for Learning Ranking
Functions Using Relative Relevance Judgments
Zheng et al. SIRIG’07
• Non-linear v.s. linear
– Comparing with RankingSVM
Guest Lecture for Learing to Rank
31
Where do we get the relative orders
• Human annotations
– Small scale, expensive to acquire
• Clickthroughs
– Large amount, easy to acquire
Guest Lecture for Learing to Rank
32
Accurately Interpreting Clickthrough Data
as Implicit Feedback
• Position bias
Thorsten Joachims, et al., SIGIR’05
Your click is not because
of its relevance, but its
position?!
Guest Lecture for Learing to Rank
33
Accurately Interpreting Clickthrough Data
as Implicit Feedback
Thorsten Joachims, et al., SIGIR’05
• Controlled experiment
– Over trust of the top ranked positions
Guest Lecture for Learing to Rank
34
Accurately Interpreting Clickthrough Data
as Implicit Feedback
Thorsten Joachims, et al., SIGIR’05
• Pairwise preference matters
– Click: examined and clicked document
– Skip: examined but non-clicked document
Click > Skip
Guest Lecture for Learing to Rank
35
What did we learn
• Predicting relative order
– Getting closer to the nature of ranking
• Promising performance in practice
– Pairwise preferences from click-throughs
Guest Lecture for Learing to Rank
36
Listwise Learning to Rank
• Can we directly optimize the ranking?
– 𝑓 → order → metric
• Tackle the challenge
– Optimization without gradient
Guest Lecture for Learing to Rank
37
From RankNet to LambdaRank to
LambdaMART: An Overview
Christopher J.C. Burges, 2010
• Minimizing mis-ordered pair => maximizing IR
metrics?
Mis-ordered pairs: 6
AP:
Mis-ordered pairs: 4
5
8
AP:
DCG: 1.333
5
12
DCG: 0.931
Position is crucial!
Guest Lecture for Learing to Rank
38
From RankNet to LambdaRank to
LambdaMART: An Overview
Christopher J.C. Burges, 2010
• Weight the mis-ordered pairs?
– Some pairs are more important to be placed in the
right order
Depend on the ranking of document i, j
– Inject into object function in the whole list
•
𝑦𝑖 >𝑦𝑗 Ω
𝑑𝑖 , 𝑑𝑗 exp −𝑤ΔΦ 𝑞𝑛 , 𝑑𝑖 , 𝑑𝑗
– Inject into gradient
• 𝜆𝑖𝑗 =
𝜕𝑂𝑎𝑝𝑝𝑟𝑜
𝜕𝑤
Δ𝑂
Change in original object, e.g., NDCG, if
we switch the documents i and j, leaving
the other documents unchanged
Gradient with respect to approximated object,
i.e., exponential loss on mis-ordered
pairs
Guest Lecture for Learing to Rank
39
From RankNet to LambdaRank to
LambdaMART: An Overview
Christopher J.C. Burges, 2010
• Lambda functions
– Gradient?
• Yes, it meets the sufficient and necessary condition of
being partial derivative
– Lead to optimal solution of original problem?
• Empirically
Guest Lecture for Learing to Rank
40
From RankNet to LambdaRank to
LambdaMART: An Overview
Christopher J.C. Burges, 2010
• Evolution
Object
Gradient
(𝜆 function)
Optimization
method
RankNet
LambdaRank
LambdaMART
Cross entropy over
the pairs
Unknown
Unknown
Gradient of cross
entropy
Gradient of cross
entropy times
pairwise change in
target metric
Gradient of cross
entropy times
pairwise change in
target metric
neural network
stochastic gradient
descent
Multiple Additive
Regression Trees
(MART)
As we discussed
Optimize solely
Guest Lecture for Learing to Rank
in RankBoost
by gradient
Non-linear
combination
41
From RankNet to LambdaRank to
LambdaMART: An Overview
Christopher J.C. Burges, 2010
• A Lambda tree
splitting
Combination of
features
Guest Lecture for Learing to Rank
42
AdaRank: a boosting algorithm for
information retrieval
Jun Xu & Hang Li, SIGIR’07
• Loss defined by IR metrics
–
𝑞∈𝑄 𝑃𝑟
𝑞 𝑒𝑥𝑝[−𝑂(𝑞)]
– Optimizing by boosting
BM25
PageRank
Target metrics: MAP, NDCG, MRR
Cosine
Updating Pr(𝑞)
Credibility of each
committee member
(ranking feature)
Guest Lecture for Learing to Rank
from Pattern Recognition
43 and
Machine Learning, P658
A Support Vector Machine for Optimizing
Average Precision
Yisong Yue, et al., SIGIR’07
RankingSVM
• Minimizing the pairwise loss
SVM-MAP
• Minimizing the structural
loss
MAP difference
Loss defined on the
quality of the whole list
of ordered documents
Loss defined on the
number of mis-ordered
document pairs
Guest Lecture for Learing to Rank
44
A Support Vector Machine for Optimizing
Average Precision
Yisong Yue, et al., SIGIR’07
• Max margin principle
– Push the ground-truth far away from any mistakes
you might make
– Finding the most violated constraints
Guest Lecture for Learing to Rank
45
A Support Vector Machine for Optimizing
Average Precision
Yisong Yue, et al., SIGIR’07
• Finding the most violated constraints
– MAP is invariant to permutation of (ir)relevant
documents
– Maximize MAP over a series of swaps between
relevant and irrelevant documents
•
Start from the reverse
Right-hand side of constraints order of ideal ranking
Greedy solution
Guest Lecture for Learing to Rank
46
A Support Vector Machine for Optimizing
Average Precision
Yisong Yue, et al., SIGIR’07
• Experiment results
Guest Lecture for Learing to Rank
47
Other listwise solutions
• Soften the metrics to make them
differentiable
– Michael Taylor et al., SoftRank: optimizing nonsmooth rank metrics, WSDM'08
• Minimize a loss function defined on
permutations
– Zhe Cao et al., Learning to rank: from pairwise
approach to listwise approach, ICML'07
Guest Lecture for Learing to Rank
48
What did we learn
• Taking a list of documents as a whole
– Positions are visible for the learning algorithm
– Directly optimizing the target metric
• Limitation
– The search space is huge!
Guest Lecture for Learing to Rank
49
Summary
• Learning to rank
– Automatic combination of ranking features for
optimizing IR evaluation metrics
• Approaches
– Pointwise
• Fit the relevance labels individually
– Pairwise
• Fit the relative orders
– Listwise
• Fit the whole order
Guest Lecture for Learing to Rank
50
Experimental Comparisons
• Ranking performance
Guest Lecture for Learing to Rank
51
Experimental Comparisons
• Winning count
– Over seven different data sets
Guest Lecture for Learing to Rank
52
Experimental Comparisons
• My experiments
– 1.2k queries, 45.5K documents with 1890 features
– 800 queries for training, 400 queries for testing
ListNET
LambdaMART
RankNET
RankBoost
RankingSVM
AdaRank
pLogistic
Logistic
MAP
0.2863
0.4644
0.3005
0.4548
0.3507
0.4321
0.4519
0.4348
P@1
0.2074
0.4630
0.2222
0.4370
0.2370
0.4111
0.3926
0.3778
ERR
0.1661
0.2654
0.1873
0.2463
0.1895
0.2307
0.2489
0.2410
Guest Lecture for Learing to Rank
MRR
0.3714
0.6105
0.3816
0.5829
0.4154
0.5482
0.5535
0.5526
NDCG@5
0.2949
0.5236
0.3386
0.4866
0.3585
0.4421
0.4945
0.4762
53
Analysis of the Approaches
• What are they really optimizing?
– Relation with IR metrics
g
a
p
Guest Lecture for Learing to Rank
54
Pointwise Approaches
• Regression based
Discount coefficients
in DCG
Regression loss
• Classification based
Discount coefficients Classification loss
in DCG
Guest Lecture for Learing to Rank
55
From
Guest Lecture for Learing to Rank
56
Discount coefficients in DCG
From
Guest Lecture for Learing to Rank
57
Listwise Approaches
• No general analysis
– Method dependent
– Directness and consistency
Guest Lecture for Learing to Rank
58
Connection with Traditional IR
• People have foreseen this topic long time ago
– Nicely fit in the risk minimization framework
Guest Lecture for Learing to Rank
59
Applying Bayesian Decision Theory
Loss
Choice: (D1,1)
L
Choice: (D2,2)
L
query q
user U
q
1
...
L
Choice: (Dn,n)
doc set C
source S
N
Metric to be optimized Available ranking features
( D *,  *)  arg min  L ( D ,  ,  ) p ( | q , U , C , S ) d 
D ,
RISK MINIMIZATION

loss
hidden observed
Bayes risk for choice (D, )
Guest Lecture for Learing to Rank
60
Traditional Solution
• Set-based models (choose D)
• Ranking models (choose )
– Independent loss
Pointwise
• Relevance-based loss
Probabilistic relevance model
Generative Relevance Theory
Vector-space Model
Two-stage LM
KL-divergence model
• Distance-based loss
– Dependent loss
• MMR loss
• MDR loss
Boolean model
Subtopic retrieval model
Pairwise/Listwise
Guest Lecture for Learing to Rank
61
Traditional Notion of Relevance
Relevance constraints
[Fang et al. 04]
Relevance
(Rep(q), Rep(d))
Similarity
Different
rep & similarity
…
P(d q) or P(q d)
Probabilistic inference
P(r=1|q,d) r {0,1}
Probability of Relevance
Regression
Model (Fuhr 89)
Generative
Model
Learn. To Rank Doc
(Joachims 02,
generation
Berges et al. 05)
Vector space
Prob. distr.
model
model
(Salton et al., 75) (Wong & Yao, 89)
Div. from Randomness
(Amati & Rijsbergen 02)
Classical
prob. Model
(Robertson &
Sparck Jones, 76)
Query
generation
Different
inference system
Prob. concept Inference
network
space model
(Wong & Yao, 95) model
(Turtle & Croft, 91)
LM
approach
(Ponte & Croft, 98)
(Lafferty & Zhai, 01a)
Guest Lecture for Learing to Rank
62
Broader Notion of Relevance
• Traditional view
– Content-driven
• Vector space model
• Probability relevance model
• Language model
Query-Document specific
Unsupervised
• Modern view
– Anything related to the quality of the document
•
•
•
•
•
Clicks/views
Link structure
Visual structure
Social network
….
Query, Document, QueryDocument specific
Supervised
Guest Lecture for Learing to Rank
63
Broader Notion of Relevance
Language Model
Query
Cosine
Documents
BM25
Linkage structure
Query relation
Language Model
Social network
Query
Cosine
Click/View
Documents
Visual
Structure
BM25
Linkage structure
Query relation
Language Model
Query
Cosine
Documents
likes
BM25
Guest Lecture for Learing to Rank
64
Future
•
•
•
•
Tighter bounds
Faster solution
Larger scale
Wider application scenario
Guest Lecture for Learing to Rank
65
Resources
• Books
– Liu, Tie-Yan. Learning to rank for information retrieval. Vol. 13.
Springer, 2011.
– Li, Hang. "Learning to rank for information retrieval and natural
language processing." Synthesis Lectures on Human Language
Technologies 4.1 (2011): 1-113.
• Helpful pages
– http://en.wikipedia.org/wiki/Learning_to_rank
• Packages
– RankingSVM: http://svmlight.joachims.org/
– RankLib: http://people.cs.umass.edu/~vdang/ranklib.html
• Data sets
– LETOR http://research.microsoft.com/enus/um/beijing/projects/letor//
– Yahoo! Learning to rank challenge
http://learningtorankchallenge.yahoo.com/
Guest Lecture for Learing to Rank
66
References
• Liu, Tie-Yan. "Learning to rank for information retrieval." Foundations and
Trends in Information Retrieval 3.3 (2009): 225-331.
• Cossock, David, and Tong Zhang. "Subset ranking using
regression." Learning theory (2006): 605-619.
• Shashua, Amnon, and Anat Levin. "Ranking with large margin principle:
Two approaches." Advances in neural information processing systems 15
(2003): 937-944.
• Joachims, Thorsten. "Optimizing search engines using clickthrough data."
Proceedings of the eighth ACM SIGKDD. ACM, 2002.
• Freund, Yoav, et al. "An efficient boosting algorithm for combining
preferences." The Journal of Machine Learning Research 4 (2003): 933969.
• Zheng, Zhaohui, et al. "A regression framework for learning ranking
functions using relative relevance judgments." Proceedings of the 30th
annual international ACM SIGIR. ACM, 2007.
Guest Lecture for Learing to Rank
67
References
• Joachims, Thorsten, et al. "Accurately interpreting clickthrough data as
implicit feedback." Proceedings of the 28th annual international ACM
SIGIR. ACM, 2005.
• Burges, C. "From ranknet to lambdarank to lambdamart: An overview."
Learning 11 (2010): 23-581.
• Xu, Jun, and Hang Li. "AdaRank: a boosting algorithm for information
retrieval." Proceedings of the 30th annual international ACM SIGIR. ACM,
2007.
• Yue, Yisong, et al. "A support vector method for optimizing average
precision." Proceedings of the 30th annual international ACM SIGIR. ACM,
2007.
• Taylor, Michael, et al. "Softrank: optimizing non-smooth rank metrics."
Proceedings of the international conference WSDM. ACM, 2008.
• Cao, Zhe, et al. "Learning to rank: from pairwise approach to listwise
approach." Proceedings of the 24th ICML. ACM, 2007.
Guest Lecture for Learing to Rank
68
Thank you!
• Q&A
Guest Lecture for Learing to Rank
69
Recap of last lecture
• Goal
– Design the ranking module for Bing.com
Guest Lecture for Learing to Rank
70
Basic Search Engine Architecture
• The anatomy of a large-scale hypertextual
Web search engine
– Sergey Brin and Lawrence Page. Computer
networks and ISDN systems 30.1 (1998): 107-117.
Your Job
Guest Lecture for Learing to Rank
71
Learning to Rank
• Given: (query, document) pairs represented by
a set of relevance estimators, a.k.a., features
QueryID
DocID
BM25
LM
PageRank
Label
0001
0001
1.6
1.1
0.9
0
0001
0002
2.7
1.9
0.2
1
• Needed: a way of combining the estimators
– 𝑓 𝑞, 𝑑
𝐷
𝑖=1
→ ordered 𝑑
𝐷
𝑖=1
• Criterion: optimize IR metrics
Key!
– P@k, MAP, NDCG, etc.
Guest Lecture for Learing to Rank
72
Challenge: how to optimize?
• Order is essential!
– 𝑓 → order → metric
• Evaluation metrics are not continuous and not
differentiable
Guest Lecture for Learing to Rank
73
Approximating the Objects!
• Pointwise
– Fit the relevance labels individually
• Pairwise
– Fit the relative orders
• Listwise
– Fit the whole order
Guest Lecture for Learing to Rank
74
Pointwise Learning to Rank
• Ideally perfect relevance prediction leads to
perfect ranking
– 𝑓 → score → order → metric
• Reduce ranking problem to
– Regression
- Classification
http://en.wikipedia
.org/wiki/Regressio
n_analysis
Guest Lecture for Learing to Rank
http://en.wikipedia.or
g/wiki/Statistical_clas
sification
75
Deficiency
• Cannot directly optimize IR metrics
– (0 → 1, 2 → 0) worse than (0->-2, 2->4)
• Position of documents are ignored
2
– Penalty on documents1 at higher
positions should
𝑑1′′
be larger -1 0
𝑑2
𝑑1
• Favor the queries with more documents
-2
𝑑1′
Guest Lecture for Learing to Rank
76
Pairwise Learning to Rank
• Ideally perfect partial order leads to perfect ranking
– 𝑓 → partial order → order → metric
• Ordinal regression
– 𝑂 𝑓 𝑄, 𝐷 , 𝑌 =
𝑖≠𝑗 𝛿(𝑦𝑖
> 𝑦𝑗 )𝛿(𝑓 𝑞𝑖 , 𝑑𝑖 < 𝑓(𝑞𝑖 , 𝑑𝑖 ))
• Relative ordering between different documents is
significant
2
1
0
-1
-2
𝑑1′′
𝑑2
𝑑1
𝑑1′
Guest Lecture for Learing to Rank
77
RankingSVM
Thorsten Joachims, KDD’02
• Minimizing the number of mis-ordered pairs
linear combination of features
𝑦1 > 𝑦2
𝑓 𝑞, 𝑑 = 𝑤 𝑇 𝑋
𝑦1 > 𝑦2 , 𝑦2 > 𝑦3 , 𝑦1 > 𝑦4
𝑞,𝑑
1
0
QueryID
DocID
BM25
PageRank
Label
0001
0001
1.6
0.9
4
0001
0002
2.7
0.2
2
0001
0003
1.3
0.2
1
0.7
0
0001
Keep the0004
relative 1.2
orders
Guest Lecture for Learing to Rank
78
RankingSVM
Thorsten Joachims, KDD’02
• Minimizing the number of mis-ordered pairs
𝑤ΔΦ 𝑞𝑛 , 𝑑𝑖 , 𝑑𝑗 ≥ 1 − 𝜉𝑖,𝑗,𝑛
Guest Lecture for Learing to Rank
79
General Idea of Pairwise Learning to Rank
• For any pair of 𝑦𝑖 > 𝑦𝑗
square loss (GBRank) max 0,1 − 𝑤ΔΦ 𝑞𝑛 , 𝑑𝑖 , 𝑑𝑗
2
hinge loss (RankingSVM)
max 0,1 − 𝑤ΔΦ 𝑞𝑛 , 𝑑𝑖 , 𝑑𝑗
0/1 loss
𝛿 𝑤ΔΦ 𝑞𝑛 , 𝑑𝑖 , 𝑑𝑗 < 0
exponential loss
exp −𝑤ΔΦ 𝑞𝑛 , 𝑑𝑖 , 𝑑𝑗
𝑤ΔΦ 𝑞𝑛 , 𝑑𝑖 , 𝑑𝑗
Guest Lecture for Learing to Rank
80