Transcript game-theory-ir-2015-dais - University of Illinois at Urbana

Towards a Game-Theoretic Framework
for Information Retrieval
ChengXiang (“Cheng”) Zhai
Department of Computer Science
University of Illinois at Urbana-Champaign
http://www.cs.uiuc.edu/homes/czhai
Email: [email protected]
Yahoo!-DAIS Seminar, UIUC, Jan 23, 2015
1
Search is everywhere,
and part of everyone’s life
Web Search
Desk Search
Enterprise Search
Social Media Search
Site Search
……
2
Search accuracy matters!
# Queries /Day
X 1 sec
X 10 sec
4,700,000,000
~1,300,000 hrs ~13,000,000 hrs
1,600,000,000
~440,000 hrs ~4,400,000 hrs
3,000,000
……
~550 hrs
~5,500 hrs
How can we optimize all search engines in a general way?
Sources:
Google: http://www.statisticbrain.com/google-searches/
Twitter: http://www.statisticbrain.com/twitter-statistics/
PubMed: http://www.nlm.nih.gov/services/pubmed_searches.html
3
How can we optimize all search engines in a general way?
However, this is an ill-defined
question!
What is a search engine?
What is an optimal search engine?
What should be the objective function to optimize?
4
Current-generation search engines
number of queries
k search engines
Document collection
Query
Q
Retrieval task = rank documents for a query
Interface = ranked list
( “10 blue links”)
Ranked
list 
Score(Q,D)
Optimal Search Engine=optimal score(q,d)
D
Machine Learning
Retrieval
Model
Objective = ranking accuracy on training data
Minimum NLP
5
Current search engines are well justified
• Probability ranking principle [Robertson 77]:returning a
ranked list of documents in descending order of
probability that a document is relevant to the query is
the optimal strategy under two assumptions:
– The utility of a document (to a user) is independent of
the utility of any other document
– A user would browse the results sequentially
• Intuition: if a user sequentially examines one doc
at each time, we’d like the user to see the very
best ones first
6
Success of Probability Ranking Principle
• Vector Space Models: [Salton et al. 75], [Singhal et al. 96], …
• Classic Probabilistic Models: [Maron & Kuhn 60], [Harter 75],
[Robertson & Sparck Jones 76], [van Rijsbergen 77], [Robertson 77],
[Robertson et al. 81], [Robertson & Walker 94], …
• Language Models: [Ponte & Croft 98], [Hiemstra & Kraaij 98], [Zhai &
Lafferty 01], [Lavrenko & Croft 01], [Kurland & Lee 04], …
• Non-Classic Logic Models: [van Rijsbergen 86], [Wong & Yao 95], …
• Divergence from Randomness: [Amati & van Rijsbergen 02], [He &
Ounis 05], …
• Learning to Rank: [Fuhr 89], [Gey 94], ...
• Axiomatic retrieval framework [Fang et al. 04], [Clinchant & Gaussier
10], [Fang et al. 11], …
• …
Most information retrieval models are to optimize score(Q,D)
7
Limitations of PRP 
Limitations of optimizing Score(Q,D)
• Assumptions made by PRP don’t hold in practice
– Utility of a document depends on others
– Users don’t strictly follow sequential browsing
• As a result
– Redundancy can’t be handled (duplicated docs have
the same score!)
– Collective relevance can’t be modeled
– Heuristic post-processing of search results is
inevitable
8
Improvement: instead of scoring one
document, score a whole ranked list
• Instead of scoring an individual document, score an
entire candidate ranked list of documents [Zhai 02; Zhai &
Lafferty 06]
– A list with redundant documents on the top can be
penalized
– Collective relevance can be captured also
– Powerful machine learning techniques can be used
[Cao et al. 07]
• PRP extended to address interaction of users [Fuhr 08]
• However, scoring is still for just one query: score(Q, )
Optimal SE = optimal score(Q, )
Objective = Ranking accuracy on training data
9
Limitations of single query scoring
•
•
•
•
No consideration of past queries and history
No modeling of users
Can’t optimize the utility over an entire session
…
10
Heuristic solutions  emerging topics in IR
• No consideration of past queries and history
 Implicit feedback (e.g, [Shen et al. 05] ), personalized search
(see, e.g., [Teevan et al. 10])
• No modeling of users
 intent modeling (see, e.g. , [Shen et al. 06]), task inference
(see, e.g., [Wang et al. 13])
• Can’t optimize the utility over an entire session
 Active feedback (e.g., [Shen & Zhai 05]), exploration-exploitation
tradeoff (e.g., [Agarwal et al. 09], [Karimzadehgan & Zhai 13])
 POMDP for session search [Luo et al. 14]
Can we solve all these problems in a more principled way with
a unified formal framework?
11
Going back to the basic questions…
•
•
•
•
What is a search engine?
What is an optimal search engine?
What should be the objective function to optimize?
How can we solve such an optimization problem?
12
Proposed Solution:
A Game-Theoretic Framework for IR
• Retrieval process = cooperative game-playing
• Players: Player 1= search engine; Player 2= user
• Rules of game:
–
–
–
–
–
Each player takes turns to make “moves”
User or system (in case of recommender system) makes the first move
User makes the last move (usually)
For each move of the user, the system makes a response move
Current search engine:
• User’s moves= {query, click}; system’s moves = {ranked list, show doc}
• Objective: multiple possibilities
– satisfying the user’s information need with minimum effort of user and
minimum resource overhead of the system.
– Given a constant effort of a user, subject to constraints of system resources,
maximize the utility of delivered information to the user
– Given a fixed “budget” for system resources, and an upper bound of user
effort, maximize the utility of delivered information
13
Search as a Sequential Game
(Satisfy an information need
with minimum effort)
User
A1 : Enter a query
Which items
to view?
A2 : View item
View more?
(Satisfy an information need
with minimum user effort, minimum resource)
System
Which information items to present?
How to present them?
Ri: results (i=1, 2, 3, …)
Which aspects/parts of the item
to show? How?
R’: Item summary/preview
A3 : Scroll down or click on
“Back”/”Next” button
14
Retrieval Task = Sequential Decision-Making
History H={(Ai,Ri)}
i=1, …, t-1
Given U, C, At , and H, choose
the best Rt from all possible
responses to At
Query=“light laptop”
User U:
System:
A1 A2 … … At-1
R1 R2 … … Rt-1
C
Info Item
Collection
Click on “Next” button
At
Rt =?
The best ranking for the query
The best ranking of unseen items
Rt  r(At)
All possible rankings of items in C
All possible rankings of unseen items
15
Formalization based on Bayesian Decision
Theory : Risk Minimization Framework
[Zhai & Lafferty 06, Shen et al. 05]
Observed
User Model
User:
U
Interaction history: H
Current user action: At
Document collection: C
Seen items
M=(S, U,… )
Information need
All possible responses:
r(At)={r1, …, rn}
L(ri,At,M)
Loss Function
Optimal response: r* (minimum loss)
Rt  arg min rr ( At )  L(r , At , M ) P( M | U , H , At , C )dM
M
Bayes risk
Inferred
Observed
16
A Simplified Two-Step
Decision-Making Procedure
• Approximate the Bayes risk by the loss at the
mode of the posterior distribution
Rt  arg min rr ( At )  L(r , At , M ) P( M | U , H , At , C )dM
M
 arg min rr ( At ) L(r , At , M *) P( M * | U , H , At , C )
 arg min rr ( At ) L(r , At , M *)
where M *  arg max M P( M | U , H , At , C )
• Two-step procedure
– Step 1: Compute an updated user model M* based on
the currently available information
– Step 2: Given M*, choose a response to minimize the
loss function
17
Optimal Interactive Retrieval
User
A1
Many possible actions:
-type in a query character
- scroll down a page
A
- click on any 2button
-…
U
M*1
C
Collection
P(M1|U,H,A1,C)
Many possible responses:
L(r,A1,M*1)
-query completion
R1
-display adaptive summaries
-recommendation/advertising
-clarification
M*2
P(M2|U,H,A
-…2,C)
L(r,A2,M*2)
…
* M (user model) can be regarded
as states in an MDP or POMDP.
R2
A3
Thus reinforcement
learning willIR be
useful
system
(see SIGIR’14 tutorial on dynamic IR modeling [Yang et al. 14])
* Interaction can be modeled at different levels: keyboard input,
result clicking , and query formulations, multisession tasks, …
18
Refinement of Risk Minimization
Framework
• r(At): decision space (At dependent)
–
–
–
–
–
r(At) = all possible rankings of items in C
r(At) = all possible rankings of unseen items
r(At) = all possible summarization strategies
r(At) = all possible ways to diversify top-ranked items
r(At) = all possible ways to mix results with query suggestions (or topic map)
–
–
–
–
Essential component: U = user information need
S = seen items
n = “new topic?” (or “Never purchased such a product before”?)
t = user’s task?
• M: user model
• L(Rt ,At,M): loss function
– Generally measures the utility of Rt for a user modeled as M
– Often encodes relevance criteria, but may also capture other preferences
– Can be based on long-term gain (i.e., “winning the whole “game” of info service)
• P(M|U, H, At, C): user model inference
– Often involves estimating the information need U
– May involve inference of other variables also (e.g., task, exploratory vs. fixed item
search)
19
Case 1: Context-Insensitive IR
–
–
–
–
At=“enter a query Q”
r(At) = all possible rankings of docs in C
M= U, unigram language model (word distribution)
p(M|U,H,At,C)=p(U |Q)
L(ri , At , M )  L((d1 ,..., d N ), U )
N
  p (viewed | d i )D (U ||  di )
i 1
Since p (viewed | d1 )  p (viewed | d 2 )  ....
the optimal ranking Rt is given by ranking documents by D (U ||  di )
20
Optimal Ranking for Independent Loss
 *  arg min

 L( , ) p( | q,U , C, S )d

N
i
i 1
j 1
N
i
L( ,  )   si  l ( j |1... j 1 )
  si  l ( j )
i 1
j 1
N
N  j 1
j 1
i 1
 (

N
 *  arg min   (

 j 1
Sequential browsing
Independent loss
si )l ( j )
N  j 1

i 1
N
N  j 1
j 1
i 1
 arg min  (

Decision space = {rankings}

si )l ( j ) p( | q, U , C , S ) d
si )  l ( j ) p( j | q, U , C , S )d j

r (d k | q, U , C , S )   l ( k ) p ( k | q, U , C , S )d k
Independent risk
= independent scoring

 *  Ranking based on r (d k | q,U , C , S )
“Risk ranking principle”
[Zhai 02, Zhai & Lafferty 06]
21
Case 2: Implicit Feedback
–
–
–
–
–
At=“enter a query Q”
r(At) = all possible rankings of docs in C
M= U, unigram language model (word distribution)
H={previous queries} + {viewed snippets}
p(M|U,H,At,C)=p(U |Q,H)
L(ri , At , M )  L((d1 ,..., d N ), U )
N
  p (viewed | d i )D (U ||  di )
i 1
Since p (viewed | d1 )  p (viewed | d 2 )  ....
the optimal ranking Rt is given by ranking documents by D (U ||  di )
22
Case 3: General Implicit Feedback
–
–
–
–
–
At=“enter a query Q” or “Back” button, “Next” button
r(At) = all possible rankings of unseen docs in C
M= (U, S), S= seen documents
H={previous queries} + {viewed snippets}
p(M|U,H,At,C)=p(U |Q,H)
L(ri , At , M )  L((d1 ,..., d N ), U )
N
  p (viewed | d i )D (U ||  di )
i 1
Since p (viewed | d1 )  p (viewed | d 2 )  ....
the optimal ranking Rt is given by ranking documents by D (U ||  di )
23
Case 4: User-Specific Result Summary
–
–
–
–
At=“enter a query Q”
r(At) = {(D,)}, DC, |D|=k, {“snippet”,”overview”}
M= (U, n), n{0,1} “topic is new to the user”
p(M|U,H,At,C)=p(U, n|Q,H), M*=(*, n*)
L( i , n*)
L(ri , At , M )  L( Di ,  i ,  *, n*)
n*=1 n*=0
 L( Di ,  *)  L( i , n*)

 D( * || 
d Di
Choose k most relevant docs
d
)  L( i , n*)
i=snippet
i=overview
1
0
0
1
If a new topic (n*=1),
give an overview summary;
otherwise, a regular snippet summary
24
Case 5: Modeling Different Notions of
Diversification
• Redundancy reduction  reduce user effort
• Diverse information needs (e.g., overview,
subtopic retrieval)  increase the immediate
utility
• Active relevance feedback  increase future
utility
25
Risk Minimization for Diversification
• Redundancy reduction: Loss function includes a
redundancy measure
– Special case: list presentation + MMR [Zhai et al. 03]
• Diverse information needs: loss function defined
on latent topics
– Special case: PLSA/LDA + topic retrieval [Zhai 02]
• Active relevance feedback: loss function considers
both relevance and benefit for feedback
– Special case: hard queries + feedback only [Shen & Zhai 05]
26
Subtopic Retrieval [Zhai et al. 03]
Query: What are the applications of robotics in the world today?
Find as many DIFFERENT applications as possible.
Example subtopics:
A1: spot-welding robotics
A2: controlling inventory
A3: pipe-laying robots
A4: talking robot
A5: robots for loading & unloading
memory tapes
A6: robot [telephone] operators
A7: robot cranes
……
Subtopic judgments
d1
d2
d3
….
dk
A1 A2 A3 … ...
Ak
1 1 0 0… 0 0
0 1 1 1… 0 0
0 0 0 0… 1 0
1 0 1 0 ... 0 1
This is a non-traditional retrieval task …
27
5.1 Diversify = Remove Redundancy
N  N


 *  arg min  L( , ) p( | q, U , C , S )d  arg min    si  r (d j | d1 ,..., d j1 )


j 1  i  j



r (d k | d1 ,..., d k 1 )   r (d k | d1 ,..., d k 1 ,  ) p ( | q, U , C , S )d

Greedy Algorithm for Ranking: Maximal Marginal Relevance (MMR)
l (d k | d1 ,..., d k 1 ,  Q , { i }ik11 )  c2 p(Re l | d k )(1  p ( New | d k ))  c3 (1  p (Re l | d k ))
Cost
REL
NON-REL
NEW
0
C3
where,  
NOT-NEW
C2
C3
c3
1
c2
Rank


Rank
p(Re l | d k )(1    p( New | d k ))
p(q | d k ) (1    p( New | d k ))
“Willingness to tolerate redundancy”
C2<C3, since a redundant relevant doc is
better than a non-relevant doc
28
5.2 Diversity = Satisfy Diverse Info. Need
[Zhai 02]
• Need to directly model latent aspects and then
optimize results based on aspect/topic matching
• Reducing redundancy doesn’t ensure complete
coverage of diverse aspects
29
Aspect Loss Function: Illustration
perfect
redundant
Desired coverage“Already covered”
p(a|Q)
p(a|1)... p(a|k -1)
non-relevant
New candidate
p(a|k)
Combined coverage
p(a|k)
30
5.3 Diversify = Active Feedback [Shen & Zhai 05]
Decision problem:
Decide subset of documents for relevance judgment
D  arg min  L( D, ) p( | U , q, C)d
*

D


L ( D,  )   l ( D , j ,  ) p ( j | D ,  , U )

j

k
  l ( D, j , ) p ( ji | di , ,U )

j
i 1
31
Independent Loss

k
L( D,  )   l ( D, j ,  ) p( ji | di ,  , U )

i 1
j
Independent Loss
k
l ( D, j, )   l (di , ji ,  )
k
k
L( D,  )   l (di , ji ,  ) p( ji | d i ,  ,U )
i 1
i 1

j
i 1
k
D*  arg min   l (di , ji , ) p( ji | di , ,U ) p( | U , q, C )d
D
i 1
ji

r (di )    l (di , ji , ) p( ji | di ,  ,U ) p( | U , q, C )d
ji

32
Independent Loss (cont.)
r (di )    l (d i , ji ,  ) p ( ji | d i ,  , U ) p ( | U , q, C ) d 
ji

di  C , l (di ,1, )  C1 ,
l (di , 0, )  C0 , C1  C0
l (di ,1, )  log p( R  1| di , ) di  C
l (di ,0, )  log p( R  0 | di , ) di  C
r (di )  C0  (C1  C0 )  p( ji  1| di , ,U ) p( | U , q, C )d

Top K
r (di )    H ( R | di ,  ) p( | U , q, C )d

Uncertainty
Sampling
33
Dependent Loss
k
L( D,U , )   p( ji  1| di ,  ,U )  ( D,  )
i 1
Heuristics: consider relevance
first, then diversity
Select Top N documents
…
N  (G  1) K Cluster N docs into K clusters
K Cluster Centroid
Gapped Top K
MMR
34
Illustration of Three AF Methods
Gapped
Top-K
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
…
Top-K
(normal feedback)
K-cluster centroid
Experiment results show that Top-K is worse
than all others [Shen & Zhai 05]
35
Suggested answers to the basic questions
• Search Engine = Game System
• Optimal Search Engine = Optimal Game Plan/Strategy
• Objective function: based on 3 factors and at the session level
– Utility of information delivered to the user
– Effort needed from the user
– System resource overhead
• How can we solve such an optimization problem?
– Bayesian decision theory in general, partially observable Markov
decision process (POMDP) [Luo et al. 14]
– Reinforcement learning
– ...
36
Major benefits of IR as game playing
• Naturally optimize performance on an entire
session instead of that on a single query
(optimizing the chance of winning the entire
game)
• It optimizes the collaboration of machines and
users (maximizing collective intelligence)
• It opens up many interesting new research
directions (e.g., crowdsourcing + interactive IR)
37
An interesting new problem: Crowdsourcing to users
for relevance judgments collection
• Assumption: Approximate relevance judgments
with clickthroughs
• Question: how to optimize the explorationexploitation tradeoff when leveraging users to
collect clicks on lowly-ranked (“tail”) documents?
– Where to insert a candidate ?
– Which user should get this “assignment” and when?
• Potential solution must include a model for a
user’s behavior
38
General Research Questions Suggested by
the Game-Theoretic Framework
• How should we design an IR game?
– How to design “moves” for the user and the system?
– How to design the objective of the game?
– How to go beyond search to support access and task
completion?
• How to formally define the optimization problem
and compute the optimal strategy for the IR system?
– To what extent can we directly apply existing game
theory? Does Nash equilibrium matter?
– What new challenges must be solved?
• How to evaluate such a system? MOOC?
39
A few specific questions
• How can we support natural interaction via “explanatory feedback”?
– I want documents similar to this one except for not matching “X”
– I want documents similar to this one, but also further matching “Y”
– …
• How can we model a user’s non-topical preferences?
– Readability
– Freshness
– …
•
•
•
•
•
How can we perform syntactic and semantic analysis of queries?
How can we generate adaptive explanatory summaries of documents?
How can we generate coherent preview of search results ?
How can we generate a topic map to enable users to browse freely?
…
40
Intelligent IR System in the Future:
Optimizing multiple games simultaneously
Game 2
Game 1
Learning engine
(MOOC)
Mobile service
search
Intelligent
IR System
Game k
Medical advisor
–Support whole workflow of a user’s task (multimodel
info access, info analysis, decision support, task support)
–Minimize user effort (maximum relevance, natural
dialogue)
–Minimize system resource overhead
–Learn to adapt & improve over time from all users/data
Log
Documents
41
Action Item: future research requires
integration of multiple fields
Psychology
User action
Human-Computer Interactive Service
Game
Theory
(Economics)
(Search,
Browsing,
Recommend…)
Interaction
System response
Document
Collection
Traditional Information Retrieval
User
Understanding
User
Model
Natural Language Processing
Document
Representation
Document
Understanding
Natural Language Processing
Machine Learning
(particularly reinforcement learning)
External User
External Doc
User interaction Log
Info (social network)
Info (structures)
42
References
Note: the references are inevitably incomplete due to
the breadth of the topic;
if you know of any important missing references, please email me at [email protected].
•
•
•
•
•
•
•
[Salton et al. 1975] A theory of term importance in automatic text analysis. G. Salton,
C.S. Yang and C. T. Yu. Journal of the American Society for Information Science, 1975.
[Singhal et al. 1996] Pivoted document length normalization. A. Singhal, C. Buckley and
M. Mitra. SIGIR 1996.
[Maron&Kuhn 1960] On relevance, probabilistic indexing and information retrieval. M. E.
Maron and J. L. Kuhns. Journal o fhte ACM, 1960.
[Harter 1975] A probabilistic approach to automatic keyword indexing. S. P. Harter.
Journal of the American Society for Information Science, 1975.
[Robertson&Sparck Jones 1976] Relevance weighting of search terms. S. Robertson and
K. Sparck Jones. Journal of the American Society for Information Science, 1976.
[van Rijsbergen 1977] A theoretical basis for the use of co-occurrence data in
information retrieval. C. J. van Rijbergen. Journal of Documentation, 1977.
[Robertson 1977] The probability ranking principle in IR. S. E. Robertson. Journal of
Documentation, 1977.
43
References (cont.)
•
•
•
•
•
•
•
•
•
[Robertson 1981] Probabilistic models of indexing and searching. S. E. Robertson, C. J.
van Rijsbergen and M. F. Porter. Information Retrieval Search, 1981.
[Robertson&Walker 1994] Some simple effective approximations to the 2-Poisson model
for probabilistic weighted retrieval. S. E. Robertson and S. Walker. SIGIR 1994.
[Ponte&Croft 1998] A language modeling approach to information retrieval. J. Ponte and
W. B. Croft. SIGIR 1998.
[Hiemstra&Kraaij 1998] Twenty-one at TREC-7: ad-hoc and cross-language track. D.
Hiemstra and W. Kraaij. TREC-7. 1998.
[Zhai&Lafferty 2001] A study of smoothing methods for language models applied to ad
hoc information retrieval. C. Zhai and J. Lafferty. SIGIR 2001.
[Lavrenko&Croft 2001] Relevance-based language models. V. Lavrenko and B. Croft.
SIGIR 2001.
[Kurland&Lee 2004] Corpus structure, language models, and ad hoc information
retrieval. O. Kurland and L. Lee. SIGIR 2004.
[van Rijsbergen 1986] A non-classical logic for information retrieval. C. J. van Rijsbergen.
The Computer Journal, 1986.
[Wong&Yao 1995] On modeling information retrieval with probabilistic inference. S. K.
M. Wong and Y. Y. Yao. ACM Transactions on Information Systems. 1995.
44
References (cont.)
•
•
•
•
•
•
•
•
•
[Amati&van Rijsbergen 2002] Probabilistic models of information retrieval based on
measuring the divergence from randomness. G. Amati and C. J. van Rijsbergen. ACM
Transactions on Information Retrieval. 2002.
[He&Ounis 2005] A study of the dirichlet priors for term frequency normalization. B. He
and I. Ounis. SIGIR 2005.
[Fuhr 89] Norbert Fuhr: Optimal Polynomial Retrieval Functions Based on the Probability
Ranking Principle. ACM Trans. Inf. Syst. 7(3): 183-204 (1989)
[Gey 1994] Inferring probability of relevance using the method of logistic regression. F.
Gey. SIGIR 1994.
[Fang et al. 2004] H. Fang, T. Tao, C. Zhai, A formal study of information retrieval
heuristics. SIGIR 2004.
[Clinchant & Gaussier 2010] Stéphane Clinchant, Éric Gaussier: Information-based
models for ad hoc IR. SIGIR 2010: 234-241
[Fang et al. 2011] H. Fang, T. Tao, C. Zhai, Diagnostic evaluation of information retrieval
models, ACM Transactions on Information Systems, 29(2), 2011
[Zhai & Lafferty 06] ChengXiang Zhai, John D. Lafferty: A risk minimization framework for
information retrieval. Inf. Process. Manage. 42(1): 31-55 (2006)
[Zhai 02] ChengXiang Zhai, Risk Minimization and Language Modeling in Information
Retrieval, Ph.D. thesis, Carnegie Mellon University, 2002.
45
References (cont.)
•
•
•
•
•
•
[Cao et al. 07] Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. 2007.
Learning to rank: from pairwise approach to listwise approach. In Proceedings of the
24th international conference on Machine learning (ICML '07), pp.129-136, 2007
[Fuhr 08] Norbert Fuhr. 2008. A probability ranking principle for interactive information
retrieval. Inf. Retr. 11, 3 (June 2008), 251-265.
[Shen et al. 05] Xuehua Shen, Bin Tan, and ChengXiang Zhai, Implicit User Modeling for
Personalized Search , In Proceedings of the 14th ACM International Conference on
Information and Knowledge Management ( CIKM'05), pages 824-831.
[Zhai et al. 03] ChengXiang Zhai, William W. Cohen, and John Lafferty, Beyond
Independent Relevance: Methods and Evaluation Metrics for Subtopic Retrieval ,
Proceedings of the 26th Annual International ACM SIGIR Conference on Research and
Development in Information Retrieval ( SIGIR'03 ), pages 10-17, 2003.
[Shen & Zhai 05] Xuehua Shen, ChengXiang Zhai, Active Feedback in Ad Hoc
Information Retrieval, Proceedings of the 28th Annual International ACM SIGIR
Conference on Research and Development in Information Retrieval ( SIGIR'05), 59-66,
2005.
[Teevan et al. 10] Jaime Teevan, Susan T. Dumais, Eric Horvitz: Potential for
personalization. ACM Trans. Comput.-Hum. Interact. 17(1) (2010)
46
References (cont.)
•
•
•
•
•
•
[Shen et al. 06] Dou Shen, Jian-Tao Sun, Qiang Yang, and Zheng Chen. 2006. Building
bridges for web query classification. In Proceedings of the 29th annual international
ACM SIGIR 2006, pp. 131-138.
[Wang et al. 13] Hongning Wang, Yang Song, Ming-Wei Chang, Xiaodong He, Ryen W.
White, and Wei Chu. 2013. Learning to extract cross-session search tasks, WWW’ 2013.
1353-1364.
[Agarwal et al. 09] Deepak Agarwal, Bee-Chung Chen, and Pradheep Elango. 2009.
Explore/Exploit Schemes for Web Content Optimization. In Proceedings of the 2009
Ninth IEEE International Conference on Data Mining (ICDM '09), 2009.
[Karimzadehgan & Zhai 13] Maryam Karimzadehgan, ChengXiang Zhai. A Learning
Approach to Optimizing Exploration-Exploitation Tradeoff in Relevance Feedback,
Information Retrieval , 16(3), 307-330, 2013.
[Luo et al. 14] J. Luo, S. Zhang, G. H. Yang, Win-Win Search: Dual-Agent Stochastic
Game in Session Search. ACM SIGIR 2014.
[Yang et al. 14] G. H. Yang, M. Sloan, J. Wang, Dynamic Information Retrieval Modeling,
ACM SIGIR 2014 Tutorial; http://www.slideshare.net/marcCsloan/dynamicinformation-retrieval-tutorial
47
Thank You!
Questions/Comments?
48