Transcript Click Evidence Signals and Tasks

Click Evidence
Signals and Tasks
Vishwa Vinay
Microsoft Research, Cambridge
Introduction
• Signals
– Explicit Vs Implicit
• Evidence
– Of what?
– From where?
– Used how?
• Tasks
– Ranking, Evaluation & many more things search
Clicks as Input
• Task = Relevance Ranking
– Feature in relevance ranking function
• Signal
– select URL, count(*) as DocFeature
from Historical_Clicks group by URL
– select Query, URL, count(*) as QueryDocFeature
from Historical_Clicks group by Query, URL
Clicks as Input
• Feature in relevance ranking function
• Static feature (popularity)
• Dynamic feature (for this query-doc pair)
– “Query Expansion using Associated Queries”, Billerbeck et al,
CIKM 2003
– “Improving Web Search Ranking by Incorporating User
Behaviour”, Agichtein et al, SIGIR 2006
– ‘Document Expansion’
– Signal bleeds to similar queries
Clicks as Output
• Task = Relevance Ranking
– Result Page = Ranked list of documents
– Ranked list = Documents sorted based on Score
– Score = Probability that this result will be clicked
• Signal
– Did my prediction agree with the user’s action?
– “Web-Scale Bayesian Click-through rate Prediction for
Sponsored Search Advertising in Microsoft’s Bing Search
Engine”, Graepel et al, ICML 2010
Clicks as Output
• Calibration: Merging results from different
sources (comparable scores)
– “Adaptation of Offline Vertical Selection Predictions in the
Presence of User Feedback”, Diaz et al, SIGIR 2009
• Onsite Adaptation of ranking function
– “A Decision Theoretic Framework for Ranking using Implicit
Feedback”, Zoeter et al, SIGIR 2008
Clicks for Training
• Task = Learning a ranking function
• Signal
Query=“Search Solutions 2010”
Rank 1
Doc1
http://irsg.bcs.org/SearchSolutions/2010/sse2010.php
Rank 2
Doc2
http://irsg.bcs.org/SearchSolutions/2009/sse2009.php
Rank 3
Doc3
http://isquared.wordpress.com/2010/10/10/searchsolutions-2010-titles-and-abstracts/
Absolute: Relevant={Doc1, Doc3}, NotRelevant={Doc2}
Preferences: {Doc2 ≺ Doc1}, {Doc2 ≺ Doc3}
Clicks for Training
• Preferences from Query-> {URL, Click} events
– Rank bias & Lock-in
• Randomisation & Exploration
– “Accurately Interpreting Clickthrough Data as Implicit Feedback”,
Joachims et al, SIGIR 2005
• Preference Observations into Relevance Labels
– “Generating Labels from Clicks”, Agrawal et al, WSDM 2010
Clicks for Evaluation
• Task = Evaluating a ranking function
• Signal
– Engagement and Usage metrics
Query=“Search Solutions 2010”
Rank
Old Ranker
New (and Improved?)
1
http://irsg.bcs.org/SearchSolutions/2009/sse
2009.php
http://irsg.bcs.org/SearchSolutions/2010/
sse2010.php
2
http://www.onlineinformation.co.uk/online2010/trails/searchsolutions.html
http://isquared.wordpress.com/2010/10/
10/search-solutions-2010-titles-andabstracts/
3
http://irsg.bcs.org/SearchSolutions/2010/sse
2010.php
http://irsg.bcs.org/SearchSolutions/2009/
sse2009.php
Controlled experiments for A/B Testing
Clicks for Evaluation
• Disentangling relevance from other effects
– “An experimental comparison of click position-bias models”,
Craswell et al, WSDM 2008
• Label-free evaluation of retrieval systems
(‘Interleaving’)
– “How Does Clickthrough Data Reflect Retrieval Quality?”,
Radlinski et al, CIKM 2008
Personalisation with Clicks
• Task = Separate out Individual preferences
from aggregates
• Signal : {User, Query, URL, Click} tuples
Query=“Search Solutions 2010”
Rank
URL
1
http://irsg.bcs.org/SearchSolutions/2010/sse2010.php
2
http://isquared.wordpress.com/2010/10/10/searchsolutions-2010-titles-and-abstracts/
3
http://irsg.bcs.org/SearchSolutions/2009/sse2009.php
Tony
Vinay
Personalisation with Clicks
• Click event as a rating
– “Matchbox: Large Scale Bayesian Recommendations”,
Stern et al, WWW 2009
• Sparsity
- collapse using user groups (groupisation)
“Discovering and Using Groups to Improve Personalized
Search”, Teevan et al, WSDM 2009
- collapse using doc structure
Miscellaneous
• Using co-clicking for query suggestions
– “Random Walks on the Click Graph”, Craswell et al, SIGIR 2007
• User behaviour models for
– Ranked lists: “Click chain model in Web Search”, Guo et al,
WWW 2009
– Whole page: “Inferring Search Behaviors Using Partially
Observable Markov Model”, Wang et al, WSDM 2010
• User activity away from the result page
– “BrowseRank: Letting Web Users Vote for Page Importance”, Liu
et al, SIGIR 2008
Additional Thoughts
• Impressions & Examinations
– Raw click counts versus normalised ratios
Query=“Search Solutions 2010”
Page / Rank
URL
Impression
Examination
1/1
http://irsg.bcs.org/SearchSolutions/2
010/sse2010.php
1
1
1/2
http://www.onlineinformation.co.uk/online2010/trails/s
earch-solutions.html
1
1
1/3
http://isquared.wordpress.com/2010
/10/10/search-solutions-2010-titlesand-abstracts/
1
1
1/4
http://irsg.bcs.org/SearchSolutions/2
009/sse2009.php
1
0?
…
…
…
…
http://somesite.org/irrelevant.htm
1
0
http://someothersite.org/alsoirreleva
nt.htm
0
0
1 / 10
2/1
• All clicks are not created equal
- Skip ≺ Click ≺ LastClick ≺ OnlyClick
Clicks and Enterprise Search
• Relying on the click signal
– Machine learning and non-click features
– Performance Out-Of-the-Box
– Shipping a shrink-wrapped product
• The self-aware adapting system
– Good OOB
– Gets better with use
– Knows when things go wrong
Thank you
[email protected]