Transcript Presentation (PPTX)

Time-Sensitive Web Image Ranking
and Retrieval via Dynamic Multi-Task
Regression
Gunhee Kim
Eric P. Xing
School of Computer Science,
Carnegie Mellon University
February 6, 2013
1
Image Ranking and Retrieval
Goal: Find the images for a given query
ex. Cardinal
Text-based image retrieval
2
Image Ranking and Retrieval
Goal: Find the images for a given query
ex. Cardinal
Text-based image retrieval
Url
http://www.allaboutbirds.org/
guide/Northern_Cardinal/id
File name
northern_cardinal_glamour.jpg
• Scalable and
successful so far
• Ambiguity and
noise due to
3
mismatch.
Recent Image Ranking and Retrieval
Various efforts to improve text-based image search
User relevance feedback [Wang et al. CVPR 11]
Text-based search by apple
Reranking on visual features
chosen by a user
Pseudo-relevance feedback
[Liu et al. CVPR 11]
Human labeled training data
[Yang et al. MM10]
Image click data
[Jain et al. WWW11]
4
Time-Sensitive Image Ranking and Retrieval
Discovery of temporal patterns of Web image collections
[Related work] Exploring temporal dynamics of Web queries
• Popular search keywords and relevant documents change over time.
• ex) Keyword search, Product search, News recommendation
No previous work using temporal info on image retrieval
From experiments of 7.5 millions of Flickr images of 30 topics
we found three good reasons …
• [D08] Dakka et al. CIKM 2008
• [V11] Amodeo et al, CIKM2011
• [M09] Metzler et al. SIGIR 2009 • [R12] Radinsky et al, WWW 2012
• [K10] Kulkani et al, WSDM 2011 • …..
5
Why Time-Sensitive Image Retrieval? (1/3)
1. Knowing when search takes place is useful to infer users'
implicit intents.
Google
Bing
Cardinal: (1) the red bird in America.
(3) St. Louis cardinals (baseball)
(2) Arizona cardinals (football)
Fall to Winter
(Sep. ~ Feb.)
Spring to Fall
(Mar. ~ Oct.)
• Severely redundant. Almost identical all year long.
6
Why Time-Sensitive Image Retrieval? (1/3)
1. Knowing when search takes place is useful to infer users'
implicit intents.
Google
Bing
• Diversity can make search interesting.
at Feb. 7, 2009
Our
at May 4, 2009
results
Football
baseball
7
Why Time-Sensitive Image Retrieval? (2/3)
2. Timing suitability can be used as a complementary
attribute to relevance.
Google
Bing
• There are so many almost equally good images.
Background: snow
at Feb. 7, 2009
Our
at May 4, 2009
results
Background: Green Baby birds or eggs
8
Why Time-Sensitive Image Retrieval? (3/3)
3. Temporal information is synergetic in personalized
image retrieval.
At Nov. 7, 2009 for user 30033302
Louisville Men's College Basketball
Each user’ term usages are relatively stationary, and
predictable once they are learned.
9
Algorithm
Regularized multi-task regression on multivariate point process
• Goal: Scalably learn temporal models for each topic keyword.
• Multi-task framework: allows multiple image descriptors.
• Several regularization schemes
• Personalization by locally-weighted learning
10
Thank you !
Stop by our poster!
11
Multivariate Point Process Models
Given a stream of hornet pictures up to T
Time
t1
t2
t3
t5
t6
Clustering by descriptor 1
t7
t9
t10
Clustering by descriptor 2
v12
v13
v
1
1
1st descriptor (v1)
v12
v22
v
2
4
v32
2nd descriptor (v2)
12
Regularized GLM on Point Processes
Given a stream of hornet pictures up to T
Time
t1
(v1,v2) (3, 2)
t2
(3, 2)
t3
(3, 2)
t5
t6
(2, 1)
(1, 3)
t7
(1, 4)
t9
(2, 3)
t10
(2, 1)
Formulate a regression between occurrence rates and covariates.
(
= exp (q
rate(l ) of v1 = exp q11 × f1 (factor1)+q 21 × f2 (factor2)+
rate(l ) of v 2
2
1
× f1 (factor1)+q 22 × f2 (factor2)+
)
)
Covariates: any likely factors to be associated with image occurrence
(ex. Time, season, and other external events)
Compute sparse regularized MLE solutions
Q = {qij }
For each visual cluster, we select only a small number of strong factors.
13
A Toy Example of Image Reranking
Covariates: only year and months
log l i (tk | q i ) = q 0i +
2009
å q I (t ) + åq
i
y y
y=2003
(Sea tour)
(Ice hockey)
(Aquarium)
12
k
m=1
i
m m
I (tk )
Peaked in summer
Peaked in January
Stationary
14