Content-Based Image Retrieval Selim Aksoy Department of Computer Engineering Bilkent University [email protected] Image retrieval Searching a large database for images that match a query: What kind of.
Download ReportTranscript Content-Based Image Retrieval Selim Aksoy Department of Computer Engineering Bilkent University [email protected] Image retrieval Searching a large database for images that match a query: What kind of.
Slide 1
Content-Based Image Retrieval
Selim Aksoy
Department of Computer Engineering
Bilkent University
[email protected]
Slide 2
Image retrieval
Searching a large database for images that match
a query:
What kind of databases?
What kind of queries?
What constitutes a match?
How do we make such searches efficient?
CS 484, Fall 2012
©2012, Selim Aksoy
2
Slide 3
Applications
Art Collections
Medical Image Databases
Earth Sciences
General Image Collections for Licensing
CT, MRI, Ultrasound, The Visible Human
Scientific Databases
Fine Arts Museum of San Francisco
Corbis, Getty Images
The World Wide Web
Google, Microsoft, Flickr
CS 484, Fall 2012
©2012, Selim Aksoy
3
Slide 4
Corel data set
60,000 images with annotated keywords
CS 484, Fall 2012
©2012, Selim Aksoy
4
Slide 5
Fine Arts Museum of San Francisco
80,000 images
CS 484, Fall 2012
©2012, Selim Aksoy
5
Slide 6
Query formulation
Text description (keywords)
Query by example
Query by sketch
Symbolic description (man and woman on a beach)
Relevance feedback
CS 484, Fall 2012
©2012, Selim Aksoy
6
Slide 7
Google query on “rose”
CS 484, Fall 2012
©2012, Selim Aksoy
7
Slide 8
Corel query on “rose”
CS 484, Fall 2012
©2012, Selim Aksoy
8
Slide 9
Corbis query on “rose”
CS 484, Fall 2012
©2012, Selim Aksoy
9
Slide 10
Difficulties with keywords
Images may not have keywords.
Query is not easily satisfied by keywords.
(An image is worth … how many key-words?)
“A casually dressed couple gazing into each others eyes
lovingly with dramatic clouds in the background.”
“Pretty girl doing something active, sporty in a
summery setting, beach - not wearing lycra, exercise
clothes - more relaxed in tee-shirt. Feature is about
deodorant so girl should look active - not sweaty but
happy, healthy, carefree - nothing too posed or set up nice and natural looking.”
Content-based image retrieval (CBIR)
CS 484, Fall 2012
©2012, Selim Aksoy
10
Slide 11
Content-based image retrieval
Query Image
Retrieved Images
User
Image Database
Distance Measure
Images
CS 484, Fall 2012
Image Feature
Extraction
©2012, Selim Aksoy
Feature Space
11
Slide 12
Image representations and features
Image representations:
Iconic
Global
Region-based
Object-based
Image features:
Color
Texture
Shape
Objects and their relationships
(this is the most powerful, but you have to be able to
recognize the objects!)
CS 484, Fall 2012
©2012, Selim Aksoy
12
Slide 13
Image similarity
Distance measures:
Euclidean distance
Other Lp metrics
Histogram intersection
Cosine distance
Earth mover’s distance
Probabilistic similarity measures:
P( relevance | two images )
P( relevance | two images ) / P( irrelevance | two images)
CS 484, Fall 2012
©2012, Selim Aksoy
13
Slide 14
Global histograms
Searching using global color histograms
CS 484, Fall 2012
©2012, Selim Aksoy
14
Slide 15
Global histograms
“Airplanes” using color histograms (4/12)
CS 484, Fall 2012
“Sunsets” using Gabor texture (3/12)
©2012, Selim Aksoy
15
Slide 16
Query by image content (QBIC)
CS 484, Fall 2012
©2012, Selim Aksoy
16
Slide 17
Color histograms in QBIC
The QBIC color histogram distance is:
dhist(I,Q) = (h(I) - h(Q))T A (h(I) - h(Q)).
h(I) is a K-bin histogram of a database image.
h(Q) is a K-bin histogram of the query image.
A is a K x K similarity matrix.
R G
R 1 0
G 0 1
B 0 0
Y
C
V
?
CS 484, Fall 2012
B Y C V
0 .5 0 .5
0 .5 .5 0
1
1
1
1
?
How similar is blue to cyan?
©2012, Selim Aksoy
17
Slide 18
Color percentages in QBIC
%40 red, %30 yellow, %10 black
CS 484, Fall 2012
©2012, Selim Aksoy
18
Slide 19
Color layout in QBIC
CS 484, Fall 2012
©2012, Selim Aksoy
19
Slide 20
Earth mover’s distance
CS 484, Fall 2012
©2012, Selim Aksoy
20
Slide 21
Earth mover’s distance
Visualization using EMD and multidimensional
scaling
CS 484, Fall 2012
©2012, Selim Aksoy
21
Slide 22
Probabilistic similarity measures
Two classes:
Bayes classifier
Relevance class A
Irrelevance class B
Assign (i,j) to
Discriminant function for classification
Δ( ξ i , ξ j )
Α if P(A | ( ξ i , ξ j )) P(B | ( ξ i , ξ j ))
B otherwise
P(A | ( ξ i , ξ j ))
P(B | ( ξ i , ξ j ))
P(( ξ i , ξ j ) | A) P(A)
P(( ξ i , ξ j ) | B) P(B)
Rank images according to posterior ratio values
based on feature differences.
CS 484, Fall 2012
©2012, Selim Aksoy
22
Slide 23
Probabilistic similarity measures
“Residential interiors” (12/12)
CS 484, Fall 2012
“Fields” (12/12)
©2012, Selim Aksoy
23
Slide 24
Shape-based retrieval
CS 484, Fall 2012
©2012, Selim Aksoy
24
Slide 25
Shape-based retrieval
CS 484, Fall 2012
©2012, Selim Aksoy
25
Slide 26
Elastic shape matching
CS 484, Fall 2012
©2012, Selim Aksoy
26
Slide 27
Iconic matching
CS 484, Fall 2012
©2012, Selim Aksoy
27
Slide 28
Iconic matching
Wavelet-based image compression
Quantization of coefficients
CS 484, Fall 2012
©2012, Selim Aksoy
28
Slide 29
Iconic matching
CS 484, Fall 2012
©2012, Selim Aksoy
29
Slide 30
Region-based retrieval: Blobworld
CS 484, Fall 2012
©2012, Selim Aksoy
30
Slide 31
Region-based retrieval: Blobworld
CS 484, Fall 2012
©2012, Selim Aksoy
31
Slide 32
Retrieval using spatial relationships
Build graph using regions and their
spatial relationships.
Similarity is computed using graph
matching.
sky
image
above
adjacent
above
tiger
inside
above
adjacent
above
abstract regions
CS 484, Fall 2012
grass
sand
©2012, Selim Aksoy
32
Slide 33
Combining multiple features
Text query
on
“rose”
CS 484, Fall 2012
©2012, Selim Aksoy
33
Slide 34
Combining multiple features
Visual query
on
CS 484, Fall 2012
©2012, Selim Aksoy
34
Slide 35
Combining multiple features
Text query
on
“rose”
and visual query
on
CS 484, Fall 2012
©2012, Selim Aksoy
35
Slide 36
Video Google: object matching
CS 484, Fall 2012
©2012, Selim Aksoy
36
Slide 37
Video Google
Viewpoint invariant
descriptors
Visual vocabulary
CS 484, Fall 2012
©2012, Selim Aksoy
37
Slide 38
Video Google
Inverted index
Document 1
Now is the time
for all good men
to come to the aid
of their country.
Word
aid
1
all
1
and
2
can
2
come
Document 2
Summer has come
and passed. The innocent
can never last.
©2012, Selim Aksoy
1, 2
country
1
for
1
good
1
…
the
…
CS 484, Fall 2012
Document
…
1, 1, 2
…
38
Slide 39
Video skimming
CS 484, Fall 2012
©2012, Selim Aksoy
39
Slide 40
Event detection, indexing, retrieval
CS 484, Fall 2012
©2012, Selim Aksoy
40
Slide 41
Informedia Digital Video Library
IDVL interface returned for "El Nino" query along with
different multimedia abstractions from certain documents.
CS 484, Fall 2012
©2012, Selim Aksoy
41
Slide 42
Informedia Digital Video Library
IDVL interface returned
for “bin ladin" query.
The results can be tuned
using many classifiers.
CS 484, Fall 2012
©2012, Selim Aksoy
42
Slide 43
Relevance feedback
In real interactive CBIR systems, the user should
be allowed to interact with the system to “refine”
the results of a query until he/she is satisfied.
CS 484, Fall 2012
©2012, Selim Aksoy
43
Slide 44
Relevance feedback
Example methods:
Query point movement
Weighting features
Query point is moved toward positive examples and moved
away from negative examples.
The CBIR system should automatically adjust the weight that
were given by the user for the relevance of previously retrieved
documents.
Weighting similarity measures
Feature density estimation
Probabilistic relevance feedback
CS 484, Fall 2012
©2012, Selim Aksoy
44
Slide 45
Relevance feedback
Positive feedback
,)
,p(
ξ) |(n)
A)p(A)
)p(|A)p(A
p(
ξ| A)p(A
|| A)p(A
(0) |) (0)| ξ, (0)(1), ξ) (1) , , ξ (n- 1) )
p(A | ξ (0) ,)ξ(1) p(
(2)(0)
(1)
(2) (n)
,)
,p(
ξ) |(n)
B)p(B)
)p(|B)p(B
p(
ξ| B)p(B
| B)p(B
(0) |) (0)| ξ, (0)(1), ξ) (1) , , ξ (n- 1) )
p(B | ξ (0) ,)ξ(1) p(
(2)(0)
(1)
(2) (n)
Negative feedback
, , ξ (n) ,,ξ((1)
, )
(2), ξ),(m)
(3)) )
p(A | ξ ((0)0 ) , ξ ((1)
1 ) , , (n)
1) ,
p( ξ (m)
B)p(A ||ξ(0)
, ξ (1 ) , , ξ (n) ,)ξ ((1)
,,
||B)p(A
(2), ξ) (m- 1) )
( 0 ), (1)
1) )
(3)
(2)
(1)
,,ξ(n)
(2), ξ),(m)
(3)) )
p(B | ξ ((0)0 ) , ξ ((1)
,,
,,ξ((1)
, ,)
(n)
1)
1)
p( ξ (m)
| A)p(B | ( 0 ),,ξ(1)
, , ξ (n) ,)ξ ((1)
(2), ξ) (m- 1) )
,),
(3)
(2)
(1) | A)p(B | ξ(0)
(1 )
1)
CS 484, Fall 2012
©2012, Selim Aksoy
45
Slide 46
Relevance feedback
“Sunsets” using color histograms
(1/12)
CS 484, Fall 2012
Using combined features (6/12)
©2012, Selim Aksoy
After 1st feedback (12/12)
46
Slide 47
Relevance feedback
“Auto racing” using color
histograms (3/12)
CS 484, Fall 2012
Using combined features (9/12)
©2012, Selim Aksoy
After 1st feedback (12/12)
47
Slide 48
Indexing for fast retrieval
Use of key images and the triangle inequality for
efficient retrieval.
CS 484, Fall 2012
©2012, Selim Aksoy
48
Slide 49
Indexing for fast retrieval
Offline
1. Choose a small set of key images.
2. Store distances from database images to keys.
Online (given query Q)
1. Compute the distance from Q to each key.
2. Obtain lower bounds on distances to database images.
3. Threshold or return all images in order of lower
bounds.
CS 484, Fall 2012
©2012, Selim Aksoy
49
Slide 50
Indexing for fast retrieval
Hierarchical cellular tree
CS 484, Fall 2012
©2012, Selim Aksoy
50
Slide 51
Indexing for fast retrieval
CS 484, Fall 2012
©2012, Selim Aksoy
51
Slide 52
Performance evaluation
Two traditional measures for retrieval
performance in the information retrieval literature
are precision and recall.
Given a particular number of images retrieved,
precision is defined as the percentage of retrieved
images that are actually relevant, and
recall is defined as the percentage of relevant images
that are retrieved.
CS 484, Fall 2012
©2012, Selim Aksoy
52
Slide 53
Current research objective
Query Image
Retrieved Images
boat
User
Image Database
…
Animals
Buildings
Office Buildings
Houses
Transportation
•Boats
•Vehicles
Images
CS 484, Fall 2012
Object-oriented
Feature Extraction
©2012, Selim Aksoy
…
Categories
53
Slide 54
Demos
Blobworld (http://elib.cs.berkeley.edu/blobworld/)
Video Google (http://www.robots.ox.ac.uk/~vgg/
research/vgoogle/index.html)
FIDS (http://www.cs.washington.edu/research/
imagedatabase/demo/fids/)
Like Visual Shopping (http://www.like.com/)
Google Image Search (http://images.google.com/)
Yahoo Image Search
(http://images.search.yahoo.com/)
Flickr (http://flickr.com/)
The ESP game (http://www.espgame.org/)
CS 484, Fall 2012
©2012, Selim Aksoy
54
Slide 55
Demos
Vitalas
Google Similar Images
http://googleblog.blogspot.com/2009/10/similarimages-graduates-from-google.html
Google Image Swirl
http://vitalas.ercim.eu/
http://googleresearch.blogspot.com/2009/11/exploreimages-with-google-image-swirl.html
Microsoft Bing
http://www.bing.com/
First use keywords, then mouse over an image and click
on show similar images
CS 484, Fall 2012
©2012, Selim Aksoy
55
Content-Based Image Retrieval
Selim Aksoy
Department of Computer Engineering
Bilkent University
[email protected]
Slide 2
Image retrieval
Searching a large database for images that match
a query:
What kind of databases?
What kind of queries?
What constitutes a match?
How do we make such searches efficient?
CS 484, Fall 2012
©2012, Selim Aksoy
2
Slide 3
Applications
Art Collections
Medical Image Databases
Earth Sciences
General Image Collections for Licensing
CT, MRI, Ultrasound, The Visible Human
Scientific Databases
Fine Arts Museum of San Francisco
Corbis, Getty Images
The World Wide Web
Google, Microsoft, Flickr
CS 484, Fall 2012
©2012, Selim Aksoy
3
Slide 4
Corel data set
60,000 images with annotated keywords
CS 484, Fall 2012
©2012, Selim Aksoy
4
Slide 5
Fine Arts Museum of San Francisco
80,000 images
CS 484, Fall 2012
©2012, Selim Aksoy
5
Slide 6
Query formulation
Text description (keywords)
Query by example
Query by sketch
Symbolic description (man and woman on a beach)
Relevance feedback
CS 484, Fall 2012
©2012, Selim Aksoy
6
Slide 7
Google query on “rose”
CS 484, Fall 2012
©2012, Selim Aksoy
7
Slide 8
Corel query on “rose”
CS 484, Fall 2012
©2012, Selim Aksoy
8
Slide 9
Corbis query on “rose”
CS 484, Fall 2012
©2012, Selim Aksoy
9
Slide 10
Difficulties with keywords
Images may not have keywords.
Query is not easily satisfied by keywords.
(An image is worth … how many key-words?)
“A casually dressed couple gazing into each others eyes
lovingly with dramatic clouds in the background.”
“Pretty girl doing something active, sporty in a
summery setting, beach - not wearing lycra, exercise
clothes - more relaxed in tee-shirt. Feature is about
deodorant so girl should look active - not sweaty but
happy, healthy, carefree - nothing too posed or set up nice and natural looking.”
Content-based image retrieval (CBIR)
CS 484, Fall 2012
©2012, Selim Aksoy
10
Slide 11
Content-based image retrieval
Query Image
Retrieved Images
User
Image Database
Distance Measure
Images
CS 484, Fall 2012
Image Feature
Extraction
©2012, Selim Aksoy
Feature Space
11
Slide 12
Image representations and features
Image representations:
Iconic
Global
Region-based
Object-based
Image features:
Color
Texture
Shape
Objects and their relationships
(this is the most powerful, but you have to be able to
recognize the objects!)
CS 484, Fall 2012
©2012, Selim Aksoy
12
Slide 13
Image similarity
Distance measures:
Euclidean distance
Other Lp metrics
Histogram intersection
Cosine distance
Earth mover’s distance
Probabilistic similarity measures:
P( relevance | two images )
P( relevance | two images ) / P( irrelevance | two images)
CS 484, Fall 2012
©2012, Selim Aksoy
13
Slide 14
Global histograms
Searching using global color histograms
CS 484, Fall 2012
©2012, Selim Aksoy
14
Slide 15
Global histograms
“Airplanes” using color histograms (4/12)
CS 484, Fall 2012
“Sunsets” using Gabor texture (3/12)
©2012, Selim Aksoy
15
Slide 16
Query by image content (QBIC)
CS 484, Fall 2012
©2012, Selim Aksoy
16
Slide 17
Color histograms in QBIC
The QBIC color histogram distance is:
dhist(I,Q) = (h(I) - h(Q))T A (h(I) - h(Q)).
h(I) is a K-bin histogram of a database image.
h(Q) is a K-bin histogram of the query image.
A is a K x K similarity matrix.
R G
R 1 0
G 0 1
B 0 0
Y
C
V
?
CS 484, Fall 2012
B Y C V
0 .5 0 .5
0 .5 .5 0
1
1
1
1
?
How similar is blue to cyan?
©2012, Selim Aksoy
17
Slide 18
Color percentages in QBIC
%40 red, %30 yellow, %10 black
CS 484, Fall 2012
©2012, Selim Aksoy
18
Slide 19
Color layout in QBIC
CS 484, Fall 2012
©2012, Selim Aksoy
19
Slide 20
Earth mover’s distance
CS 484, Fall 2012
©2012, Selim Aksoy
20
Slide 21
Earth mover’s distance
Visualization using EMD and multidimensional
scaling
CS 484, Fall 2012
©2012, Selim Aksoy
21
Slide 22
Probabilistic similarity measures
Two classes:
Bayes classifier
Relevance class A
Irrelevance class B
Assign (i,j) to
Discriminant function for classification
Δ( ξ i , ξ j )
Α if P(A | ( ξ i , ξ j )) P(B | ( ξ i , ξ j ))
B otherwise
P(A | ( ξ i , ξ j ))
P(B | ( ξ i , ξ j ))
P(( ξ i , ξ j ) | A) P(A)
P(( ξ i , ξ j ) | B) P(B)
Rank images according to posterior ratio values
based on feature differences.
CS 484, Fall 2012
©2012, Selim Aksoy
22
Slide 23
Probabilistic similarity measures
“Residential interiors” (12/12)
CS 484, Fall 2012
“Fields” (12/12)
©2012, Selim Aksoy
23
Slide 24
Shape-based retrieval
CS 484, Fall 2012
©2012, Selim Aksoy
24
Slide 25
Shape-based retrieval
CS 484, Fall 2012
©2012, Selim Aksoy
25
Slide 26
Elastic shape matching
CS 484, Fall 2012
©2012, Selim Aksoy
26
Slide 27
Iconic matching
CS 484, Fall 2012
©2012, Selim Aksoy
27
Slide 28
Iconic matching
Wavelet-based image compression
Quantization of coefficients
CS 484, Fall 2012
©2012, Selim Aksoy
28
Slide 29
Iconic matching
CS 484, Fall 2012
©2012, Selim Aksoy
29
Slide 30
Region-based retrieval: Blobworld
CS 484, Fall 2012
©2012, Selim Aksoy
30
Slide 31
Region-based retrieval: Blobworld
CS 484, Fall 2012
©2012, Selim Aksoy
31
Slide 32
Retrieval using spatial relationships
Build graph using regions and their
spatial relationships.
Similarity is computed using graph
matching.
sky
image
above
adjacent
above
tiger
inside
above
adjacent
above
abstract regions
CS 484, Fall 2012
grass
sand
©2012, Selim Aksoy
32
Slide 33
Combining multiple features
Text query
on
“rose”
CS 484, Fall 2012
©2012, Selim Aksoy
33
Slide 34
Combining multiple features
Visual query
on
CS 484, Fall 2012
©2012, Selim Aksoy
34
Slide 35
Combining multiple features
Text query
on
“rose”
and visual query
on
CS 484, Fall 2012
©2012, Selim Aksoy
35
Slide 36
Video Google: object matching
CS 484, Fall 2012
©2012, Selim Aksoy
36
Slide 37
Video Google
Viewpoint invariant
descriptors
Visual vocabulary
CS 484, Fall 2012
©2012, Selim Aksoy
37
Slide 38
Video Google
Inverted index
Document 1
Now is the time
for all good men
to come to the aid
of their country.
Word
aid
1
all
1
and
2
can
2
come
Document 2
Summer has come
and passed. The innocent
can never last.
©2012, Selim Aksoy
1, 2
country
1
for
1
good
1
…
the
…
CS 484, Fall 2012
Document
…
1, 1, 2
…
38
Slide 39
Video skimming
CS 484, Fall 2012
©2012, Selim Aksoy
39
Slide 40
Event detection, indexing, retrieval
CS 484, Fall 2012
©2012, Selim Aksoy
40
Slide 41
Informedia Digital Video Library
IDVL interface returned for "El Nino" query along with
different multimedia abstractions from certain documents.
CS 484, Fall 2012
©2012, Selim Aksoy
41
Slide 42
Informedia Digital Video Library
IDVL interface returned
for “bin ladin" query.
The results can be tuned
using many classifiers.
CS 484, Fall 2012
©2012, Selim Aksoy
42
Slide 43
Relevance feedback
In real interactive CBIR systems, the user should
be allowed to interact with the system to “refine”
the results of a query until he/she is satisfied.
CS 484, Fall 2012
©2012, Selim Aksoy
43
Slide 44
Relevance feedback
Example methods:
Query point movement
Weighting features
Query point is moved toward positive examples and moved
away from negative examples.
The CBIR system should automatically adjust the weight that
were given by the user for the relevance of previously retrieved
documents.
Weighting similarity measures
Feature density estimation
Probabilistic relevance feedback
CS 484, Fall 2012
©2012, Selim Aksoy
44
Slide 45
Relevance feedback
Positive feedback
,)
,p(
ξ) |(n)
A)p(A)
)p(|A)p(A
p(
ξ| A)p(A
|| A)p(A
(0) |) (0)| ξ, (0)(1), ξ) (1) , , ξ (n- 1) )
p(A | ξ (0) ,)ξ(1) p(
(2)(0)
(1)
(2) (n)
,)
,p(
ξ) |(n)
B)p(B)
)p(|B)p(B
p(
ξ| B)p(B
| B)p(B
(0) |) (0)| ξ, (0)(1), ξ) (1) , , ξ (n- 1) )
p(B | ξ (0) ,)ξ(1) p(
(2)(0)
(1)
(2) (n)
Negative feedback
, , ξ (n) ,,ξ((1)
, )
(2), ξ),(m)
(3)) )
p(A | ξ ((0)0 ) , ξ ((1)
1 ) , , (n)
1) ,
p( ξ (m)
B)p(A ||ξ(0)
, ξ (1 ) , , ξ (n) ,)ξ ((1)
,,
||B)p(A
(2), ξ) (m- 1) )
( 0 ), (1)
1) )
(3)
(2)
(1)
,,ξ(n)
(2), ξ),(m)
(3)) )
p(B | ξ ((0)0 ) , ξ ((1)
,,
,,ξ((1)
, ,)
(n)
1)
1)
p( ξ (m)
| A)p(B | ( 0 ),,ξ(1)
, , ξ (n) ,)ξ ((1)
(2), ξ) (m- 1) )
,),
(3)
(2)
(1) | A)p(B | ξ(0)
(1 )
1)
CS 484, Fall 2012
©2012, Selim Aksoy
45
Slide 46
Relevance feedback
“Sunsets” using color histograms
(1/12)
CS 484, Fall 2012
Using combined features (6/12)
©2012, Selim Aksoy
After 1st feedback (12/12)
46
Slide 47
Relevance feedback
“Auto racing” using color
histograms (3/12)
CS 484, Fall 2012
Using combined features (9/12)
©2012, Selim Aksoy
After 1st feedback (12/12)
47
Slide 48
Indexing for fast retrieval
Use of key images and the triangle inequality for
efficient retrieval.
CS 484, Fall 2012
©2012, Selim Aksoy
48
Slide 49
Indexing for fast retrieval
Offline
1. Choose a small set of key images.
2. Store distances from database images to keys.
Online (given query Q)
1. Compute the distance from Q to each key.
2. Obtain lower bounds on distances to database images.
3. Threshold or return all images in order of lower
bounds.
CS 484, Fall 2012
©2012, Selim Aksoy
49
Slide 50
Indexing for fast retrieval
Hierarchical cellular tree
CS 484, Fall 2012
©2012, Selim Aksoy
50
Slide 51
Indexing for fast retrieval
CS 484, Fall 2012
©2012, Selim Aksoy
51
Slide 52
Performance evaluation
Two traditional measures for retrieval
performance in the information retrieval literature
are precision and recall.
Given a particular number of images retrieved,
precision is defined as the percentage of retrieved
images that are actually relevant, and
recall is defined as the percentage of relevant images
that are retrieved.
CS 484, Fall 2012
©2012, Selim Aksoy
52
Slide 53
Current research objective
Query Image
Retrieved Images
boat
User
Image Database
…
Animals
Buildings
Office Buildings
Houses
Transportation
•Boats
•Vehicles
Images
CS 484, Fall 2012
Object-oriented
Feature Extraction
©2012, Selim Aksoy
…
Categories
53
Slide 54
Demos
Blobworld (http://elib.cs.berkeley.edu/blobworld/)
Video Google (http://www.robots.ox.ac.uk/~vgg/
research/vgoogle/index.html)
FIDS (http://www.cs.washington.edu/research/
imagedatabase/demo/fids/)
Like Visual Shopping (http://www.like.com/)
Google Image Search (http://images.google.com/)
Yahoo Image Search
(http://images.search.yahoo.com/)
Flickr (http://flickr.com/)
The ESP game (http://www.espgame.org/)
CS 484, Fall 2012
©2012, Selim Aksoy
54
Slide 55
Demos
Vitalas
Google Similar Images
http://googleblog.blogspot.com/2009/10/similarimages-graduates-from-google.html
Google Image Swirl
http://vitalas.ercim.eu/
http://googleresearch.blogspot.com/2009/11/exploreimages-with-google-image-swirl.html
Microsoft Bing
http://www.bing.com/
First use keywords, then mouse over an image and click
on show similar images
CS 484, Fall 2012
©2012, Selim Aksoy
55