Transcript Quality Adaptive Video Caching and Transport

Outline

Content-Based Image Retrieval



Query-by-Example
Query-by-Feature
Feature Vector
CBIR and CBR



Content-based Image Retrieval (CBIR) como
exemplo de Content-based Retrieval (CBR)
concentra em low-level features.
Principais idéias de CBIR:



Representar uma imagem como um conjunto de feature
descriptors.
Definir medidas de similaridade dos descritores
Quando um usuário especificar uma query, o sistema
retorna imagens, que são ordenadas por similaridade.
CBIR Architecture
Image
data
Similarity
Metric
query
User Interface
Feature
Extraction
Image
Browsing
Database
Database
Creation
Image data
Image Representation
Query
Comparison
Image Retrieval
Database Images
Feature
Extraction
Query Image
Feature
Extraction
Select
Compare
Feature Vectors
4
Query Result
CBIR de Butterflies


Permitir non-expert users encontrar
algumas espécies de butterflies usando
informações de aparência de butterflies
Aparência:

Color, Texture, Shape
Problemas


Como podemos descrever uma
butterfly?
Como podemos comunicar nossa
descrição para uma máquina?
Problemas




Usuários diferentes têm percepções
diferentes.
Usuários podem não se lembrar
claramente a aparêcia de uma butterfly.
Usuários normalmente não têm
expertise para descrever butterflies.
Usuários normalmente não têm
paciência para fazer o browse num
grande conjunto resultado.
Soluções

Usar um processo de consulta interativo e
direcionado ao usuário: QBF/QBE query
process




Query By Features e Query By Example
Fuzzy feature description para cada butterfly
Uma “What You See Is What You Get” query
interface
Um conjunto representativo de coleção
butterflies
QBF/QBE query process (1)

QBF query:

A QBF query is to choose some features
of butterflies and expect that the system
returns all butterflies with those features.

Features of butterflies:


Dominant color, texture pattern, shape.
QBE query:

A QBE query is to point an image and
expect that the system returns all
butterflies similar to that.
QBF/QBE query process (2)

Properties of QBF:


Rough search
When to use:


The first query and when users want to enlarge
the view in the search space
Properties of QBE:


Fine search
When to use:

Usually the last query and when users want to
see the neighbors of the query one in the
search space.
QBF/QBE query process (3)

Result page: (Each result page should contain two parts)

Result Images:
These are the butterfly images satisfy the query
conditions.
 Users can invoke QBE queries from these images.


Related Features:
These are the features related to the previous query
conditions.
 Users can invoke QBF queries from these features.

Feature Description (1)

Feature Description for a butterfly:




Like metadata which describe the appearance
of this butterfly.
This makes QBF queries possible.
Feature Description consists of some feature
descriptors.
Feature descriptor:

A ( “feature value” , “match level” ) pair.
Feature Description (2)
Feature
Type
Feature Value
mixed_with_black_and_orange
Color orange_yellow
orange_red
many_spots
fore_half_different_color
Texture
horizontal_bands
edge_with_different_color
Shape wave
Degree of
Match
52/57
12/42
3/38
58/62
27/33
41/60
10/74
98/110
Feature Description (3)
Color
Figure
Feature Value
black
brown
bister
orange_red
orange_yellow
yellow
green
blue
purple
gray
white
mixed_with_black_and_white
mixed_with_black_and_yellow
mixed_with_black_and_orange
mixed_with_black_and_blue
mixed_with_black_and_red
mixed_with_wood_and_white
mixed_with_many_colors
Feature Description (4)
Texture
Feature Description (5)
Shape
Feature Description (6)

QBF query:

Single feature query:
Result images: images with its corresponding
degree of match > 0.
 Ranked by: degree of match.
 We call this ranked sequence “Feature
sequence.”


Multiple features query:

Merge the corresponding feature sequences.
Result Presentation

For QBF query:



Property: rough search
Presentation: representative butterflies only
For QBE query:


Property: fine search
Presentation:
 For very similar images: present them all
 For less similar images: representative ones
Feature Vector Indexing

Goal:


Problems of Indexing in CBIR:



To make search efficiently.
Dimension of feature space is very high.
Index structure should support
Euclidean and non-Euclidean similarity
measures.
Solution:


Dimension reduction: KLT, DCT, DWT.
Similarity indexing: R*-tree, SS-tree, SRtree.
Semi-Automatic Feature
Extraction

Segmentation:



Background segmentation
Butterfly object segmentation
Feature extraction:



Color: color histogram
Texture: manual annotation
Shape: manual annotation
Classic CBIR with Color
Feature

Most of the CBR systems rely on the
notion of color, this may differ:



Dominant color
Scalable color based on color histograms
(local for one region, global for the whole
image)
Color Structure Descriptor (incoporates the
spatial structure)
What color is the apple ? We are so
visual !!!!
I’d say it is
Bright Red
I really
couldn’t tell you
(I am color blind)
I think it is
“Crimson”
It is
Red!
Color Histogram: Representation

A list of Color-Percentage pairs:

Describe the colors and its percentages in an
image.




f c  (I j , Pj ) I j  ColorValue,0  Pj  1,  Pj  1, and 1  j  N 


1 j  N


Color Quantization



Indexed Colors
A jpg Image with 256-color components in each
RGB channel
256 x 256 x 256 colors in total → n groups, e.g,
in 256 groups, that makes a reduction 256x256,
I.e., that each group takes 256 colors to count.
Similarity Measures - Overview
d(h q , h t )  (
M 1
 h [k ]  h [k ]
q
t
r 1/ r
)
r 1
k 0

Minkowski Similarity

Distance L1 : r = 1

Distance L2 : r = 2
d(h q , h t )  (h q  h t )T A(h q  h t )

Quadratic Similarity

Intersection Similarity
(Swain et Ballard 1991)
M 1
d q,t 
 min(h
k 0
q [ k ], h t [ k ])
M 1
 h [k ]
t
k 0
Example (cont.)

Minkowski Similarity

Is a L-1 metric
n
D( H I , H J )   | I k  J k |
k 1
where Ik and Jk is the number of pixels in bin k
for image I and J

Distance between above three images
D(H1, H2) = 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 = 8
 D(H1, H3) = 6 + 6 + 2 + 2 + 2 + 2 + 2 + 2 = 24
 D(H2, H3) = 5 + 5 + 3 + 3 + 1 + 1 + 1 + 1 = 23

Example (cont.)

Minkowski Similarity


Is a L-2 metric
Distance between above three images
D(H1, H2) = (1 + 1 + 1 + 1 + 1 + 1 + 1 + 1)1/2 = 2.8
 D(H1, H3) = (36 + 36 + 4 + 4 + 4 + 4 + 4 + 4)1/2 = 9.8
 D(H2, H3) = (25 + 25 + 9 + 9 + 1 + 1 + 1 + 1) 1/2 = 8.5

QBIC distance

Weighted Euclidean distance (QBIC)

Is a L-2 metric(?)
distance between histogram H1 and H2:
D = (H1 - H2)T A (H1 - H2)
where A is a symmetric color similarity matrix
A (i, j) = 1 – d (ci, cj) / dmax
where ci and cj are the i-th and j-th color bins,
d (ci , cj) is the color distance in the color space,
and dmax is the maximum distance between any
two colors in the color space
Limitation

Ignore similarity between colors
 Example
 Two color bins
 Bin-1 color range: 1 – 10
 Bin-2 color range: 11 – 20
•Three color pixels
–Pixel 1 is Color 10  Bin-1
–Pixel 2 is Color 11  Bin-2
–Pixel 3 is Color 20  Bin-2
–Pixel 2 is similar to Pixel 3 than Pixel 1 
unreasonable !
Limitation (cont.)

Ignore spatial relationships among pixels
Different image with same histogram
Noise-Free Queries (NFQ’s)
Rectangular query

NFQ is more precise.

User can specify semantic constraints:


Spatial constraints (relative distances)
Scaling constraints (relative sizes)
Noise-free
query
Similar
32
Less relevant
Challenges


Noise-free
query
33
How do we extract features
if we do not know the
matching areas beforehand ?
How do we index the images ?
One Solution – Local Color
Histogram (LCH)


34
Each subimage has a
color histogram.
Any combination of the
histograms can be
selected for comparison
with the corresponding
color histograms of the
query image.
Query image
Database image
Limitations of LCH

Dilemma:



Using small partitions is
too expensive
Query image
Limitation:

35
Using large partitions is
not precise
difficult to handle scaling
Database image
Resultados esperados de uma
boa CBIR com segmentação
Query
2
Query
36
4
3
2
3
5
216
12
396
18
DEMOS

Hermitage Museum Web Site (QBIC)
http://hermitagemuseum.org/
http://hermitagemuseum.org/fcgibin/db2www/qbicColor.mac/qbic?selLang=En
glish

37
http://www.aa-lab.cs.uu.nl/cbirsurvey/cbirsurvey/