Transcript An introduction to image retrieval for digital libraries

Workshop on Preserving Intellectual
Assets: Institutional Repositories and
Open Access
TEI Thessalonikis, Sindos, September 2006
An introduction to image retrieval
Professor Dick Hartley
Manchester Metropolitan University
Introduction to image retrieval
Why is image retrieval important for digital
libraries and institutional repositories?
Why is image retrieval difficult?
What are the approaches to image
retrieval?
How are we going to achieve this?
• The bad news
– I am going to do some talking
– So, you are going to do some listening (I
hope!!!!)
• The good news
– You are going to do some work ! (well it is a
workshop!!!!)
How are we going to achieve this?
• Day 1
– Why is image retrieval important?
– Why is it difficult?
– One approach to image retrieval
– Practical image retrieval exercise
• Day 2
– A second approach to image retrieval
– Practical exercise in image indexing
– Research on image seeking behaviour
What do I mean by image retrieval?
• Digitized images of text
• Digital images in every conceivable
subject from medical imaging, through
satellite imagery to art history
Why is image retrieval important?
• Image information is crucial in many
contexts
• Huge quantities of image data is now
available in digital form
• Digital information on every imaginable
subject is readily available on the Web
• Many digital libraries contain digital
information; this is pointless unless it can
be effectively retrieved
So….
Research and practical developments in
image retrieval and in understanding of
image seeking behaviour have been
major areas of development in information
retrieval in the last decade
Why is it difficult?
What is an image about?
Look at the following examples……
Are you Offended?
Why is it difficult?
• I want a picture of a tower in Greece at
night
• I want a picture of a bridge in Edinburgh
during the summer
Approaches to image retrieval
• Content-based image retrieval
• Concept-based image retrieval
Content-Based Image Retrieval
Feature
Extraction
Image
Collection
Visual
Features
Multidimensional
Indexing
Query
Processing
Retrieval Engine
Query
Interface
User
Text
Annotation
• Semi-automatic
or automatic
extraction,
indexing and
retrieval of
images by their
visual attributes.
Similarity Measures
2
2
d ds1s1,,ss 22 
x

y
x2 kx11 k y 2  yk1 
n
Y
Minkowsky Distance
d s1 , s2   x2  x1  y2  y1
s2
y2
y1
City-Block
Distance
s
1
x1
x2
X
Similar by
Colour
(Global)
Retrieval
Colour
(Global)
hR,G,B r, g, b  N  ProbR  r, G  g, B  b
DH I Q , I D    | H I Q , j   H I D , j  |
n
j 1
False Positives
Dissimilar
Local Colour
Retrieval
by Colour
(Local)
 | H I   H I  |
n
DH I Q , I D  
Q, j
j 1
D, j
 H I 
n
j 1
D, j
Colour Histogram Intersection
Shape Retrieval
 p ,q
 p ,q
pq2
 y , 
2
 0, 0
Inference
Noise
Trademark Image Retrieval
Device Marks
Texture
#  p1 , p 2   I | p1  i  p 2  j
Pi, j  
#I
Fcrs
1 m


m xn i 1
Fcon
n
 S i, j 
j 1
best

 1/ 4
4
np
Fdir  
p
2





H D  

w p
Image Query Paradigms
•• Relational-Based
Query
by Pictorial
Tabular-based
dataExample
model (QPE) [Chang
– SQL [Codd, 1970].
and
Fu, By
1980].
–
Query
Example
(QBE)
– ISQL [Assmann,
Venema,
and [Zloof,
Hohne, 1977].
1986],
• Aggregate
by Example
[Klug,
1981]
– PROBE
[Orenstein,
and Manola,
1988],
• PICQUERY
[Joseph
and
Cardenas,
1988].
– PSQL
[Roussopoulos,
Faloutsos,
Sellis,and
1988]
• Generalised
Query by
Exampleand
[Jacobs
– Spatial
SQL [Egenhofer,
1991].
Walczak,
1983]
• Office by Example [Whang et al. 1987]
SELECT city, state, population, location
• Time
Example [Tansel et al., 1989]
FROMbycities
ON us-map
• Natural
Forms Query Language (NFQL) [Embley,
WHERE location within (4+4, 11+9)
1989].
AND population > 450,000
Query by Visual Example
Q  I Q , FQ , SQ , ZQ 
QBIC
“One
of theetkey
• Niblack
al., 1993.
that
• challenges
Lee et al., 1994.
remains in making
• Bird et al., 1996.
the technology
• Bird
et al., and
1999.
pervasive
useful
is the design of the
user interface.”
[Flickner et al., 1997.]
Query by Image
Query by Icon
Query by Paint
Query
by Visual
Example
Query
by Sketch
Beyond Query by Visual Example
• 2D/3D Visualization
– ImageVIBE [Cinque et al., 1998].
– 3DVIBE [Santini and Jain, 1997].
• Virtual Reality
– Query by Photograph [Assfalg et al., 2000].
• Taxonomy Human Perception of Images
– Burford et al. [2002].
Learning-Based
Feature Relevance
Computation
User
Feedback
Feature Weight Vector
Metric 1
Query
Image
Reinforcement
Learning
Retrieval
Results
User
Interaction
Metric n
Feature
Database
Image
Database
CBIR Summarized
• CBIR permits retrieval by image attributes
– Colour, shape, texture (or a combination)
CBIR Advantages
• No metadata necessary
• Possible to “index” a huge volume of
material and rapidly
• Does not depend on interpretation of
meaning
CBIR disadvantages
• “Semantic gap” between what users want
and what CBIR systems can achieve
• No sensible means by which queries can
be presented to a CBIR system
CBIR uses
• Restricted areas such as trade mark
searching
Time for a break, then you can experiment
with an operational CBIR system