Transcript Document

Shape-Representation
and
Shape Similarity
Motivation
WHY SHAPE ?
Motivation
We’ve seen this already in the introduction
of this course:
These objects are recognized by…
Motivation
These objects are recognized by…
Texture
Color
X
X
Context
X
Shape
X
X
X
X
X
X
Motivation
Shape is not the only, but a very
powerful descriptor of
image content
Why Shape ?
Several applications in computer vision use
shape processing:
• Object recognition
• Image retrieval
• Processing of pictorial information
• Video compression (eg. MPEG-7)
…
(Reminder: this course focuses on
object recognition and image retrieval)
Blobworld
Example 1: Blobworld
http://elib.cs.berkeley.edu/photos/blobworld/start.html
BLOB = “Binary Large Object”,
“an indistinct shapeless (really ?) form”
Blobworld
Description of Blobworld:
PROJECT !
ISS Database
Example 2: ISS-Database
http://knight.cis.temple.edu/~shape
The Interface (JAVA – Applet)
ISS Database
ISS: Query by Shape
Sketch of
Shape
Query:
by Shape
only
Result:
Satisfying ?
ISS Database
The ISS-Database will be topic
of this tutorial
Overview
Overview
•
•
•
•
•
•
•
Why shape ?
What is shape ?
Shape similarity
(Metrices)
Classes of similarity measures
(Feature Based Coding)
Examples for global similarity
Why Shape ?
Why Shape ?
•
Shape is probably the most important property
that is perceived about objects. It allows to
predict more facts about an object than other
features, e.g. color (Palmer 1999)
•
Thus, recognizing shape is crucial for object
recognition. In some applications it may be the
only feature present, e.g. logo recognition
Why Shape ?
Shape is not only perceived by visual means:
•
•
tactical sensors can also provide shape
information that are processed in a similar
way.
robots’ range sensor provide shape
information, too.
Shape
Typical problems:
• How to describe shape ?
• What is the matching
transformation?
• No one-to-one correspondence
• Occlusion
• Noise
Shape
• Partial match:
only a part of
the query
appears in a
part of the
database shape
What is Shape ?
What is Shape ?
Plato, "Meno", 380 BC:
• "figure is the only existing thing that
is found always following color“
• "figure is limit of solid"
What is Shape ?
… let’s start with some properties easier
to agree on:
• Shape describes a spatial region
Shape is a (the ?) specific part of spatial cognition
• Typically addresses 2D space
What is Shape ?
• 3D => 2D projection
What is Shape ?
• the original 3D (?) object
What is Shape ?
Moving on from the naive
understanding, some questions arise:
• Is there a maximum size for a shape to be a
shape?
• Can a shape have holes?
• Does shape always describe a connected
region?
• How to deal with/represent partial shapes
(occlusion / partial match) ?
What is Shape ?
Shape or Not ?
Continuous transformation from shape to two
shapes: Is there a point when it stops being
a single shape?
What is Shape ?
But there’s no doubt that
a single, connected region
is a shape.
Right ?
What is Shape ?
A single, connected region.
But a shape ?
A question of scale !
What is Shape ?
• There’s no easy, single definition of shape
•
In difference to geometry, arbitrary shape is not
covered by an axiomatic system
• Different applications in object recognition
focus on different shape related features
•
Special shapes can be handled
• Typically, applications in object recognition
employ a similarity measure to determine a
plausibility that two shapes correspond to
each other
Similarity
So the new question is:
What is Shape Similarity ?
or
How to Define a Similarity Measure
Similarity
Again: it’s not so simple (sorry).
There’s nothing like
THE
similarity measure
Similarity
which similarity measure,
depends on
which required properties,
depends on
which particular matching problem,
depends on
which application
Similarity
…which application
Simple Recognition (yes / no)
... robustness
Common Rating (best of ...)
Analytical Rating (best of, but...)
... invariance to basic transformations
Similarity
…which problem
• computation problem: d(A,B)
• decision problem: d(A,B) <e ?
• decision problem: is there g: d(g(A),B) <e ?
• optimization problem: find g: min d(g(A),B)
Similarity
…which properties:
We concentrate here on the
computational problem d(A,B)
Similarity Measure
Requirements to a similarity measure
• Should not incorporate context knowledge
(no AI), thus computes generic shape
similarity
Similarity Measure
Requirements to a similarity measure
• Must be able to deal with noise
• Must be invariant with respect to basic
transformations
Scaling (or resolution)
Next:
Strategy
Rotation
Rigid / non-rigid deformation
Similarity Measure
Requirements to a similarity measure
• Must be able to deal with noise
• Must be invariant with respect to basic
transformations
• Must be in accord with human perception
Similarity Measure
Some other aspects worth consideration:
• Similarity of structure
• Similarity of area
Can all these aspects be expressed by a
single number?
Similarity Measure
Desired Properties of a Similarity Function C
(Basri et al. 1998)
• C should be a metric
• C should be continous
• C should be invariant (to…)
Properties
Metric Properties
S set of patterns
Metric: d: S  S R satisfying
1. Self-identity: "xS, d(x,x)=0
2. Positivity: "x yS, d(x,y)>0
3. Symmetry: "x, yS, d(x,y)= d(y,x)
4. Triangle inequality: "x, y, zS,
d(x,z)d(x,y)+d(y,z)
• Semi-metric: 1, 2, 3
• Pseudo-metric: 1, 3, 4
• S with fixed metric d is called metric space
Properties
1. Self-identity: "xS, d(x,x)=0
2. Positivity: "x yS, d(x,y)>0
…surely makes sense
Properties
Properties
Properties
In general:
• a similarity measure in accordance with
human perception is NOT a metric. This
leads to deep problems in further
processing, e.g. clustering, since most of
these algorithms need metric spaces !
Properties
Properties
Properties
Some more properties:
• One major difference should cause a
greater dissimilarity than some minor ones.
• S must not diverge for curves that are not
smooth (e.g. polygons).
Similarity Measures
Classes of Similarity Measures:
Similarity Measure depends on
• Shape Representation
• Boundary
• Area (discrete: = point set)
• Structural (e.g. Skeleton)
• Comparison Model
• feature vector
• direct
Similarity Measures
Boundary
Area (point set)
direct
feature based
Spring model, Cum.
Angular Function,
Chaincode, Arc
Decomposition (ASRAlgorithm)
Central Dist. Fourier
Hausdorff
Moments
…
Zernike Moments
Distance histogram
…
…
Structure
Skeleton
…
---
Feature Based Coding
Feature Based Coding (again…)
This category defines all approaches that
determine a feature-vector for a given shape.
Two operations need to be defined: a mapping of
shape into the feature space and a similarity of
feature vectors.
Representation
Feature Extraction
Vector Comparison
Vector Comparison
We’ve done this already with
• histograms,
• fourier spectra,
• shape features (centroid distance spectrum)
Vector Comparison
Another feature you should have heard of:
(Discrete) Moments
Shape A,B given as
• Area (continous) or
• Point Sets (discrete)
Moments
Discrete Point Sets
Moments
Moments
Moments
Discrete Moments
Exercise:
Please compute all 7 moments for the following shapes, compare the
vectors using different comparison techniques

Discrete Moments
Result: each shape is transformed to a 7dimensional vector. To compare the shapes,
compare the vectors (how ?).
3D Distance Histogram
Another Example
3D Distance Histogram
Shape A,B given as 3D point set
3D Distance Histogram
PROJECT !
Feature Based Coding
Again:
Two operations need to be defined: a mapping of
shape into the feature space and a similarity of
feature vectors.
We hence have TWO TIMES an information
reduction of the basic representation, which by
itself is already a mapping of the ‘reality’.
Representation
Feature Extraction
Vector Comparison
Direct Comparison
End of Feature Based Coding !
Next:
Direct Comparison
Vector
Direct Comparison
Comparison
Example 1
Hausdorff Distance
Shape A,B given as point sets
A={a1,a2,…}
B={b1,b2,…}
Feature
Vector Based
Comparison
Coding
Hausdorff
Vector Comparison
Distance
Boundary
Vector Comparison
Representation
Hausdorff:
Unstable with respect to noise
(This is easy to fix ! How ?)
Problem: Invariance !
Nevertheless: Hausdorff is the motor behind many applications
in specific fields (e.g. character recognition)
Boundary
Vector Comparison
Representation
Example 2
Chaincode Comparison
Shape A,B given as chaincode
Boundary
Vector Comparison
Representation
Boundary
Vector Comparison
Representation
A binary image can be converted into a ‘chain code’
representing the boundary. The boundary is traversed
and a string representing the curvature is
constructed.
3
2
1
4
C
0
5
6
7
5,6,6,3,3,4,3,2,3,4,5,3,…
Boundary
Vector Comparison
Representation
Resulting strings are then compared using
classical string-matching techniques.
Not very robust.
Boundary
Vector Comparison
Representation
Digital curves suffer from effects caused by
digitalization, e.g. rotation:
Structural
Vector Comparison
Representation
Structural approaches capture the
structure of a shape, typically by
representing shape as a graph.
Typical example: skeletons
Structural
Vector Comparison
Representation
Skeletons
Shape A,B primarily given as area or
boundary, structure is derived from
representation
Structural
Vector Comparison
Representation
The computation can be described as
a medial axis transform, a kind of
discrete generalized voronoi.
Structural
Vector Comparison
Representation
The graph is constructed mirroring the
adjacency of the skeleton’s parts. Edges are
labeled according to the qualitative classes.
Matching two shapes requires matching two
usually different graphs against each other.
Structural
Vector Comparison
Representation
Problems of skeletons:
- Pruning
Structural
Vector Comparison
Representation
-Robustness
Vector
Shape
Comparison
similarity
All similarity measures shown can not deal with
occlusions or partial matching (except
skeletons ?) !
They are useful (and used) for specific
applications, but are not sufficient to deal with
arbitrary shapes
Solution: Part – based similarity !
Shape-Representation
and
Shape Similarity
PART 2: PART BASED SIMILARITY
Motivation
WHY PARTS ?
Motivation
Motivation
Motivation
Motivation
Global similarity measures fail at:
• Occlusion
• Global Deformation
• Partial Match
• (actually everything that occurs under
‘real’ conditions)
Parts
Requirements for a Part Based
Shape Representation
(Siddiqi / Kimia ’96: ‘Parts of Visual Form: Computational
Aspects’)
Parts
How should parts be defined / computed ?
Some approaches:
• Decomposition of interior
• Skeletons
• Maximally convex parts
• Best combination of primitives
• Boundary Based
• High Curvature Points
• Constant Curvature Segments
Parts
Principal approach:
Hoffman/Richards (’85):
‘Part decomposition should precede part
description’
=> No primitives, but general principles
Parts
No primitives, but general principals
“When two arbitrarily shaped
surfaces are made to
interpenetrate they always meet
in a contour of concave
discontinuity of their tangent
planes” (transversality
principle)
Parts
“When two arbitrarily shaped
surfaces are made to
interpenetrate they always meet
in a contour of concave
discontinuity of their tangent
planes” (transversality
principle)
Divide a plane curve into parts at
negative minima of curvature
Parts
Different notions of parts:
•
Parts: object is composed of rigid
parts
•
Protrusions: object arises from object
by deformation due to a (growth)
process (morphology)
• Bends: Parts are result of bending the
base object
Parts
The Shape Triangle
Parts
This lecture focuses on parts, i.e. on
partitioning a shape
Framework
A Framework for a Partitioning Scheme
Scheme must be invariant to 2 classes of
changes:
• Global changes : translations, rotations
& scaling of 2D shape, viewpoint,…
• Local changes: occlusions, movement
of parts (rigid/non-rigid deformation)
Framework
A general decomposition of a shape
should be based on the
interaction between two parts
rather than on their shapes.
-> Partitioning by Part Lines
Framework
Definition 1:
A part line is a curve whose end
points rest on the boundary of the
shape, which is entirely
embedded in it, and which divides
it into two connected
components.
Framework
Definition 2:
A partitioning scheme is a mapping
of a connected region in the
image to a finite set of connected
regions separated by part-lines.
Framework
Definition 3:
A partitioning scheme is invariant if
the part lines of a shape that is
transformed by a combination of
translations, rotations and
scalings are transformed in
exactly the same manner.
Framework
Definition 4:
A partitioning scheme is robust if
for any two shapes A and B, which
are exactly the same in some
neighborhood N, the part lines
contained in N for A and B are
exactly equivalent.
Framework
Definition 5:
A partitioning scheme is stable if
slight deformations of the
boundary of a shape cause only
slight changes in its part lines
Framework
Definition 6:
A partitioning scheme is scaletuned if when moving from coarse
to fine scale, part lines are only
added, not removed, leading to a
hierarchy of parts.
Framework
A general purpose partitioning
scheme that is consistent with
these requirements is the
partitioning by
limbs and necks
Limbs and Necks
Definition :
A limb is a part-line going through a
pair of negative curvature minima
with co-circular boundary
tangents on (at least) one side of
the part-line
Limbs and Necks
Motivation: co-circularity
The decomposition of the
right figure is no longer
intuitive: absence of
‘good continuation’
Limbs and Necks
Smooth continuation: an example
for
form from function
• Shape of object is given by natural
•
•
function
Different parts having different functions
show sharp changes in the 3d surface of
the connection
Projection to 2d yields high curvature
points
Limbs and Necks
Examples of limb based parts
Limbs and Necks
Definition :
A neck is a part-line which is also a
local minimum of the diameter of
an inscribed circle
Limbs and Necks
Motivation for necks:
Form From Function
• Natural requirements (e.g. space for
articulation and economy of mass at the
connection) lead to a narrowing of the joint
between two parts
Limbs and Necks
The Limb and Neck partitioning
scheme is consistent with the
previously defined requirements
•
•
•
•
Invariance
Robustness
Stability
Scale tuning
Limbs and Necks
Examples:
Limbs and Necks
The scheme presented does NOT
include a similarity measure !
Algorithms
Part Respecting Similarity Measures
CSS
Curvature Scale Space
(Mokhtarian/Abbasi/Kittler)
A similarity measure implicitely
respecting parts
CSS
Creation of reflection-point based feature-vector which
implicitely contains part – information
CSS
Properties:
• Boundary Based
• Continous Model (!)
• Computes Feature Vector
• compact representation of shape
• Performs well !
CSS
PROJECT !
CSS
Some results (Database: 450 marine animals)
CSS
The main problem:
CSS is continous, the computer vision
world is discrete.
How to measure curvature in discrete
boundaries ?
Dominant Points
Local curvature = average curvature in
‘region of support’
To define regions of support, ‘dominant
points’ are needed !
Dominant Points
Dominant Points
(“Things should be expressed as simple as possible, but not simpler”, A.
Einstein)
Idea: given a discrete boundary S
compute polygonal boundary S’ with
minimum number of vertices which is
visually similar to S.
Dominant Points
Example Algorithms
( 3 of billions…)
• Ramer
• Line Fitting
• Discrete Curve Evolution
DCE
Discrete Curve Evolution
(Latecki / Lakaemper ’99)
Idea:
Detect subset of visually significant
points
Curve Evolution
Target: reduce data by elimination of irrelevant features,
preserve relevant features
... noise reduction
... shape simplification:
Curve Evolution: Tangent Space
Transformation from image-space to tangent-space
Tangent Space: Properties
In tangent space...
... the height of a step shows the turn-angle
... monotonic increasing intervals represent convex arcs
... height-shifting corresponds to rotation
... the resulting curve can be interpreted as 1 –
dimensional signal => idea: filter signal in tangent
space (demo: 'fishapplet')
Curve Evolution: Step Compensation
(Nonlinear) filter: merging of 2 steps with area – difference F
given by:
(a-b)pq
F=
p+q
a
g
b
q
F
F
p
Curve Evolution: Step Compensation
Interpretation in image – space:
... Polygon – linearization
... removal of visual irrelevant vertices
p
q
removed vertex
Curve Evolution: Step Compensation
Interpretation in image – space:
... Polygon – linearization
... removal of visual irrelevant vertices
next:
Iterative SC
Curve Evolution: Iterative Step Compensation
Keep it simple: repeated step compensation !
(demo: EvoApplet)
Remark: there are of course some traps ...
Curve Evolution: Iterative Step Compensation
Remark: there are of course some traps:

Self intersection / Topology preservation

Stop parameter

Edge movement
Curve Evolution: Properties
The evolution...
... reduces the shape-complexity
... is robust to noise
... is invariant to translation, scaling and rotation
... preserves the position of important vertices
... extracts line segments
... is in accord with visual perception
... offers noise-reduction and shape abstraction
... is parameter free
... is translatable to higher dimensions
Curve Evolution: Properties
Robustness (demo: noiseApplet)
Curve Evolution: Properties
Preservation of position, no blurring !
Curve Evolution: Properties
Strong relation to digital lines and segments
Curve Evolution: Properties
Noise reduction as well as shape abstraction
Curve Evolution: Properties
Parameter free (?)
Curve Evolution: Properties
Extendable to higher dimensions
Curve Evolution: Properties
Extendable to higher dimensions
Curve Evolution: Properties
Extendable to higher dimensions
Curve Evolution: Properties
Extendable to higher dimensions
Curve Evolution: Properties
Result:
 The DCE creates a polygonal shape
representation in different levels of
granularity: Scale Space
 Curvature can be defined as the
turning angle at the vertices
 Regions of support are defined by
vertices
 Easy traceable Scale Space is
created, since no points are relocated
Scale Space
Ordered set of representations on different information levels
Polygonal Representation
The polygonal representation
achieved by the DCE has a huge
advantage:
It allows easy boundary
partitioning using convex / concave
parts (remember the limbs !)
(MATLAB Demo MatchingDemo)
DCE
Some results of part line decomposition:
ASR
The ASR (Advanced Shape
Recognition) Algorithm uses the
boundary parts achieved by the
polygonal representation for a part
based similarity measure !
(Note: this is NOT the area partitioning shown in the previous slide)
ASR / ISS
The ASR is used in the ISS Database
How does it work ?
Behind The Scenes of the ISS Database:
Modern Techniques of Shape
Recognition and Database Retrieval
The 2nd Step First: Shape Comparison
ISS implements the ASR (Advanced Shape Recognition)
Algorithm
Developed by Hamburg University in cooperation with
Siemens AG, Munich, for industrial applications in...
... robotics
... multimedia (MPEG – 7)
Reticent Proudness…
MPEG-7: ASR outperformes classical
approaches !
Similarity test (70 basic shapes, 20 different deformations):
ASR
Curvature Scale Space
Multilayer Eigenvector
Zernicke Moments
Wavelet Contour
DAG Ordered Trees
(Capitulation :-)
Hamburg Univ./Siemens AG
Mitsubishi ITE-VIL
Hyundai
Hanyang University
Heinrich Hertz Institute Berlin
Mitsubishi/Princeton University
IBM
76.45 %
75.44 %
70.33 %
70.22 %
67.67 %
60.00 %
--.-- %
Requirements
Robust automatic recognition of arbitrary shaped objects
which is in accord with human visual perception
Wide range of applications...
... recognition of complex and arbitrary patterns
... invariance to basic transformations
... results which are in accord with human perception
... applicable to three main tasks of recognition
... parameter-free operation
Industrial requirements...
... robustness
... low processing time
Requirements
Scaling (or resolution)
Next:
Strategy
Rotation shaped objects
Robust automatic recognition of arbitrary
which is in accord with human visual
Rigidperception
/ non-rigid deformation
Wide range of applications...
... recognition of complex and arbitrary patterns
... invariance to basic transformations
... results which are in accord with human perception
... applicable to three main tasks of recognition
... parameter-free operation
Industrial requirements...
... robustness
... low processing time
Simple Recognition (yes / no)
Requirements
... robustness
low processing
time of arbitrary shaped objects
Robust...automatic
recognition
Rating
(best
of ...) visual perception
which is Common
in accord
with
human
Wide range of applications...
... recognition
of complex
arbitrary patterns
Analytical Rating
(best of,and
but...)
... results
which are
in accord
with human
... invariance
to basic
transformations
perception
... applicable to three main tasks of recognition
... parameter-free operation
Industrial requirements...
... robustness
... low processing time
Different Approaches
Pattern Matching...
... Correlation
Geometrical description...
... Hough – Transformation
Feature – Vectors...
... (Zernicke - ) Moments
Based on Visual Parts...
... Mokhtarian
... ASR
ASR: Strategy
Source:
2D - Image
Object - Segmentation
Contour Extraction
Evolution
Contour – Segmentation
Arc – Matching
ASR: Strategy
DCE
Contour – Segmentation
Arc – Matching
Contour Segmentation
Correspondence ?
Similarity of parts ?
Part Similarity
Similarity of parts ?
= Boundary Similarity Measure
= Similarity of polygons
Tangent Space
Transformation from image-space to tangent-space
Shape Comparison: Measure
Tangent space offers an intuitive measure:
lg 0  lg1 

T g 0 sT g 1 s  ads  maxlg 0 , lg 1 max l g , l g 
1
0
2
1
2
Shape Comparison: Measure
Drawback:
not adaptive to unequally distributed noise if used globally !
…but works for single parts
Shape Comparison: Contour Segmentation
Shape Comparison: Correspondence
Optimal arc-correspondence:
find one to many (many to one) correspondence, that
minimizes the arc-measure !
next:
Corr. -example
Graph of Correspondence
arc
a0
a3
a2
a1
b0
a0 a1 a2 a3
b0 b1 b2 b3
b3
b2
b1
Graph:
... edge represents correspondence
... node represents matched arcs
correspondence
Shape Comparison: Correspondence
Example:
a0 a1 a2 a3
a0
a3
a2
a1
b0 b1 b2 b3
b0
b3
b2
b1
Shape Comparison: Correspondence
Result:
Optimal correspondence is given by cheapest way
next:
Corr. - Results
Correspondence: Results
(MATLAB Demo)
Correspondence: Results
Correspondence and arc-measure allow...
... the identification of visual parts as well as
... the identification of the entire object
... a robust recognition of defective parts
... a shape matching which is in accord with
human perception
ASR Results
Correspondence and arc-measure meet
the requirements stated by Kimia et al.


Discrete

Easy computable
ASR Results
Problem:
Time Consuming Algorithm (~10ms on
2GHz Pentium)
No Metric (-> no clustering) !
How to build a database with this
approach ?