#### Transcript Document

Shape-Representation and Shape Similarity Motivation WHY SHAPE ? Motivation We’ve seen this already in the introduction of this course: These objects are recognized by… Motivation These objects are recognized by… Texture Color X X Context X Shape X X X X X X Motivation Shape is not the only, but a very powerful descriptor of image content Why Shape ? Several applications in computer vision use shape processing: • Object recognition • Image retrieval • Processing of pictorial information • Video compression (eg. MPEG-7) … (Reminder: this course focuses on object recognition and image retrieval) Blobworld Example 1: Blobworld http://elib.cs.berkeley.edu/photos/blobworld/start.html BLOB = “Binary Large Object”, “an indistinct shapeless (really ?) form” Blobworld Description of Blobworld: PROJECT ! ISS Database Example 2: ISS-Database http://knight.cis.temple.edu/~shape The Interface (JAVA – Applet) ISS Database ISS: Query by Shape Sketch of Shape Query: by Shape only Result: Satisfying ? ISS Database The ISS-Database will be topic of this tutorial Overview Overview • • • • • • • Why shape ? What is shape ? Shape similarity (Metrices) Classes of similarity measures (Feature Based Coding) Examples for global similarity Why Shape ? Why Shape ? • Shape is probably the most important property that is perceived about objects. It allows to predict more facts about an object than other features, e.g. color (Palmer 1999) • Thus, recognizing shape is crucial for object recognition. In some applications it may be the only feature present, e.g. logo recognition Why Shape ? Shape is not only perceived by visual means: • • tactical sensors can also provide shape information that are processed in a similar way. robots’ range sensor provide shape information, too. Shape Typical problems: • How to describe shape ? • What is the matching transformation? • No one-to-one correspondence • Occlusion • Noise Shape • Partial match: only a part of the query appears in a part of the database shape What is Shape ? What is Shape ? Plato, "Meno", 380 BC: • "figure is the only existing thing that is found always following color“ • "figure is limit of solid" What is Shape ? … let’s start with some properties easier to agree on: • Shape describes a spatial region Shape is a (the ?) specific part of spatial cognition • Typically addresses 2D space What is Shape ? • 3D => 2D projection What is Shape ? • the original 3D (?) object What is Shape ? Moving on from the naive understanding, some questions arise: • Is there a maximum size for a shape to be a shape? • Can a shape have holes? • Does shape always describe a connected region? • How to deal with/represent partial shapes (occlusion / partial match) ? What is Shape ? Shape or Not ? Continuous transformation from shape to two shapes: Is there a point when it stops being a single shape? What is Shape ? But there’s no doubt that a single, connected region is a shape. Right ? What is Shape ? A single, connected region. But a shape ? A question of scale ! What is Shape ? • There’s no easy, single definition of shape • In difference to geometry, arbitrary shape is not covered by an axiomatic system • Different applications in object recognition focus on different shape related features • Special shapes can be handled • Typically, applications in object recognition employ a similarity measure to determine a plausibility that two shapes correspond to each other Similarity So the new question is: What is Shape Similarity ? or How to Define a Similarity Measure Similarity Again: it’s not so simple (sorry). There’s nothing like THE similarity measure Similarity which similarity measure, depends on which required properties, depends on which particular matching problem, depends on which application Similarity …which application Simple Recognition (yes / no) ... robustness Common Rating (best of ...) Analytical Rating (best of, but...) ... invariance to basic transformations Similarity …which problem • computation problem: d(A,B) • decision problem: d(A,B) <e ? • decision problem: is there g: d(g(A),B) <e ? • optimization problem: find g: min d(g(A),B) Similarity …which properties: We concentrate here on the computational problem d(A,B) Similarity Measure Requirements to a similarity measure • Should not incorporate context knowledge (no AI), thus computes generic shape similarity Similarity Measure Requirements to a similarity measure • Must be able to deal with noise • Must be invariant with respect to basic transformations Scaling (or resolution) Next: Strategy Rotation Rigid / non-rigid deformation Similarity Measure Requirements to a similarity measure • Must be able to deal with noise • Must be invariant with respect to basic transformations • Must be in accord with human perception Similarity Measure Some other aspects worth consideration: • Similarity of structure • Similarity of area Can all these aspects be expressed by a single number? Similarity Measure Desired Properties of a Similarity Function C (Basri et al. 1998) • C should be a metric • C should be continous • C should be invariant (to…) Properties Metric Properties S set of patterns Metric: d: S S R satisfying 1. Self-identity: "xS, d(x,x)=0 2. Positivity: "x yS, d(x,y)>0 3. Symmetry: "x, yS, d(x,y)= d(y,x) 4. Triangle inequality: "x, y, zS, d(x,z)d(x,y)+d(y,z) • Semi-metric: 1, 2, 3 • Pseudo-metric: 1, 3, 4 • S with fixed metric d is called metric space Properties 1. Self-identity: "xS, d(x,x)=0 2. Positivity: "x yS, d(x,y)>0 …surely makes sense Properties Properties Properties In general: • a similarity measure in accordance with human perception is NOT a metric. This leads to deep problems in further processing, e.g. clustering, since most of these algorithms need metric spaces ! Properties Properties Properties Some more properties: • One major difference should cause a greater dissimilarity than some minor ones. • S must not diverge for curves that are not smooth (e.g. polygons). Similarity Measures Classes of Similarity Measures: Similarity Measure depends on • Shape Representation • Boundary • Area (discrete: = point set) • Structural (e.g. Skeleton) • Comparison Model • feature vector • direct Similarity Measures Boundary Area (point set) direct feature based Spring model, Cum. Angular Function, Chaincode, Arc Decomposition (ASRAlgorithm) Central Dist. Fourier Hausdorff Moments … Zernike Moments Distance histogram … … Structure Skeleton … --- Feature Based Coding Feature Based Coding (again…) This category defines all approaches that determine a feature-vector for a given shape. Two operations need to be defined: a mapping of shape into the feature space and a similarity of feature vectors. Representation Feature Extraction Vector Comparison Vector Comparison We’ve done this already with • histograms, • fourier spectra, • shape features (centroid distance spectrum) Vector Comparison Another feature you should have heard of: (Discrete) Moments Shape A,B given as • Area (continous) or • Point Sets (discrete) Moments Discrete Point Sets Moments Moments Moments Discrete Moments Exercise: Please compute all 7 moments for the following shapes, compare the vectors using different comparison techniques Discrete Moments Result: each shape is transformed to a 7dimensional vector. To compare the shapes, compare the vectors (how ?). 3D Distance Histogram Another Example 3D Distance Histogram Shape A,B given as 3D point set 3D Distance Histogram PROJECT ! Feature Based Coding Again: Two operations need to be defined: a mapping of shape into the feature space and a similarity of feature vectors. We hence have TWO TIMES an information reduction of the basic representation, which by itself is already a mapping of the ‘reality’. Representation Feature Extraction Vector Comparison Direct Comparison End of Feature Based Coding ! Next: Direct Comparison Vector Direct Comparison Comparison Example 1 Hausdorff Distance Shape A,B given as point sets A={a1,a2,…} B={b1,b2,…} Feature Vector Based Comparison Coding Hausdorff Vector Comparison Distance Boundary Vector Comparison Representation Hausdorff: Unstable with respect to noise (This is easy to fix ! How ?) Problem: Invariance ! Nevertheless: Hausdorff is the motor behind many applications in specific fields (e.g. character recognition) Boundary Vector Comparison Representation Example 2 Chaincode Comparison Shape A,B given as chaincode Boundary Vector Comparison Representation Boundary Vector Comparison Representation A binary image can be converted into a ‘chain code’ representing the boundary. The boundary is traversed and a string representing the curvature is constructed. 3 2 1 4 C 0 5 6 7 5,6,6,3,3,4,3,2,3,4,5,3,… Boundary Vector Comparison Representation Resulting strings are then compared using classical string-matching techniques. Not very robust. Boundary Vector Comparison Representation Digital curves suffer from effects caused by digitalization, e.g. rotation: Structural Vector Comparison Representation Structural approaches capture the structure of a shape, typically by representing shape as a graph. Typical example: skeletons Structural Vector Comparison Representation Skeletons Shape A,B primarily given as area or boundary, structure is derived from representation Structural Vector Comparison Representation The computation can be described as a medial axis transform, a kind of discrete generalized voronoi. Structural Vector Comparison Representation The graph is constructed mirroring the adjacency of the skeleton’s parts. Edges are labeled according to the qualitative classes. Matching two shapes requires matching two usually different graphs against each other. Structural Vector Comparison Representation Problems of skeletons: - Pruning Structural Vector Comparison Representation -Robustness Vector Shape Comparison similarity All similarity measures shown can not deal with occlusions or partial matching (except skeletons ?) ! They are useful (and used) for specific applications, but are not sufficient to deal with arbitrary shapes Solution: Part – based similarity ! Shape-Representation and Shape Similarity PART 2: PART BASED SIMILARITY Motivation WHY PARTS ? Motivation Motivation Motivation Motivation Global similarity measures fail at: • Occlusion • Global Deformation • Partial Match • (actually everything that occurs under ‘real’ conditions) Parts Requirements for a Part Based Shape Representation (Siddiqi / Kimia ’96: ‘Parts of Visual Form: Computational Aspects’) Parts How should parts be defined / computed ? Some approaches: • Decomposition of interior • Skeletons • Maximally convex parts • Best combination of primitives • Boundary Based • High Curvature Points • Constant Curvature Segments Parts Principal approach: Hoffman/Richards (’85): ‘Part decomposition should precede part description’ => No primitives, but general principles Parts No primitives, but general principals “When two arbitrarily shaped surfaces are made to interpenetrate they always meet in a contour of concave discontinuity of their tangent planes” (transversality principle) Parts “When two arbitrarily shaped surfaces are made to interpenetrate they always meet in a contour of concave discontinuity of their tangent planes” (transversality principle) Divide a plane curve into parts at negative minima of curvature Parts Different notions of parts: • Parts: object is composed of rigid parts • Protrusions: object arises from object by deformation due to a (growth) process (morphology) • Bends: Parts are result of bending the base object Parts The Shape Triangle Parts This lecture focuses on parts, i.e. on partitioning a shape Framework A Framework for a Partitioning Scheme Scheme must be invariant to 2 classes of changes: • Global changes : translations, rotations & scaling of 2D shape, viewpoint,… • Local changes: occlusions, movement of parts (rigid/non-rigid deformation) Framework A general decomposition of a shape should be based on the interaction between two parts rather than on their shapes. -> Partitioning by Part Lines Framework Definition 1: A part line is a curve whose end points rest on the boundary of the shape, which is entirely embedded in it, and which divides it into two connected components. Framework Definition 2: A partitioning scheme is a mapping of a connected region in the image to a finite set of connected regions separated by part-lines. Framework Definition 3: A partitioning scheme is invariant if the part lines of a shape that is transformed by a combination of translations, rotations and scalings are transformed in exactly the same manner. Framework Definition 4: A partitioning scheme is robust if for any two shapes A and B, which are exactly the same in some neighborhood N, the part lines contained in N for A and B are exactly equivalent. Framework Definition 5: A partitioning scheme is stable if slight deformations of the boundary of a shape cause only slight changes in its part lines Framework Definition 6: A partitioning scheme is scaletuned if when moving from coarse to fine scale, part lines are only added, not removed, leading to a hierarchy of parts. Framework A general purpose partitioning scheme that is consistent with these requirements is the partitioning by limbs and necks Limbs and Necks Definition : A limb is a part-line going through a pair of negative curvature minima with co-circular boundary tangents on (at least) one side of the part-line Limbs and Necks Motivation: co-circularity The decomposition of the right figure is no longer intuitive: absence of ‘good continuation’ Limbs and Necks Smooth continuation: an example for form from function • Shape of object is given by natural • • function Different parts having different functions show sharp changes in the 3d surface of the connection Projection to 2d yields high curvature points Limbs and Necks Examples of limb based parts Limbs and Necks Definition : A neck is a part-line which is also a local minimum of the diameter of an inscribed circle Limbs and Necks Motivation for necks: Form From Function • Natural requirements (e.g. space for articulation and economy of mass at the connection) lead to a narrowing of the joint between two parts Limbs and Necks The Limb and Neck partitioning scheme is consistent with the previously defined requirements • • • • Invariance Robustness Stability Scale tuning Limbs and Necks Examples: Limbs and Necks The scheme presented does NOT include a similarity measure ! Algorithms Part Respecting Similarity Measures CSS Curvature Scale Space (Mokhtarian/Abbasi/Kittler) A similarity measure implicitely respecting parts CSS Creation of reflection-point based feature-vector which implicitely contains part – information CSS Properties: • Boundary Based • Continous Model (!) • Computes Feature Vector • compact representation of shape • Performs well ! CSS PROJECT ! CSS Some results (Database: 450 marine animals) CSS The main problem: CSS is continous, the computer vision world is discrete. How to measure curvature in discrete boundaries ? Dominant Points Local curvature = average curvature in ‘region of support’ To define regions of support, ‘dominant points’ are needed ! Dominant Points Dominant Points (“Things should be expressed as simple as possible, but not simpler”, A. Einstein) Idea: given a discrete boundary S compute polygonal boundary S’ with minimum number of vertices which is visually similar to S. Dominant Points Example Algorithms ( 3 of billions…) • Ramer • Line Fitting • Discrete Curve Evolution DCE Discrete Curve Evolution (Latecki / Lakaemper ’99) Idea: Detect subset of visually significant points Curve Evolution Target: reduce data by elimination of irrelevant features, preserve relevant features ... noise reduction ... shape simplification: Curve Evolution: Tangent Space Transformation from image-space to tangent-space Tangent Space: Properties In tangent space... ... the height of a step shows the turn-angle ... monotonic increasing intervals represent convex arcs ... height-shifting corresponds to rotation ... the resulting curve can be interpreted as 1 – dimensional signal => idea: filter signal in tangent space (demo: 'fishapplet') Curve Evolution: Step Compensation (Nonlinear) filter: merging of 2 steps with area – difference F given by: (a-b)pq F= p+q a g b q F F p Curve Evolution: Step Compensation Interpretation in image – space: ... Polygon – linearization ... removal of visual irrelevant vertices p q removed vertex Curve Evolution: Step Compensation Interpretation in image – space: ... Polygon – linearization ... removal of visual irrelevant vertices next: Iterative SC Curve Evolution: Iterative Step Compensation Keep it simple: repeated step compensation ! (demo: EvoApplet) Remark: there are of course some traps ... Curve Evolution: Iterative Step Compensation Remark: there are of course some traps: Self intersection / Topology preservation Stop parameter Edge movement Curve Evolution: Properties The evolution... ... reduces the shape-complexity ... is robust to noise ... is invariant to translation, scaling and rotation ... preserves the position of important vertices ... extracts line segments ... is in accord with visual perception ... offers noise-reduction and shape abstraction ... is parameter free ... is translatable to higher dimensions Curve Evolution: Properties Robustness (demo: noiseApplet) Curve Evolution: Properties Preservation of position, no blurring ! Curve Evolution: Properties Strong relation to digital lines and segments Curve Evolution: Properties Noise reduction as well as shape abstraction Curve Evolution: Properties Parameter free (?) Curve Evolution: Properties Extendable to higher dimensions Curve Evolution: Properties Extendable to higher dimensions Curve Evolution: Properties Extendable to higher dimensions Curve Evolution: Properties Extendable to higher dimensions Curve Evolution: Properties Result: The DCE creates a polygonal shape representation in different levels of granularity: Scale Space Curvature can be defined as the turning angle at the vertices Regions of support are defined by vertices Easy traceable Scale Space is created, since no points are relocated Scale Space Ordered set of representations on different information levels Polygonal Representation The polygonal representation achieved by the DCE has a huge advantage: It allows easy boundary partitioning using convex / concave parts (remember the limbs !) (MATLAB Demo MatchingDemo) DCE Some results of part line decomposition: ASR The ASR (Advanced Shape Recognition) Algorithm uses the boundary parts achieved by the polygonal representation for a part based similarity measure ! (Note: this is NOT the area partitioning shown in the previous slide) ASR / ISS The ASR is used in the ISS Database How does it work ? Behind The Scenes of the ISS Database: Modern Techniques of Shape Recognition and Database Retrieval The 2nd Step First: Shape Comparison ISS implements the ASR (Advanced Shape Recognition) Algorithm Developed by Hamburg University in cooperation with Siemens AG, Munich, for industrial applications in... ... robotics ... multimedia (MPEG – 7) Reticent Proudness… MPEG-7: ASR outperformes classical approaches ! Similarity test (70 basic shapes, 20 different deformations): ASR Curvature Scale Space Multilayer Eigenvector Zernicke Moments Wavelet Contour DAG Ordered Trees (Capitulation :-) Hamburg Univ./Siemens AG Mitsubishi ITE-VIL Hyundai Hanyang University Heinrich Hertz Institute Berlin Mitsubishi/Princeton University IBM 76.45 % 75.44 % 70.33 % 70.22 % 67.67 % 60.00 % --.-- % Requirements Robust automatic recognition of arbitrary shaped objects which is in accord with human visual perception Wide range of applications... ... recognition of complex and arbitrary patterns ... invariance to basic transformations ... results which are in accord with human perception ... applicable to three main tasks of recognition ... parameter-free operation Industrial requirements... ... robustness ... low processing time Requirements Scaling (or resolution) Next: Strategy Rotation shaped objects Robust automatic recognition of arbitrary which is in accord with human visual Rigidperception / non-rigid deformation Wide range of applications... ... recognition of complex and arbitrary patterns ... invariance to basic transformations ... results which are in accord with human perception ... applicable to three main tasks of recognition ... parameter-free operation Industrial requirements... ... robustness ... low processing time Simple Recognition (yes / no) Requirements ... robustness low processing time of arbitrary shaped objects Robust...automatic recognition Rating (best of ...) visual perception which is Common in accord with human Wide range of applications... ... recognition of complex arbitrary patterns Analytical Rating (best of,and but...) ... results which are in accord with human ... invariance to basic transformations perception ... applicable to three main tasks of recognition ... parameter-free operation Industrial requirements... ... robustness ... low processing time Different Approaches Pattern Matching... ... Correlation Geometrical description... ... Hough – Transformation Feature – Vectors... ... (Zernicke - ) Moments Based on Visual Parts... ... Mokhtarian ... ASR ASR: Strategy Source: 2D - Image Object - Segmentation Contour Extraction Evolution Contour – Segmentation Arc – Matching ASR: Strategy DCE Contour – Segmentation Arc – Matching Contour Segmentation Correspondence ? Similarity of parts ? Part Similarity Similarity of parts ? = Boundary Similarity Measure = Similarity of polygons Tangent Space Transformation from image-space to tangent-space Shape Comparison: Measure Tangent space offers an intuitive measure: lg 0 lg1 T g 0 sT g 1 s ads maxlg 0 , lg 1 max l g , l g 1 0 2 1 2 Shape Comparison: Measure Drawback: not adaptive to unequally distributed noise if used globally ! …but works for single parts Shape Comparison: Contour Segmentation Shape Comparison: Correspondence Optimal arc-correspondence: find one to many (many to one) correspondence, that minimizes the arc-measure ! next: Corr. -example Graph of Correspondence arc a0 a3 a2 a1 b0 a0 a1 a2 a3 b0 b1 b2 b3 b3 b2 b1 Graph: ... edge represents correspondence ... node represents matched arcs correspondence Shape Comparison: Correspondence Example: a0 a1 a2 a3 a0 a3 a2 a1 b0 b1 b2 b3 b0 b3 b2 b1 Shape Comparison: Correspondence Result: Optimal correspondence is given by cheapest way next: Corr. - Results Correspondence: Results (MATLAB Demo) Correspondence: Results Correspondence and arc-measure allow... ... the identification of visual parts as well as ... the identification of the entire object ... a robust recognition of defective parts ... a shape matching which is in accord with human perception ASR Results Correspondence and arc-measure meet the requirements stated by Kimia et al. Discrete Easy computable ASR Results Problem: Time Consuming Algorithm (~10ms on 2GHz Pentium) No Metric (-> no clustering) ! How to build a database with this approach ?