Stochastic Systems Group

Some Rambling (Müjdat) and Some More Serious Discussion (Ayres, Junmo, Walter) on

Shape Priors

Stochastic Systems Group

Desire to use Shape Priors in Segmentation

• The posterior: Segmenting curve Observed image data • The most common (implicit) prior used is the curve length penalty: • Want to be able to use better prior models

Stochastic Systems Group

Challenges and remarks

• Need probabilistic descriptions in the space of shapes – A non-linear, infinite-dimensional manifold – Distance (similarity) measures in the shape space • Of course, a statistical description for shapes has uses other than segmentation as well – Sampling from a shape density – Recognition of objects – Completion of incomplete shapes

Stochastic Systems Group

The PCA World

• Cootes & Taylor – Shape representation using marker points • Leventon, Tsai – PCA and level sets • More… • Want a more principled approach

• • • • • • • • • Stochastic Systems Group

Some Literature

“ Active shape models - their training and application ,” T. Cootes, C. J. Taylor, D.H. Cooper, J. Graham, 1995.

“ Embedding Gestalt laws in Markov random fields ,” Song-Chun Zhu, 1999.

“ On the incorporation of shape into geometric active contours ,” Y. Chen, H.D. Tagare et al., 2001.

“ Image segmentation based on prior probabilistic shape models ,” A. Litvin and W. C. Karl, 2002.

“ Shape priors for level set representations ,” M. Rousson and N. Paragios, ECCV, 2002.

“ Nonlinear shape statistics in Mumford-Shah based segmentation ,” D. Cremers, T. Kohlberger, and C. Schnorr, 2002.

“ Geometric analysis of constrained curves for image understanding ,” A. Srivastava, W. Mio, E. Klassen, X. Liu, 2003.

“ Analysis of planar shapes using geodesic paths on shape spaces ,” E. Klassen, A. Srivastava, and W. Mio (in review) “ Gaussian distributions on Lie groups and their application to stat. shape analysis ,” P. T. Fletcher, S. Joshi, C. Lu, and S. Pizer, 2003.

Stochastic Systems Group

Overview of Anuj Srivastava’s Work

• Specify a space of continuous curves with constraints (e.g. simple closed) and exploit the differential geometry of this space • Use geodesic paths for deformations between curves and to compute distances – do not have analytical exps. for geodesics • To move in the shape manifold, first move in a linear space and then project back (using tangents/normals) • Given two curves, solve an optimization problem to find the geodesic path between them (find a local min) • Build statistical descriptions based on Karcher means; covariance of (Fourier coefficients of) tangent vectors • Use such descriptions as priors (very preliminary) • Some current limitations: – Cannot handle topological changes – Extension to surfaces in 3-D not straightforward

Stochastic Systems Group

Outline

• Some metrics proposed for shape similarity (Junmo) • Anuj Srivastava’s work – Geometric representations of curves and shape spaces (Walter) – Tangents, normals, geodesics (Ayres) – Statistical models and application to segmentation (Junmo) • Brief highlights from Pizer’s work (Walter) • Discussion on all of this

Stochastic Systems Group

Some Metrics on the Space of Shapes

• Notion of “similarity” between shapes – Basic task of vision system is to recognize similar objects which belong to the same category.

• Describing shapes : landmarks, level set • Shape can be described as • There are several metrics for two shapes

Stochastic Systems Group

Hausdorff Metric

• Given • Very sensitive to any outlier points in and

Stochastic Systems Group

Template Metric

• • Totally insensitive to outliers

Stochastic Systems Group

Transport Metric

• Fill with ‘stuff’ and find the shortest paths along which to move this ‘stuff’ so that it now fills

Stochastic Systems Group

Optimal Diffeomorphism

• If and are topologically different, the distance would be infinite.

– E.g. is minus a pinhole – E.g. A small break cuts a shape into two

Stochastic Systems Group

Geometric Representations of Curves and Shape Spaces

• Restrict attention to closed curves in R

2

.

• Classify curves which differ only by orientation preserving rigid motions (rotation and translation) and uniform scaling as the same shape.

• Consider two different representations of planar curves for simple closed curves: – Using direction functions.

– Using curvature functions.

Stochastic Systems Group

Preliminaries

• First, the issue of scaling is resolved by fixing the length of all curves to 2  .

• Curves are parameterized by arc length with period 2  & • Define the unit tangent function where S 1 is the unit circle, & • where  (s) is the direction function.

 (s+2  )  (s) = 2  n (n = rotation index { = 1})

Stochastic Systems Group

Curves to Consider

• Consider entire set of curves with rotation index 1 because this set is complete.

– Contains its own limit points.

• Note that the set of simple closed curves is an open subset of this larger set.

Stochastic Systems Group

Shape Using Direction Functions

• Direction function for S 1 is  0 (s) = s. All other closed curves have direction function  =  0 + f, where f is L

2

periodic on [0,2  ].

• To adjust for rigid rotations, restrict attention to  s.t. • To ensure curve closure, require

Stochastic Systems Group

More Direction Functions

• Define a map by • The pre-shape space C

1

is  1 -1 (  ,0,0).

• Multiple elements of C

1

may denote the same shape. An adjustment of the reference point (s=0) handles this.

Stochastic Systems Group

Geometry of Shape Manifolds

• Constraints define a manifold embedded in  0 + L 2 • Move along manifold by moving in tangent space and projecting back to manifold • Tangent space is infinite dimensional, but normal space is characterized by three constraints defined in f 1

Stochastic Systems Group

Tangents and Normals

• The derivative of f 1 in the direction of f at  is: • Implies d f 1 is surjective • If f is orthogonal to {1, sin  , cos  }, then d f 1 =0 in the direction of f and hence f is in the tangent space

Stochastic Systems Group

Projections

• Want to find the closest element in C 1 arbitrary    0 + L 2 to an • Basic idea: move orthogonal to level sets so projections under f form a straight line in R 3 • For a point b  R 3 , we define the level set as: • Let b 1 =(  ,0,0). Then its level set is the preshape space C 1

Stochastic Systems Group

Approximate Projections

• If points are close to C 1 , then one can use a faster method • Let d  be the normal vector at  for which f (  +d  )=b 1 . Can do first order approximation to compute this • Approximate Jacobian as:

Stochastic Systems Group

Iterative algorithm

• Define the residual (error) vector as • Then: where • Iteratively update  + d   zero until the error goes to • Call this projection operator P

Stochastic Systems Group

Example Projections

Fig. 1: Projections of arbitrary curves into C 1

Stochastic Systems Group

Geodesics

• Definition: For a manifold embedded in Euclidean space, a geodesic is a constant speed curve whose acceleration vector is always perpendicular to the manifold • Define the metric between two shapes as the distance along the manifold between the shapes with respect to the L 2 inner product • Nice features: – Defined for all closed curves – Interpolants are closed curves • Finds geodesics in a local sense, not necessarily global

Stochastic Systems Group

Paths from initial conditions

• Assume we have a  tangent space in C 1 and an f in the • Approximate geodesic along manifold by moving to  +f D t and projecting that back onto the manifold ( D t is step size) • So  (t+ D t) = P(  (t) +f (t) D t)

Stochastic Systems Group

Transporting the tangent vector

• Now f (t) is not in the tangent space of  (t+ D t) • Two conditions for a geodesic: – The acceleration vector must be perpendicular to the manifold: simply project f into the next tangent space – The curve must move at constant speed: renormalize so ||f (t+1) ||=||f (t) || • h k is the orthonormal basis of the normal space

Stochastic Systems Group

Geodesics on shape spaces

• S 1 is a quotient space of C 1 isometries, so finding geodesics in S 1 finding geodesics in C orbits 1 under actions of S 1 by equivalent to which are orthogonal to S 1 • S 1 acting by isometries implies that if a geodesic in preshape space is orthogonal to one S 1 orbit, it’s orthogonal to all S 1 orbits which it meets • So now normal space has one additional component spanned by • The algorithm is the same as detailed earlier except with an expanded normal space

Stochastic Systems Group

Geodesics between shapes

• We know how to generate geodesic paths given  and f • Now we want to construct a geodesic path from  1 to  2 • So we need to find all f that lead from  1 to an S 1 orbit of  2 in unit time, and then choose the one that leads to the shortest path • Let Y define the geodesic flow, with Y (  1 ,0,f)=  1 the initial condition • We then want Y (  1 ,1,f)=  2 as

Stochastic Systems Group

Finding the geodesic

• Define an error functional which measures how close we are to the target at t=1: • Choose the geodesic as the flow Y the smallest initial velocity ||f|| which has • i.e., min ||f|| s.t. H[f]=0 • Hard because infinite dimensional search

Stochastic Systems Group

Fourier decomposition

• f  L 2 , so it has a Fourier decomposition • Approximate f with its first m+1 cosine components and its first m sine components: • Let a be the vector containing all of the Fourier coefficients • Now optimization problem is min ||a|| s.t. H[a]=0

Stochastic Systems Group

Geodesic paths

Fig. 2: Geodesic paths between two shapes

Stochastic Systems Group

Statistical Modeling of Shapes

• Given example shapes – Mean shape – Shape variation – Shape prior – Sampling from the prior – Using shape prior for segmentation of occluded images

Stochastic Systems Group

Mean Shape

• Given the geodesic distance function , • Karcher mean of shapes be a shape is defined to for which the variance function is a local minimum • The Karcher mean exists, but may not be unique

Stochastic Systems Group

Variances on Shape Spaces

• Model the variation from the mean shape as , an element of the tangent space at the mean shape • Represent by its Fourier expansion: • Model as multivariate normal with mean 0 and covariance matrix

Stochastic Systems Group

Shape Sampling

• Sample Fourier coefficients of tangent vectors from the multivariate normal distribution • Move along the geodesic path starting from the mean shape in the direction of by distance

Stochastic Systems Group

Shape Sampling Examples

Observed shapes Mean shape Random samples from the Gaussian model

Stochastic Systems Group

Shape Prior

• : the space of curves (larger than shape space) • – can be represented as pairs : parameters for translation, rotation, and scaling – : the shape • Gaussian density with a mean shape with the shape dispersion

Stochastic Systems Group

Bayesian Discovery of Objects

Stochastic Systems Group

Some closing thoughts on Anuj Srivastava’s work

• Most energy has been spent on manipulating the shape manifold • Work on using these models as priors preliminary – Non-diagonal covariance – explore modes of variation – Non-Gaussian?

– Mean in manifold?

• Other thoughts – Non-Fourier representations for tangents – KL expansion?

– Could similar ideas be used with representations based on signed distance functions?

Stochastic Systems Group

Overview of Steve Pizer’s Work

• Medial representations (m-reps) are used to model the geometry of anatomical objects.

• Medial parameters are not in a Euclidean space; so, PCA cannot be used. However, m-reps model parameters are elements of a Lie group.

• Gaussian distributions on this Lie group are considered, with the max likelihood estimates of mean and covariance derived.

• Similar to PCA for Euclidean spaces, principal geodesic analysis (PGA) on Lie groups are defined for the study of anatomical variability.

• Framework is applied to hippocampi in a schizophrenia study.

– 86 m-rep figures are first aligned (translation/rotation/scaling) – Intrinsic mean is then computed – PGA (modes of variability) are then computed – Results yield smoother deformations when compared with PCA

Stochastic Systems Group

Medial Representation

• Introduced by Blum (1978), a 3-D object is represented by a set of connected continuous medial manifolds formed by the centers of all spheres are are interior to the object and tangent to the object boundary at two or more points. The figure below illustrates this:

Stochastic Systems Group

Medial Atom & Lie Groups

• A medial atom is represented by – The location in space (R 3 ) – The radius of the sphere (R + ) – The local frame (SO(3)) – The object angle (SO(2)) • R 3 is a Lie group under vector addition, R + is a multiplicative Lie group, and SO(2) & SO(3) are Lie groups under composition of rotations.

• The direct product of Lie groups is a Lie group.

Stochastic Systems Group

Lie Groups and Lie Algebras

• A Lie group is a group G that is a finite-dim manifold such that the two group operations of G, multiplication and inverse, are C 2 mappings.

• If e is the identity of G, the tangent space at e forms a Lie algebra. The exponential map provides a method for mapping vectors in the tangent space into the Lie group.

Stochastic Systems Group

Alignment and PGA

• Translation: each model is situated so that the average of its medial atoms is at the origin • Rotation and scaling are done in a manner which minimizes the total sum-of-squared distances between m-rep figures.

• After alignment, principal directions in the geodesic are computed, and the analog to PCA is performed.

Stochastic Systems Group

Results

• The analysis, performed on 86 aligned hippocampus m-reps, shows smooth deformations (compared with PCA). The mean shape is top left, the medial atoms are overlaid lower left, and the first 3 PGA modes are shown right.

Stochastic Systems Group

Shape Using Curvature Functions

• Alternatively, curves can be represented by curvature fcns. Since the rotation index is 1, • Using , the closure condition

Stochastic Systems Group

More Curvature Functions

• Define a map by then the pre-shape space C

2

is  2 -1 (2  ,0,0).

• As with direction functions, different placements of s=0 result in different C

2

shapes.

Thus, re-parameterization is needed.