Transcript Slides
Modeling 3D Deformable and
Articulated Shapes
Yu Chen, Tae-Kyun Kim, Roberto Cipolla
Department of Engineering
University of Cambridge
Roadmap
Brief Introductions
Our Framework
Experimental Results
Summary
Motivation
Tasks:
– To recover deformable shapes from a single
image with arbitrary camera viewpoint.
3D Shapes
+
2D Images
Uncertainty
Measurements
Previous Work
Rigid shapes [Prasad’05, Rother’09, Yu’09, etc.]
Problems:
– Cannot handle self-deformation or articulations.
Category-specific articulated shapes
e.g., human bodies [Anguelov’05, Balan’07, etc.]
Problems:
– Requiring strong shape or anatomical knowledge of the
category, such as skeletons and joint angles.
– Too many parameters to estimate;
– Hard to be generalised to other object categories.
Roadmap
Brief Introductions
Our Framework
Experimental Results
Summary
Our Contribution
A probabilistic framework for:
– Modelling different shape variations of
general categories;
– Synthesizing new shapes of the category
from limited training data;
– Inferring dense 3D shapes of deformable or
articulated objects from a single silhouette;
Explanations on the Graphical Model
Pose
Generator
Shape
Generator
Shape Synthesis
Joint Distribution:
Matching Silhouettes
Generating Shapes
Target: Simultaneous modelling two types
of shape variations:
– Phenotype variation:
fat vs. thin, tall vs. Short...
– Pose variation:
articulation, self deformation, ...
Training two GPLVMs:
– Shape generator (MS) for phenotype variation;
– Pose generator (MA) for pose variation.
Generating Shapes
Shape Generator (MS)
– Training Set:
Shapes in the canonical pose.
– Pre-processing:
Automatically register each instance with a common
3D template;
3D shape context matching and thin-plate spline
interpolation;
Perform PCA on all registered 3D shapes.
– Input:
PCA coefficients of all the data.
Generating Shapes
Pose Generator (MA)
– Training Set:
Synthetic 3D poses
sequences.
– Pre-processing:
Perform PCA on both spatial
positions of vertices and all
vertex-wise Jacobian
matrices.
– Input:
PCA coefficients of all the
data
Shape Synthesis
VA
Pose
Generator
MA
VA
Shape
Synthesis
V
V
Zero
Shape
V0
VS
VS
Shape
Generator
MS
Shape Synthesis
Modelling the local shape transfer
– Computing Jacobian matrices on the
zero shape vertex-wisely.
Ji
Shape Synthesis
Synthesizing fully-varied shape V from
phenotype-varied shape VS and posevaried shape VA.
Probabilistic formulation: a Gaussian
Approximation
Matching Silhouettes
A two-stage process:
o
Projecting the 3D shape onto the image plane
o
Chamfer matching of silhouettes
Maximizing likelihood over latent coordinates xA,
xS and camera parameters γk
o
o
Optimizing the closed-form lower bound.
Adaptive line-search with multiple initialisations.
Roadmap
Brief Introductions
Our Framework
Experimental Results
Summary
Experiments on Shape Synthesis
Task:
– To synthesize shapes in different phenotypes
and poses with the mean shape μV.
Shape Synthesis: Demo
Shape Generator
Pose Generator
(Running)
Shape Synthesis: Demo
Shape Generator
Pose Generator
(Running)
Shape Synthesis: Demo
Shape Generator
Pose Generator
(Running)
Shape Synthesis: Demo
Shape Generator
Pose Generator
(Running)
Shape Synthesis: Demo
Shape Generator
Pose Generator
(Running)
Shape Synthesis: Demo
Shape Generator
Pose Generator
(Running)
Shape Synthesis: Demo
Shape Generator
Pose Generator
(Running)
Shape Synthesis: Demo
Shape Generator
Pose Generator
(Running)
Shape Synthesis: Demo
Shape Generator
Pose Generator
(Running)
Shape Synthesis: Demo
Shape Generator
Pose Generator
(Running)
Experiments on Single View Reconstruction
Training dataset:
– Shark data:
MS: 11 3D models of different shark species .
MA: 11-frame tail-waving sequence from an animatable 3D
MEX model.
– Human data:
MS: CAESAR dataset.
MA: Animations of different 3D poses of Sydney in Poser 7.
Testing:
– Internet images (22 sharks and 20 humans in different
poses and camera viewpoints)
Segmentation: GrabCut [Rother’04]
Experiments on Single View Reconstruction
Sharks:
Experiments on Single View Reconstruction
Humans:
Experiments on Single View Reconstruction
Examples of multi-modality
Experiments on Single View Reconstruction
Qualitative Results: Precision-Recall Ratios
– SF: foreground regions
– SR: image projection of our result
A very good approximation to the results
given by parametrical models
Roadmap
Brief Introductions
Our Framework
Experimental Results
Summary
Pros and Cons:
Advantages
Fully data driven;
Requiring no strong classspecific prior knowledge,
e.g., skeleton, joint angles;
Capable of modelling
general categories;
Compact shape
representation and much
lower dimensions for
efficient optimization;
Uncertainty measurements
provided.
Disadvantages
Inaccurate at fine parts,
e.g., hands.
Lower descriptive power
on poses compared with
parametric model, when
training instances are not
enough;
Training data are
sometimes difficult to
obtain.
Future Work
A compatible framework which allows
incorporating category knowledge
Incorporating more cues: internal edges,
texture, and colour;
Multiple view settings and video
sequences;
3D object recognition and action
recognition tasks.
Thanks!