Transcript Learning a correlated model of identity and pose

Learning a correlated model of identity
and pose-dependent body shape variation
for real-time synthesis
Brett Allen1,2, Brian Curless1, Zoran Popović1, and Aaron Hertzmann3
1
University of Washington
2 Industrial Light & Magic
3 University of Toronto
Motivation
movies
telepresence
games
design
Goal
• We would like to be able to generate body
models of any individual in any pose.
Identity
+
pose
shape
- want to synthesize models in real-time
- model should be learnable from real data
Data
• CAESAR data set: 44 subjects in 2 poses
• Multi-pose data set: 5 subjects in 16 poses
• Dense-pose data set: 1 subject in 69 poses
[Anguelov et al. 2005]
Related work
Anatomical methods
Chadwick et al. 1989
Turner and Thalmann 1993
Scheepers et al. 1997
Wilhelms and Van Gelder 1997
…
Aubel 2002
Example-based methods
 x0 
 y 
 0 
 z0 


v  
 x7000 


 y7000 
z 
 7000 
v  f (p, Φ)
v = shape vector
p = example parameters
 = function parameters
Given:
n examples v0…vn and
n sets of parameters p0…pn (optional)
find .
Scattered data interpolation
columns of  = key shapes
i = reconstruction weights from applying
k-nearest neighbors or RBFs on p
f (p, Φ)  iφi
i
1
1
3
2
2
4
3
Allen et al. 2002
4
Lewis et al. 2000
Sloan et al. 2001
Kry et al. 2002
An aside on enveloping
Enveloping + scattered data interpolation
= “corrective enveloping”
Latent variable models
f (x,{W, v})  Wx v
x = latent variable (component weights)
W = components in columns
v = average shape
Allen et al 2003
Seo et al 2003
Anguelov et al 2005
…
Blanz & Vetter 1999
Pose variation vs body variation
Sloan et al. 2001
Pose variation vs body variation
v  Wx  v  Kω
Anguelov et al. 2005
Our approach
c  W x c
 φ1 
 
c 
φ k 
 
b
v   i φ i
Intrinsic skeleton
parameters: bone
lengths and
carrying angles
i
c = “character vector”: all information
needed to put a character in any pose
v = shape in a particular pose
Two Problems
1. Scans might not be at “key” poses
2. Scans are not complete
Maximize: p(c | {e()})
…actually, we don’t know the pose or skinning
weights either:
Maximize: p(c, s, q{} | {e()})
Maximum a posteriori estimation
P  p(c, s,{q } | {e ( ) })
 n  nv

( )
P    p(ei | c, s, q )  p(c) p(s) p({q })
  1  i 1
 
n
nv
 log P  n nv 1.5 log(2 )  
2
v
 1 i 1
1
2
2
v
e
( )
i
 log p(c)  log p(s)  log p({q })
v
( ) 2
i
Results
Going to multiple characters
• One possibility: Learn several character
vectors separately, then run PCA.
• Two problems:
– the character vector contains values that have
different scales (rest positions, offsets, bone
lengths)
– we don’t have enough data!
Identity variation
c   W x  c
ei( ,  )  g (c  , s, q )i 
x  ~ N (0, I);  ~ N (0, 2 )
c is the character vector of the th example person
g(c,s,q) applies skinning and pose space deformations
Alternating optimization
• We initialize the {x} with the weights from
running PCA on estimated skeleton
parameters.
• We then optimize W, c, s, q.
• Then we optimize for {x}.
• Repeat…
Results
Results
(video)
Conclusions
We present a flexible approach for learning body
shape variation between individuals and between
poses, including the interrelationship between the
two.
+ very general: can handle irregular and
incomplete sampling in regard to both the
poses/identities scanned, and in the surfaces
themselves
+ the learned model can generate body shapes
very quickly (over 75 fps)
Limitations
• You need a lot of data! Our data set was too
sparse in some areas.
• Some poses are hard to capture.
• It’s very hard to compensate for the
skinning artifacts.
• The shape matching could be improved
(high-frequency details are lost if the
matching is poor).
Acknowledgements
•
•
•
•
•
•
•
UW Animation Research Labs
Washington Research Foundation
National Science Foundation, NSERC, CFI
Microsoft Research, Electronic Arts, Sony, Pixar
Kathleen Robinette and the AFRL lab
Dragomir Anguelov
Domi Pitturo