Transcript Slide 1

Motion Capture:
Recent Trends
Rama Hoetzlein, 2011
Lecture Notes
Aalborg University at Copenhagen
Facial Capture
Frederick I. Parke, University of Utah
Computer Generated Animation of Faces
1972
Facial Capture
1988 B. Robertson, Mike the Talking Head, Computer Graphics World 11 (7):57
Method 1 – Phonemes
When a particular type of sound is spoken: phonemes
Specific shapes of the whole face are captured. (top down)
Phonemes – the sounds that make up a word, not letters
balloon
b – ah – l – oo – n
Method 2 – Feature Tracking
Parts of the face are tracked separately.
Each part contributes to overall motion. (bottom up)
Motion is the sum of many features.
Works for speech and other facial expressions
Trend – Markerless Facial Capture
Surface is tracked based on image distortion
rather than markers.
Emily, Image Metrics, 2010
http://www.youtube.com/watch?v=UYgLFt5wfP4&feature=player_embedded
Problem:
Motion capture records the body over volumes up to:
10 x 10 sq. meters (30 sq. ft)
Facial capture records subtle details over space of:
30 x 30 cm (1 sq. ft)
How to capture both the large-scale motion of the body
and subtle motion of the face during a single performance?
Trend –
Performance capture
is a collection of
techniques that
combine to
record the total
motion of an actor.
Avater (2010),
James Cameron
Solution:
Block off the face using individual, head-mounted cameras, which
record only the face. Use motion cameras and passive markers for the body.
Allows for both large volumes and small details.
Markers include:
Body capture
Facial capture
Hair capture
Green lines, white dots
Head-mounted device, /w camera booms
Blue and red ropes
Hardware Trends
Trend – Markerless capture:
Origins in 3D laser scanning
3D Lego Digitizer
http://www.rchoetzlein.com/project/digitizer/
Trend – Markerless capture:
Structured Light
Q: High frequency gives details about
height of point. But how do we tell if the
point is on left or right side of obj?
Faster: Do all lines at once
Projector with structured light
mapped onto the object. Use two
cameras to determine object
structure.
Structured light can be linear,
binary coded, gray coded, or color
coded. The encoding allows you to
uniquely identify points.
Light may be infrared (Kinetic).
A: Low frequency gives overall
characteristics of pixels.
Point
cloud
Volume
construction
No markers.
Structured light creates a point
cloud.
Skeleton is fit inside point cloud
from root joints to extremities.
Torso defines primary orientation,
and also constraints placement of
next joint layer in hierarchy.
Fit
torse
Fit
extremities
Trend – Markerless capture:
Direct-to-3D models
Performance Capture from Sparse Multi-view Video, SIGGRAPH 2010
Christian Theobalt, Stanford University
http://www.youtube.com/watch?v=dTisU4dibSc&playnext=1&l
ist=PLD31C3C36D294EEDB
http://www.stanford.edu/group/biomotion/Markerless.html
Trend – Monocular capture
One camera, without depth,
is under-constrained.
However, the human body
has fixed limb lengths and
ratios.
Use the body ratios as an
additional constraint.
Fabio Remondino, Andreas Roditakis
Institute for Geodesy and
Photogrammetry - ETH Zurich,
Switzerland
3D Reconstruction of Human
Skeleton from Single Images or
Monocular Video Sequences
2003, 25th Pattern Recognition
Symposium
Trend – Low Cost Systems
Cheap hardware: Microsoft Kinect, Web cameras.
Open source software:
OpenKinect
libfreenect
OpenNI
FaceAPI
open kinect drivers
open kinect drivers
skeleton fitting
facial tracking
Main challenges: 1) Integration into existing frameworks,
2) Usually requires programming experience
3) Can be difficult to modify for research
Software Trends
Motion Graphs
Motion Graph:
A database of motion capture clips,
connected to one another to represent transitions
between actions.
Motion graphs can be represented by a finite state machine,
a set of states with edges representing state transitions.
Stand
Run
Jump
Trend – Motion Graphs in Gaming
Planning and Directing Motion Capture For Games
Melianthe Kines, Gamasutra. January 19, 2000
http://www.gamasutra.com/view/feature/3420/planning_and_directing_motion_.php
What are the advantages and disadvantages of
motion graphs for gaming?
Advantages
1. Fast. Motion is simply played back from pre-recorded data.
2. Interactive. Motion can be changed immediately by
transitioning to a different state.
3. Modular. Different motions can easily be swapped in.
4. Extensible. More states can be added to the graph.
Disadvantages
1. Jump transitions between capture clips
2. Motion may not match scene exactly. e.g. jump over chasm
3. Cannot grasp objects accurately. No inverse kinematics.
4. Cannot move in any direction
5. Interruptions from outside forces not easily handled
Trend – Motion Blending in Gaming
“In order to create streams of high-quality
motion, current applications [games]
assemble static clips of motion created
with traditional animation techniques such
as motion capture or keyframing. The
assembly process requires making
transitions between motions. These
transitions may be difficult to create, such
as a transition between a running clip and
one where the character is lying down, or
trivial, if the end of one clip is identical to
the beginning of the next. In practice,
simple techniques such as linear blends
are capable of creating transitions in cases
where the motions are similar.”
Michael Gleicher, Hyun Joon Shin,
Lucas Kovar, Andrew Jepsen
Snap-Together Motion:
Assembling Run-Time Animations
Interactive 3D Graphics 2003
Common solutions in Gaming:
1. Jump transitions
Linear blending between motion clips
2. Motion may not match
scene exactly
Blend with scene constraints
(extend jump over river)
3. Cannot grasp objects
Add inverse kinematics to
arms in game characters.
4. Cannot move in any
direction
Add steering.
Simple: re-orient, then play walk cycle
Advanced: add IK to legs
5. Interruptions from
outside forces
Use a rag-doll physics switch.
When object hits..
Turn on physics, apply force.
Trend – Motion Graphs
How would you instruct a character to follow an arbitrary path
using a set of pre-recorded captured motion?
Lucas Kovar, Michael Gleicher, Frederic Pighin.
Motion Graphs, SIGGRAPH 2002.
Trend – Motion Graphs
How do we make energy optimal
motion based on several,
arbitrary constraints?
Uses motion capture data, but
in arbitrary, non-acted scenarios.
Alla Safonova Jessica K. Hodgins,
Carnegie-Melon University.
Construction and optimal search
of interpolated motion graphs
SIGGRAPH 2007
INPUT
Physical
capture /
Haptics
Overview
OUTPUT
Post processing
(cleaning)
Joint
data
Animation
Skinning
Secondary motion
Marker
data
Blending
Re-targeting
Motion
capture
Monoccular
video
3D model
capture
Sequencing
Optimization
Skeleton
fitting
Point
clouds
Motion graphs
(e.g Gaming)
Performance
capture
Facial
capture