Transcript W - Geometric Algorithms for Modeling, Motion, and Animation

MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Motion Capture (Mocap) and Motion
Data Related Technologies
Tao Yu
Comp 790-058 – Robot Motion Planning
Fall 2007
September 24, 2007
OUTLINE
• Mocap Overview
• Representation of Motion
• Motion Signal Processing
• Motion Synthesis from Examples
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Mocap
Overview
What is motion capture
“Recording of motion for immediate or detailed
analysis and playback”
» David J Sturman, “A Brief History of Motion
Capture for Computer Character Animation”,
Character Motion Systems, SIGGRAPH 94,
Course 9
“The creation of a 3d representation of a live
performance”
» Alberto Menache, Understanding Motion
Capture for Computer Animation and Video
Games
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Mocap
Overview
Root of motion capture
• Eadweard Muybridge (1830-1904)
• Etienne-Jules Marey (1830-1904)
• Harold Edgerton (1903-1990)
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Eadweard Muybridge (1830-1904)
Mocap
Overview
http://www.cotianet.com.br/photo/great/Muybridge.htm
•
•
•
•
http://en.wikipedia.org/wiki/Image:The_Horse_in_Motion.jpg
“Father of the motion picture”
several cameras – successive
pictures
photographs of human and animal
motion
zoopraxiscope (zoogyroscope,
zoetrope) – a device for playing
still images in sequence
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Mocap
Overview
Etienne-Jules Marey (1830-1904)
http://www.inrp.fr/Tecne/Acexosp/Actimage/Images/Marey2.jpg
http://www.nrw-forum.de/img_ausst/img_press/Marey.jpg
• birds
• photographic gun
MOTION PLANNING
IN REAL AND VIRTUAL ENVIRONMENTS
http://www.rickwisedp.com/St%20Marys/COMM%20158/images/Marey%20Photo%20
FALL 2007
Mocap
Overview
Harold Edgerton (1903-1990)
• high speed and
stop motion
photography
• exposures as
small as a
millionth of a
second
• electronic flash
• stroboscope
http://www.personal.psu.edu/users/a/r/ark176
MOTION
PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
http://www.personal.psu.edu/users/a/r/ark176/Assignment%204.htm
FALL 2007
Rotoscoping
Mocap
Overview
• Allowed animators to trace cartoon
characters over photographed frames of live
performances.
• Invented in 1915 by Max Fleischer
• Koko the Clown
• Snow White
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Mocap
Overview
Rotoscoping
“rotoscoping can be thought of as a
primitive form or precursoro to motion
capture, where the motion is ‘captured’
painstakingly by hand.” - Sturman
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Mocap
Overview
“modern” era of mocap, 1970’s-present
•
•
•
•
•
•
•
more players
commercial players
multiple uses
70’s: development of magnetic systems
80’s: development of optical systems
90’s: mocap is hot,
00’s: mocap is used more frequently for
feature films
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Mocap
Overview
Overview: Motion capture systems
Types of Mocap systems:
• Outside-In
» sources (e.g., reflective markers) on body
» external sensors (e.g., cameras)
» optical systems
• Inside-Out
» sensors on body
» external sources
» magnetic systems
• Inside-In
» sources and sensor on body
» mechanical systems
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Mocap
Overview
implementation of a motion
capture system
•
•
•
•
prosthetic
acoustic
magnetic
optical
Liverman, The Animator’s
Motion Capture Guide
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Mocap
Overview
mechanical/prosthetic capture
• Inside-In
• external structure
attached to
performer
• structure detects
changes
» optic
» mechanical
Gypsy4
cyberglove
http://infolab.usc.edu/more/facilitiesimages/glove.jpg
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Mocap
Overview
mechanical/prosthetic capture
• advantages
» computes rotations directly
» portable, unlimited range
» less expensive
» can capture multiple performances
simultaneously
» no built in positional reference
• disadvantages
» external structure unwieldy
» cannot change the configuration, i.e., a hand
capture can’t be used for an arm
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Mocap
Overview
acoustic capture
•
•
•
Inside-Out
still developing
Active Markers –
» transmitters are attached to the performer and sequentially emit
audio signal, a “click”
•
•
receivers measure the time to receive the signal, triangulate and
compute the position of the transmitter
advantages
» no occlusion
•
disadvantages
»
»
»
»
cables unwieldy
rate of transmission not high enough to support enough transmitters
size of the capture area is limited
sound reflections can reduce accuracy
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Mocap
Overview
optical capture (passive)
• markers are attached to a
performer
» passive reflective
markers
» more on active markers
later
• a system of cameras
record the position of the
markers
» 2-many cameras
http://cgm.cs.mcgill.ca/~athens/cs644/Projects/2004/MichelLanglois-PhilippeKun
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Mocap
Overview
optical capture (passive)
•
advantages
» freedom of movement
» high quality capture
» high throughput
•
•
•
•
•
fast sampling (200 fps at a high resolution)
can capture fast motions
can have a large capture space
can capture many markers
disadvantages
» occlusion, markers are can be hidden from the camera
• additional performers will increase occlusion
• may be able to add redundant cameras
» marker crossover, which marker are you looking at?
» cost $$$
» extensive post processing (the marker’s have to be located and
identified)
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Mocap
Overview
major optical players
Motion Analysis
ViconPeak
•
http://www.motionanalysis.com/
•
•
Films
» Lord of the Rings
» King Kong
» Matrix
» Final Fantasy
Games
» NBA Live 2004
» Grand Theft Auto III
» Mortal Kombat 4 (Midway)
•
•
•
http://www.viconpeak.co
m/
Films
» Polar Express
» Harry Potter and the
Prisoner of Azkaban
» The Hulk
» Spider Man
Games
» All-Star Baseball
2002
» Buffy the Vampire
Slayer
» Everquest II
» NHL 2K3 (Mocap by
Red Eye Studio)
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Mocap
Overview
magnetic capture
•
•
•
Outside/In
receivers are attached to the performer’s body
compute the position and orientation from receiver to central
magnetic transmitter
» AC and DC systems
•
•
originally developed for targeting in tactical military aircraft
advantages
» no occlusion
•
disadvantages
»
»
»
»
cabling can interfere with movement (improvements?)
capture area can be limited by transmitter
metal in vicinity can interfere with system
capture volume can be limited
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Mocap
Overview
Mocap pipeline
•
•
•
•
•
Planning for capture
Setup and calibrate system
Capture performer, obtain marker positions
Retarget to a character
Edit/cleanup
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Mocap
Overview
planning
•
•
•
•
•
motion capture requires serious planning
anticipate how the data will be used
garbage in garbage out
shot list
games
» motions need to be able to blend into one an
another
» capture base motions and transitions
» which motions transition into which other
transitions
» cycles/loops
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Mocap
Overview
movement flowchart for games
•Planning and Directing Motion Capture For Games
By Melianthe Kines
Gamasutra
January 19, 2000
URL: http://www.gamasutra.com/features/20000119/kines_01.htm
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Mocap
Overview
processing passive markers
•
•
each camera records capture session
extraction: markers need to be identified in
the image
» determines 2d position
» problem: occlusion, marker is not
seen
• use more cameras
•
markers need to be labeled
» which marker is which?
» problem: crossover, markers
exchange labels
• may require user intervention
•
compute 3d position: if a marker is seen
by at least 2 cameras then its position in
3d space can be determined
http://www.xbox.com/NR/rdonlyres/3164D1BE-C1C4-46A1-90F0-26507CF2C9BD/0/ilmnflfever2003lightscam001.jpg
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Mocap
Overview
retargeting: the character
• the character is
controlled by
skeleton
• to control the
skeleton, need to
specify joint
rotations
• muscles?
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Mocap
Overview
retargeting
•
•
capture motion on
performer
» positions of
markers are
recorded
retarget motion on a
virtual character
» motion is usually
applied to a
skeleton
» a skeleton is
hierarchical
• linked joints
•
» need rotation
data!
need to convert
positions to rotations
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Mocap
Overview
performer→ actor → character
•
•
•
the actor is used to convert
marker positions to rotational
data
» markers are handles on
the actor
» actor should have similar
proportions as the
performer
joint rotations of the actor are
applied to the character
there are still issues with
proportions
Alias Motionbuilder: actor and markers
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Mocap
Overview
retargeting problems
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Mocap
Overview
retargeting problems
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Mocap
Overview
Applications
•
•
•
•
Entertainment
Medicine
Arts / Education
Science / Engineering
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Mocap
Overview
Entertainment: Live Action Films
• Computer generated characters in live action films
(e.g. Battle Droids and many others in Star Wars
Prequels, Gullum in The Lord of the Rings, King
Kong in King Kong , Davy Jones in Pirates of
Caribbean)
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Mocap
Overview
Entertainment: 3D computer
animations
• Characters in computer animated files (e.g. Polar
Express, Monster House)
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Mocap
Overview
Entertainment: Video Games
• Video games by Electronic Arts, Gremlin, id,
RARE, Square, Konami, Namco, and others,
(e.g. Enemy Territory, Devil May Cry)
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Mocap
Overview
Medicine
• Medicine (e.g., gait analysis, rehabilitation)
• Sports medicine (e.g. injury prevention, performance
analyses, performance enhancement)
Gait Analysis Service
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Mocap
Overview
Arts / Education
• Dance and theatrical performances
• Archiving (e.g., Marcel Marceau)
OSU/ACCAD
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Mocap
Overview
Science / Engineering
• Computer Science (e.g., human motion database,
indexing, recognitions)
• Engineering (e.g., Biped robot developments)
• Ergonomic product design
• Military (e.g., field exercises, virtual instructors,
and role-playing games)
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Aspects of the Problem
Motion
Represent
ation
• “Gross” Body movement
NOT:
»
»
»
»
Appearance Models
Facial animation
Cloth, clothing, secondary movement
Hands
• How to describe moving target – “Character”
• How to describe movement mathematically
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Motion
Represent
ation
Human Model
• Humans are complex!
Human motion
can be
understood at
a very fine
level of detail!
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Motion
Represent
ation
Abstractions
Representation of complex human structure
with varying degrees of simplification
206 bones,
muscles, fat,
organs,
clothing, …
206 bones,
complex joints
53 bones
Kinematic
joints
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Motion
Represent
ation
Standard simplified models
of humans
• Small numbers of degrees of freedom for
gross motion
• Articulated figures
» Rigid pieces
» Sometimes stretching allowed
• Kinematic joints
» Rotations between pieces
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Motion
Represent
ation
An example articulated figure
The skeleton:
• 17 joints, each with 3 degrees of freedom
(DOFs)
• Root position (3 DOFs) and orientation (3
DOFs)
• Totally 57 DOFs
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Motion
Represent
ation
Pose representation
• Initial configurations of articulated figure
(Binding pose)
» Usually T-Pose
• Rigid pieces undergo rigid transformation
• Hierarchical representation of the
configuration
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Motion
Represent
ation
Computing positions
• Each joint i provides a local rotation Ri and a
local translation Ti. Concatenating Ri and Ti
gives local Mi, the local transform at each
joint.
• The transformation of the point x on joint j is
found by concatenating all previous
transforms in the hierarchy as follows:
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Motion
Represent
ation
Parameterizing Rotations
• Goal: encode rotations in a vector
» Rn - > “set of rotations”
• Give “names” to members of the set of
possible rotations
• Many ways to do this, all flawed
» No perfect method
» Use the best one for the job
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Motion
Represent
ation
Goals for Parameterization
• Compact
» (as few variables as possible)
• Complete
» Every rotation can be represented
• 1-to-1
» Every rotation has one value
» Every value has one rotation
• Singularity free
» “close” rotations are “close” in value
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Motion
Represent
ation
Parameterizations of
Rotations
•
•
•
•
Rotation Matrices
Euler Angles
Axis Angle formulation
Unit Quaternions
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Motion
Represent
ation
Comparison of body representation
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Motion
Represent
ation
Motion definition
• Motion is a function of time
» Given time, provide a pose
» m(t): R -> Rn
• Often represented as samples
» Sparse samples + interpolation
» Dense samples (at frames)
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Motion
Signal
Processing
Preliminaries
• A motion maps times to configurations
» m(t): R -> Rn
• Vector-valued, time-varying signal
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Motion
Signal
Processing
Challenge and solution
• Get a specific motion
» From capture, keyframe, …
» Specific character, action, mood, …
• Want something else
• But need to preserve original
» But we don’t know what to preserve
» Can’t characterize motion well enough
• Apply signal processing techniques to
designing, modifying, and adapting animated
motion [Bruderlin & Williams 95][Witkin &
Popovic 95]
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Motion
Signal
Processing
Manipulating motion
• Manipulate time
» m(t) = m0( f(t) )
» F : R -> R “time warp”
• Manipulate value
• m(t) = f(m0(t))
» F : Rn-> Rn
» F : (R, Rn)-> Rn
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Motion
Signal
Processing
Motion blending
• “Add” two motions together
• Really interpolate
» m(t) = a m0(t) + (1-a) m1(t)
• Note: this is a per-frame operation
• Interpolation between poses
» Interpolating root position – Linear
interpolation
» Interpolating joint rotation – SLERP (Spherical
Linear intERPolation)
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Motion
Signal
Processing
How to use blending
• Interpolate similar motions
» Be sure to make time correspondence
• Transition between motions
• Time varying blend (a=0 -> 1)
»
»
»
»
Over a short period of time
A bad pose isn’t such a big deal
Avoids discontinuities
m(t) = a(t) m0(t) + (1-a(t)) m1(t)
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Motion
Signal
Processing
Blending for transition motion
• Okan et al. Siggraph ’02
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Motion
Signal
Processing
Time warping
• Dynamic Time Warping (DTW): Non-linear
signal matching procedure originating in field
of speech recognition
• Finding the optimal sample correspondences
between the two signals by computing the
global minimum value of warping cost
function
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Motion
Signal
Processing
Time warping + interpolation
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Motion
Signal
Processing
Displacement map
• A.K.A Motion Warping
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Motion
Signal
Processing
Displacement map
• A.K.A Motion
Warping
• Changing the
shape of a signal
locally through a
displacement map
while maintaining
continuity and
preserving the
global shape of
the signal.
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Motion
Signal
Processing
Displacement map (cntd.)
• Mapping procedure
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Motion
Synthesis
from
Examples
The Challenge
• High Quality, Expressive Motion
» Need motion capture (examples)
• Flexible, long-running, controllable
» Need synthesis
• Synthesis from Examples!
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Motion
Synthesis
from
Examples
Idea:
Put Clips Together
• New motions from pieces of old ones!
• Good news:
» Keeps the qualities of the original (with care)
» Can create long and novel “streams” (keep
putting clips together)
• Challenges:
» How to connect clips?
» How to decide what clips to connect?
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Motion
Synthesis
from
Examples
Connecting clips: transition
• Transitions between motions can be hard
• Simple method:
» Blends between aligned motions
» Cleanup footskate artifacts
» When motions are similar
•
•
•
•
•
Believe that blending will “work”
Heuristic based on geometry
Not perfect method
Measure and use threshold
Apply threshold conservatively
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Motion
Synthesis
from
Examples
Similarity Metric
• Factor out invariances and measure
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Motion
Synthesis
from
Examples
Motion Graphs (Kovar et al. ’02)
• Start with a database of motions, each with
type and constraint information.
• Goal: add transitions at opportune points.
• Other Motion Graph-like projects elsewhere
• Differ in details.
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Motion
Synthesis
from
Examples
Motion Graphs
• Idea: automatically add transitions within a
motion database
• Quality: restrict transitions
• Control: build walks that meet constraints
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Motion
Synthesis
from
Examples
Using motion graph
• Any walk on the graph is a valid motion
• Generate walks to meet goals
» Random walks (screen savers)
» Search to meet constraints
• An example:
􀂄 Given a path
􀂄 Find a motion that minimizes distance
􀂄 Combinatorial optimization
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007
Motion
Synthesis
from
Examples
Why is this good?
•
•
•
•
•
Search the graphs for motions
Look ahead avoids getting stuck
Cleanup motions as generated
Plan “around” missing transitions
Optimization gets close as possible
Not OK for Interactive Apps!
Need different tradeoffs
MOTION PLANNING IN REAL AND VIRTUAL ENVIRONMENTS
FALL 2007