Transcript Towards More Photorealistic Faces

Towards More Photorealistic
Faces
Kenneth L. Hurley
Agenda
Introduction to coding systems
Creating Content
Transforming a canonical face from photographs
Animation techniques
Achieving more realistic faces
Conclusion
FACS
Facial Action Coding System
Paul Eckman and Wallace Friesen
Foundation for most facial animation
Classified facial expression by “Action Units” AU
6 universal categories
Sadness, anger, joy, fear, disgust and surprise
FACS
Creating Content
Traditional art techniques
3D Modeler package
Triangle based
Nurbs based
FFD Based
Patches Based
Parameterized
Morph Targets
Examples
Triangles
Patches
Parameterized
3D Capture Device
Various Available on Market
Captures data with a variety of techniques
Laser Range Finders
Light Projections
Shadow Projections
Sonic Range Finders
Capture houses can do it for you
Cyberware
2D Capture Techniques
Photogrammetry
Front photo of individual is all that’s required.
Front and side give better 3D coordinates
Automated feature recognition
New techniques are being developed every day
Usually computationally expensive and still needs
user intervention
See http://www.cs.rug.nl/~peterkr/FACE/face.html
Direct Parameterized Models
Frederic I. Parke 1974 PhD Thesis entitled “A
Parametric Model for Human Faces”
Used to create parameterized heads using
Interpolation, Scaling and Translation
Steve DiPaola extended this version. Program is
available at
http://www.stanford.edu/dept/art/SUDAC/facade/
Animation of Face
Muscle Simulation
Parke and Waters system is most widely used
“Computer Facial Animation” book examples
Uses 18 muscles in example
Open Source, with OpenGL
http://www.crl.research.digital.com/publications/boo
ks/waters/Appendix1/opengl/OpenGLW95NT.html
Animation of Face (Cont)
Animation of Face
Open Source Expression Toolkit
Based of of Parkes model
Adds nice user interface
Exports data from 3DS Max
Scripting Language
http://expression.sourceforge.net/
Facial Muscles
Lip Syncing
Phonemes – What are they and why is it
important?
Helpful to determine mouth posture
45 English phonemes, more or less for other
languages
Mouth Postures
Nitchie produced 18 significant mouth postures to
match called visemes.
Lip Syncing (Cont)
Visemes can match mouth postures with speech
phonemes
Tools
Microsoft Speech SDK
Can decipher recorded speech into phonemes
Code to give mouth positions (visemes) from these
phonemes.
Free
http://www.microsoft.com/speech
Lip Syncing (Cont)
University of Edinburgh
http://www.cstr.ed.ac.uk/projects/festival/
Open source, free speech system
Coarticulation
Blending of lip-sync postures.
Pelachaud described algorithm in 1991
Cohen and Massaro developed a dominance
model.
http://mambo.ucsc.edu/psl/ca93.html
MPEG4 Facial Animation
Designed to encode low bandwidth data
Parameters are sent over transmission lines and
reconstructed at client
Facial Animation Parameters (FAP)
Facial Definition Parameters (FDP)
Uses 68 FAPs broken up into 10 groups
Uses only 14 visemes
Very similar to “Action Units” in Parkes model
FAP Groups
Group
Number of FAPs
1: visemes and expressions
2
2: jaw, chin, inner lowerlip, cornerlips, midlip
16
3: eyeballs, pupils, eyelids
12
4: eyebrow
8
5: cheeks
4
6: tongue
5
7: head rotation
3
8: outer lip positions
10
9: nose
4
10: ears
4
Viseme and Related Phonemes
Viseme #
phonemes
example
7
s, z
sir, zeal
0
none
na
8
n, l
lot, not
1
p, b, m
put, bed, mill
9
r
red
2
f, v
far, voice
10
A:
car
3
T,D
think, that
11
e
bed
4
t, d
tip, doll
12
I
tip
5
k, g
call, gas
13
Q
top
6
tS, dZ, S
chair, join, she
14
U
book
MPEG4 FDP Parameters
Bump Mapping
Requires either CPU work or GPU setup
CPU has to compute tangent space basis vectors
for light
GPU can do it on a vertex by vertex basis as the
vertices come through the pipeline.
Free Form Deformations (FFD)
Can be used to scale facial features
Problems with where to place lattice.
Good solution is DFFD
Dirichlet Free Form Deformations
Basically idea is that the control points are on the
surface
http://cui.unige.ch/~moccozet/PAPERS/CA97/
Rational Free Form Deformations
Another idea is to have non-axis aligned lattices
Still working on implementation
Free Form Deformations (FFD)
Bump Mapping (Cont).
Sample Vertex Shader
; This shader does the per-vertex dot3 work.
; It transforms the light vector by the basis vectors
; passed into the shader.
; The basis vectors only need to change if the model's
; shape changes.
; The output vector is stored in the diffuse channel
; (use the menu to look at the generated light vector)
#include "dot3.h"
#define
#define
#define
#define
V_POSITION v0
V_NORMAL v1
V_DIFFUSE v2
V_TEXTURE v3
Bump Mapping (Cont).
Sample Vertex Shader
#define
#define
#define
#define
#define
#define
#define
V_SxT v4
V_S v5
V_T v6
S_WORLD r0
T_WORLD r1
SxT_WORLD r2
LIGHT_LOCAL r3
vs.1.0
; Transform position to clip space and output it
dp4 oPos.x, V_POSITION, c[CV_WORLDVIEWPROJ_0]
dp4 oPos.y, V_POSITION, c[CV_WORLDVIEWPROJ_1]
Bump Mapping (Cont).
Sample Vertex Shader
dp4 oPos.z, V_POSITION, c[CV_WORLDVIEWPROJ_2]
dp4 oPos.w, V_POSITION, c[CV_WORLDVIEWPROJ_3]
; Transform basis vectors to world space
dp3 S_WORLD.x, V_S, c[CV_WORLD_0]
dp3 S_WORLD.y, V_S, c[CV_WORLD_1]
dp3 S_WORLD.z, V_S, c[CV_WORLD_2]
dp3 T_WORLD.x, V_T, c[CV_WORLD_0]
dp3 T_WORLD.y, V_T, c[CV_WORLD_1]
dp3 T_WORLD.z, V_T, c[CV_WORLD_2]
dp3 SxT_WORLD.x, V_NORMAL, c[CV_WORLD_0]
dp3 SxT_WORLD.y, V_NORMAL, c[CV_WORLD_1]
dp3 SxT_WORLD.z, V_NORMAL, c[CV_WORLD_2]
Bump Mapping (Cont).
Sample Vertex Shader
mul S_WORLD.xyz, S_WORLD.xyz,
c[CV_BUMP_SCALE].w
mul T_WORLD.xyz, T_WORLD.xyz,
c[CV_BUMP_SCALE].w
; transform light by basis vectors to put it
; into texture space
dp3 LIGHT_LOCAL.x, S_WORLD.xyz,
c[CV_LIGHT_DIRECTION]
dp3 LIGHT_LOCAL.y, T_WORLD.xyz,
c[CV_LIGHT_DIRECTION]
dp3 LIGHT_LOCAL.z, SxT_WORLD.xyz,
c[CV_LIGHT_DIRECTION]
; Normalize the light vector
dp3 LIGHT_LOCAL.w, LIGHT_LOCAL, LIGHT_LOCAL
Bump Mapping (Cont).
Sample Vertex Shader
rsq LIGHT_LOCAL.w, LIGHT_LOCAL.w
mul LIGHT_LOCAL, LIGHT_LOCAL, LIGHT_LOCAL.w
; Scale to 0-1
add LIGHT_LOCAL, LIGHT_LOCAL, c[CV_ONE]
mul oD0, LIGHT_LOCAL, c[CV_HALF]
; Set alpha to 1
mov oD0.w, c[CV_ONE].w
; output tex coords
mov oT0, V_TEXTURE
mov oT1, V_TEXTURE
Bump Mapping (Cont)
Sample Pixel Shader
tex
tex
dp3
mul
t0
t1
r0, v0, t1
r0, r0, t0
;
;
;
;
grab the diffuse texture map
load bump map
dot normal with light vector
calculate diffuse value for
bumped normal
Realistic looking skin
Use of high detail bump maps dramatically add
realism.
http://www.cc.gatech.edu/cpl/projects/skin/skin.pdf
Realistic looking skin (Cont)
Even better, use diffuse, bump map, specular and
environment map
Realistic looking skin (Cont)
Realistic looking skin (Cont)
Realistic looking skin (Cont)
Sample Vertex Shader (modify bump map VS)
mov oT1, V_TEXTURE
mov oT2, V_TEXTURE
mov oT3, V_NORMAL
; specular map
; pass normal for LDR
Realistic looking skin (Cont)
Sample Vertex Shader
tex
tex
tex
tex
t0
t1
t2
t3
;
;
;
;
grab
load
load
load
the diffuse texture map
bump map
specular map
environment lighting LDR
dp3
mul
r0, v0, t1
r0, r0, t0
mul
r1, t2, t3
add
r0, r0, r1
; dot normal with light vector
; calculate diffuse value for
bumped normal
; mul specular * environment
LDR
; add specular + diffuse
It’s in the Eyes
Disney once said the audience watches the eyes.
Orientation, with respect to the world as well as
each other can give sense of focus and direction of
look.
Also use environment map to give reflective glossy
look.
References
1.
2.
3.
4.
5.
6.
7.
Parke, Frederic I. And Waters, Keith. “Computer Facial Animation” A K
Peters., Wellesly, Mass. 1996
Pighin, Frederic, et. al. “Synthesizing Realistic Facial Expressions from
Photographs”. Siggraph 1998
Gray, Henry. “Gray’s Anatomy of the Human Body” Online
http://www.bartleby.com/107/
Cohen, M. M., & Massaro, D. W. “Modeling coarticulation in synthetic
visual speech”. In N. M. Thalmann & D. Thalmann (Eds.) Models and
Techniques in Computer Animation. Tokyo: Springer-Verlag. 1993
Nitchie, E.B. “How to Real Lips for Fun and Profit”. Hawthorne Books,
New York, 1979
A. Haro, B. Guenter, and I. Essa. "Real-time, Photo-realistic, Physically
Based Rendering of Fine Scale Human Skin Structure“. Proceedings 12th
Eurographics Workshop on Rendering, London, England, June 2001
Lundgren, Ulf. “Description of how the model was created”
http://www.lost.com.tj/Ulf/artwork/3d/behind.html
References
8.
9.
10.
Osterman, J. and Tekalp, Murat. “Face and 2-D Mesh Animation in MPEG4”. Online at http://www.cselt.it/leonardo/icjfiles/mpeg-4_si/8SNHC_visual_paper/8-SNHC_visual_paper.htm
http://www.nvidia.com/developer
L. Moccozet and N. M. Thalmann, “Dirichlet free-form deformations and their
application to hand simulation”, Proceedings of the Computer Animation’97,
pp93-102, 1997.