awoo016_thesis presentation_12
Download
Report
Transcript awoo016_thesis presentation_12
The University of Auckland | Computer Science | New Zealand
PRESENTATION
3D Human Face Reconstruction
and Expression Modelling
Alexander Woodward
2009
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Outline
Aim
System overview
Related work
3D face reconstruction
Expression modelling
Contributions and future work
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Overview
Aim: Integrated system for 3D face reconstruction and
expression modelling
Vision based not graphics based
Low cost and self-contained
Results can be applied to:
Biometrics and security
Biomedical visualisation
Computer and video games
Film
Teleconferencing
Human computer interaction
3
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
System overview
3D reconstruction
Static
Dynamic
Active & passive binocular stereo
3D video scanner
Active structured lighting
Active photometric stereo
3D data
Expression modelling
Marker based motion capture
Muscle inverse kinematics
Video based
Sequences from 3D video
scanner
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Related work
Complete systems for face reconstruction and
animation are uncommon
High hardware requirements
Data acquisition, motion capture and animation systems are
often provided as disparate packages or only as a service,
cf. a stand-alone solution
At least 9 prominent projects aimed toward complete
systems
Excluding in-house solutions
Large body of work in 3D face research
3D reconstruction, expressions, motion capture
13 April 2015
Department of Computer Science
5
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Related work
Borshukov et al (2003 – 2007)
Playable Universal Capture approach
3D scanner, marker based tracking, optical flow, video texture
Ma et al (2007, 2008)
Capture face reflectance
3D scanner, photometric stereo, motion capture
Light stage – 156 LED lights over an icosahedron
Image Metrics Inc. & U Sth Carolina Graphics Lab (2008)
Digital Emily project
Light stage captures geometry and reflectance
33 expressions captured; creates an animation rig
Performance data mapped to the 3D face
6
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
3D reconstruction
3D reconstruction
Static
Dynamic
Active & passive binocular stereo
3D video scanner
Active structured lighting
Active photometric stereo
3D data
Expression modelling
Marker based motion capture
Muscle inverse kinematics
Video based
Sequences from 3D video
scanner
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
3D reconstruction requirements
Off-the-shelf hardware, no special properties
Cameras, PC, projector
Low acquisition time – faces move, esp. children
Controlled lighting
Vision based
New algorithms
Useful for any type of object
8
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Static 3D reconstruction
3D reconstruction
Static
Dynamic
Active & passive binocular stereo
3D video scanner
Active structured lighting
Active photometric stereo
3D data
Expression modelling
Marker based motion capture
Muscle inverse kinematics
Video based
Sequences from 3D video
scanner
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Static 3D reconstruction
Evaluated approaches:
1. Active & passive binocular stereo
2. Active structured lighting
3. Active photometric stereo
Evaluate effectiveness
Accuracy, time complexity
Determine best approach for dynamic
3D reconstruction system
12 algorithms
Database of 15 faces
Alternative test set
Focus on stereo algorithms
Ground truth data: 3D scanner
Compared to Middlebury, algorithms rank differently for faces
Projected patterns improve and level out performance
10
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Active binocular stereo
Strip colour pattern: much higher accuracy
SAD correlation algorithm:
Strip pattern
SAD - without
pattern
SAD - with
strip pattern
80%
92%
Pattern colour should contrast strongly on skin
11
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Statistical results
BP
73%
GC
77%
FCV
69%
+ Grad. + Strip
77%
89%
+ Grad. + Strip
83%
92%
Four Path Shapelet
54%
71%
CM
88%
SAD
80%
+ Grad. + Strip
89%
92%
+ Grad. + Strip
85%
92%
DPM
+ Grad.
+ Strip
79%
84%
92%
SDPS + Grad.
89%
90%
+ Strip
93%
Gray code
97%
Ground truth
12
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Dynamic 3D reconstruction
3D reconstruction
Static
Dynamic
Active & passive binocular stereo
3D video scanner
Active structured lighting
Active photometric stereo
3D data
Expression modelling
Marker based motion capture
Muscle inverse kinematics
Video based
Sequences from 3D video
scanner
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Dynamic 3D reconstruction
Reconstruction at video rates →3D video!
From static reconstruction best results:
‘One shot’ active illumination + Symmetric Dynamic Programming (SDPS)
Project pattern every other frame to get a clean texture
(2)
(3)
(1)
Monochrome stereo pair of video cameras +
3rd colour web camera obtains colour texture.
14
2009
The University of Auckland | Computer Science | New Zealand
PRESENTATION
Colour texture generation
+
Colour image
(reprojected into same
reference frame)
→
Monochrome
image
Final texture
Low resolution colour information combined with high
resolution luminance information
Next step: colour video cameras
15
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Videos
16
The University of Auckland | Computer Science | New Zealand
Patternless reconstruction
PRESENTATION
2009
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Marker based expression modelling
3D reconstruction
Static
Dynamic
Active & passive binocular stereo
3D video scanner
Active structured lighting
Active photometric stereo
3D data
Expression modelling
Marker based motion capture
Muscle inverse kinematics
Video based
Sequences from 3D video
scanner
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Marker based expression modelling
Data driven:
Stereo web-cameras, face
markers.
Head motion - rigid
Expressions - non-rigid
Tracked 3D points
Unique 3D face model
mapping
Virtual muscle animation
17 active muscles
Muscle inverse kinematics (IK) –
Jacobian Transpose
19
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Example videos
Happiness – easiest to reproduce
Surprise
20
2009
The University of Auckland | Computer Science | New Zealand
PRESENTATION
Anger – needs teeth!
Disgust – pursing of mouth & closing of eyes not represented
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Video based expression modelling
3D Reconstruction
Static
Dynamic
Active & passive binocular stereo
3D video scanner
Active structured lighting
Active photometric stereo
3D data
Expression modelling
Marker based motion capture
Muscle inverse kinematics
Video based
Sequences from 3D video
scanner
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
3D video based expression
modelling
Image blending
Novel face expressions from
multiple video sequences
Interactive
Low preparation
Not data driven
Dense depth data – cf. marker
system
Video based → realistic 3D
movement and texture
Reconstruction data directly
used for expression modelling
Sub-region masks
11 Control
control points
points
23
2009
Synthetic expression results
The University of Auckland | Computer Science | New Zealand
PRESENTATION
Sadness: lower face region, anger: right eye region, surprise: left eye region
Happiness: lower face region, surprise: left and right eye regions
24
2009
Synthetic expression results
The University of Auckland | Computer Science | New Zealand
PRESENTATION
Fear: lower face region, happiness: right eye region, anger: left eye region
Disgust: lower face region, anger: left and right eye regions
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Contributions
3D face reconstruction and expression modelling
system
Unique tool-set
Low-cost, off-the-shelf
Vision based
To 3D face reconstruction:
Extensive reconstruction comparison
Face database
Dynamic reconstruction system for 3D video: SDPS + pattern
To expression modelling:
Marker based performance capture system
Muscle based IK animation system, unique mapping approach
Video based expression system – realistic, less flexible
26
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Future work and perspective
Many areas for future research
Refine hardware - better reconstructions ( low-cost? )
Markerless motion capture - face ( feature ) tracking
Statistical analysis on video data
Active appearance model (AAM)
New animation system (out of scope)
Full body → complete character
Synergy of computer vision and computer graphics!
Physical models for animation
Computer vision tools
Especially 3D video & markerless motion capture
The University of Auckland | Computer Science | New Zealand
PRESENTATION
Questions?
2009
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Timeline of Experiments
Ekman - 1987
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Universal expressions
Sadness
Anger
Fear
Disgust
Happiness
Surprise
Recognisable in every culture! Used as exemplar expressions to
judge my results
31
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Types of binocular stereo algorithm
Local vs global optimisation
WTA
SAD, SSD
Chen-Medioni –
local method with explicit surface constraints
Seed propagation approach
Dynamic programming – 1D optimisation
SDPS – markov chain
DPM
Cubic algorithms – 2D optimisation
Markov random field
Energy minimisation
Graph-cut (KZ1, RoyCox), Belief Propagation,
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Types of photometric stereo algorithm
Experiment focused on integration methods
Assumes C² continuity – i.e. a smooth second derivative
Local optimisation – based on curve integrals
Four path integration
Shapelet
Explicit summation of basis functions
Global optimisation
FCV – Frankot Chellappa Variant
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Structured lighting techniques
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Body modelling and animation
Body: generic skinned animation
Skeletal hierarchy, fully articulated
• The bones of the hand
• Each bone of the
• The body model with underlying
skeleton
13 April 2015
skeleton has a region of
influence, denoted in green
Department of Computer Science
• Movement of the forearm
35
The University of Auckland | Computer Science | New Zealand
PRESENTATION
Interactive personalised avatar creator
Input photograph
RBF mapping
36
2009
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Results
13 April 2015
Department of Computer Science
37
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Results
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Results
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
3D video based expression system
overview
Acquire sequences of individual expressions using
dynamic 3D face reconstruction system.
Expression sequences start from a neutral state.
Test subject’s head remains in the same position for every sequence
A reference texture and depth map are taken from the neutral
expression and used as the base for all image regions
11 control points are manually annotated on video
sequences.
Future work to automate this process.
Six sub-regions manually defined on the face.
A sub-region’s texture and depth updated by dragging
a control point residing in it and its currently chosen
expression
sequence.
13 April
2015
Department of Computer Science
40
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
System conclusions
Sinusoidal interpolation instead of a linear one. This
roughly models the biphasic nature of skin
Realistic animations are created as motion is derived
from 3D video sequences of real-life test subjects.
A user can create unnatural but interesting looking expressions that
can convey a comical feel
Texture maps sourced from video sequences solves
the loss of detail in the marker based approach
However, apart from the control points that were manually specified,
no points on the face surface are tracked
Results could be refined by improving the quality of 3D
video reconstruction.
13 April 2015
Department of Computer Science
41
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Test subject placement
Subject can be placed with knowledge of required view
area, sensor size, and camera lens:
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Projector synchronisation
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
RBF mapping approach
Radial Basis Functions
User specified point correspondences on generic
model and 3D face data
Specify divergences between data
For each dimension (in 3D)
Find RBF approximation of (1D) displacements within the 3D
space of specified points.
Using this RBF approximation all 3D points from the
generic model can be mapped to the 3D face data
proportions
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Marker tracking
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Rigid and non-rigid motion
Anchor markers:
Rigid orientation:
Remove rigid motion by using transpose of orientation and
centre of gravity of anchors
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Muscle inverse kinematics
Forward kinematics is the calculation of a new position g of
an end effector by specifying updates to parameters of a
kinematic chain
Inverse kinematics is the calculation of parameters for a
kinematic chain to meet a desired goal position g, when
starting from an initial position e.
Kinematic chain consists of joints
Each joint has DOF’s – its animitable parameters,
E.g. 3-DOF for position, 1-DOF for orientation around one axis
(position of joint implied through kinematic chain transformation)
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Jacobian Transpose approach
FK:
IK:
e = current end
effector position
d = change in end effector
position
First order estimate in positional
change:
Change in parameters:
Jacobian Transpose estimate:
g = goal end
effector position
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Estimate assured to move closer to
the goal g:
Always moving in a direction less
than 90 degrees from d
13 April 2015
Department of Computer Science
49
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Interface between Raw Data and
Generic Model
User specifies a ‘minimal’ set of correspondences between
raw and generic data
Radial Basis Functions (RBF) used as the interpolant
Model with
animation system
Depth map
Correspondences
made and mapped
via RBF with a final
nearest point map
and texture
projection
Results in a custom
face with animation
system in place
•Feature extraction as a goal
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Face Animation Model
Research primarily based on Terzopoulos, Waters, Parke collective work in
the field
Physically based model for skin tissue
Mass-Spring system
Epidermal – Fascial – Skull levels of tissue
Forces are applied to the tissue to simulate muscle contractions
Springs bring elasticity, allow forces to propagate-> stretches and pulls!
Abstract muscle definitions
Decoupled from model
Warped via RBF also
Two types
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Face Animation System Forces
Model the behaviour of the tissue
Reactionary over the evolution of applied muscle forces:
Skull Penetration Constraint
Spring Forces
g j c j (l j - lj )s j , gi g j
cj
- biphasic spring constant
l j
- rest length of spring
lj
- current length of spring
s j (x j xi ) / l j
- spring direction vector
fin n i n i when f i n n i 0
si
otherwise
0
Muscle Forces
Applied to fascia nodes based
on the abstract muscle
definitions………..(explained
later)
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Face System Forces
Volume Preservation Force
q k1 V V
e
i
e
e
n
e
i
k2 p p
e
i
e
i
nei
- epidermal normal for volume element ‘e’
p ei , pie
- current and rest nodal positions with
respect to center of mass of element ‘e’
k1 , k 2
- force scaling constants
These forces allows for tissue form restitution
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Linear Muscle
Linear Muscle:
Applies forces to nodes inside it’s angular range
Influence is weighted by angle and radius from muscle vector
Displacement formula:
p p akr
Where
a 1
pv1
pv1
cos( )
cos()
and
D
cos(
1
); for p inside sector( v1p n p m p1 )
R
2
s
r
cos( D Rs ); for p inside sector(p np r p sp m )
R f Rs 2
‘k’ = muscle contraction increment.
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Ellipsoid Muscle
Ellipsoid Muscle:
Acts like a string bag
Application of force weighted by radius only
Defined by major and 2 minor axes
Can generate puckering effects
Displacement formula:
p p kr
pv1
pv1
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Physical Simulation
Layered Tissue Model is a physically based one
Euler integration is used to run the simulation
Equations of motion
1 e
~t
~ t ~s t h
a
f i i v ti ~
git q
i
i
i
mi
t
i
Acceleration
v ti t v ti tati
Velocity
xti t xti tvti t
Nodal position
Velocity dependent
damping co-efficient.
Controls the rate of
dissipation of kinetic
energy which eventually
brings the facial mesh to
rest.
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
6 pre-built expressions
Happiness
Sadness
Surprise
Fear
Disgust
Anger
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
General conclusions and future
work (old version)
Investigated low-end and cost effective equipment to
create self-contained tools that can run entirely on any
end user system.
A unique solution has been proposed for 3D face
reconstruction and expression modelling with appropriate
hardware
Synchronised audio capture of speech sequences
would greatly add to the realism.
The attachment of the face model to a body would
complete the system, giving a fully realised virtual
human.
13 April 2015
Department of Computer Science
59
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Conclusions drawn from the static reconstruction
experiment formed the basis of a dynamic 3D face
reconstruction system
3D face reconstructions have no notion of higher order
surface structure and are just a collection of points.
This structure was addressed in the second part of this thesis which
investigated face expression modelling.
Marker motion was combined with a muscle inverse
kinematics framework to drive the facial animation
system.
A static face texture impacts on the visual result, as illumination cues
such as wrinkles and shadowing over the face are lost.
13 April 2015
Department of Computer Science
60
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
To supplement the work on 3D faces, a body model
was created
Interactive 3D video expression creation system which
ties together 3D face reconstruction and expression
modelling.
Main problems faced were dealing with hardware
constraints
But focus on low-cost and off-the-shelf solutions
Focused on the computer vision aspects of facial
reconstruction and expressions as opposed to
computer graphics
13 April 2015
Department of Computer Science
61
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
A combined marker and dense 3D reconstruction
system could be developed, to incorporate further
information for a muscle inverse kinematics system
Highly detailed face animation is best served by taking
advantage of real world data in the form of digital
images and computer vision processing
Advanced physical models of faces meets the tools
and approaches investigated within this thesis
13 April 2015
Department of Computer Science
62
13 April 2015
Department of Computer Science
63
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Details in important areas of the face that are not
currently modelled include the eyelids, lips, teeth and
inner mouth.
Loss of texture detail in the forehead where the
wrinkles are lost
Fine tuning of the preset muscle locations and
parameters when mapping a new face model was
sometimes needed to improve results or correct
muscles
13 April 2015
Department of Computer Science
64
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
System analysis
Capable of reproducing facial expressions from marker
motion.
Low cost hardware, easy retargetting to other models.
Many differences between test-subjects
Expression articulation, muscle control, gross face
movement
Difficulty in performing when no emotional tie involved.
Easy to understand the need for directors in performance capture
situations.
Some user direction was needed to describing to a test subject how
an expression should be created
13 April 2015
Department of Computer Science
65
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Issue: potentially multiple solutions for a vertex
position when influenced by multiple muscles
Illumination conditions affect coloured marker
detection
Reflectance properties of the skin surface are
important visual cues.
Missing in this system
Addressed in next chapter.
13 April 2015
Department of Computer Science
66
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Body modelling and animation
system
Skinned animation system was chosen for real-time
capability and ease of creating new body poses
A posable skeleton is associated with a body model (skin
surface description), usually in the form of geometric data
Forward and inverse kinematics used for animation
Skin surface under new pose is determined based on skeletal bone
local coordinate systems and blending between adjacent bones.
Future work: combine with the face reconstructions
and animation systems.
13 April 2015
Department of Computer Science
67
2009
Static reconstruction experiment
The University of Auckland | Computer Science | New Zealand
PRESENTATION
Evaluated three computer vision approaches to 3-D face reconstruction.
Binocular stereo: passive.
Structured lighting: active.
Photometric stereo: active.
Two main aims:
Determine their effectiveness for 3D facial reconstruction.
Accuracy, time complexity.
Provide a new and alternative test set for evaluating algorithms.
Database of faces.
We focus on stereo vision algorithms.
Integrated lab environment designed.
12 algorithms tested in total.
Results compared to ground truth data obtained from a commercial 3D scanner.
Summary:
Active illumination techniques are most accurate.
Stereo algorithm rankings were different from that expected.
‘One shot’ active illumination coupled with a traditional stereo algorithm a strong
choice.
13 April 2015
Department of Computer Science
68
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Photogrammetry Laboratory
Optical ‘Range’.
Integrated.
Multiple systems view a
common scene.
Stereo bench.
Sideways for face capture!
Example Data:
Projector for structured
lighting.
Light sources for
photometric stereo.
Commercial 3D scanner.
Solutionix Rexcan 400.
Depth map
13 April 2015
Department of Computer Science
Perspective visualisation
69
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Calibration
System calibration:
Estimates intrinsic and extrinsic
camera parameters
I.e. camera projection matrices
For cameras:
A calibration cube - 63 markings
defines a world co-ordinate system
Tsai calibration
For the lights:
A calibration sphere - estimates
directions to lights
Simple analytic derivation,
inaccurate
Could also calibrate the projector using
Tsai’s algorithm
13 April 2015
Department of Computer Science
70
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
World to image co-ordinates
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Rectification:
The camera calibration matrices were used to rectify images.
The resultant image pairs meet the epipolar constraint.
Data processing:
Data must be compared in a common co-ordinate frame.
Alignment done using a semi-automatic process involving 3D object rigid
transformations.
Small number of manual correspondences made.
Data projected into disparity space.
13 April 2015
Department of Computer Science
72
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Database of 15
people created
Data acquired
from all systems
Rexcan ground
truth
Test-bed for new
algorithms
73
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Binocular Stereo
Approach 1: Binocular stereo (stereo
vision).
System Geometry (side view)
Passive.
Active research area in our department.
Textureless regions cause problems.
Remedy via active illumination.
Test a set of local and global algorithms.
Tested algorithms:
Sum of Absolute Differences (SAD)
Use two Canon digital SLRs –
6 Mpixels
1536 x 1024
resolution.
Dynamic Programming Method (DPM)
Symmetric Dynamic Programming Stereo (SDPS)
Graph Cut (GC)
Belief–Propagation (BP)
Chen and Medioni (CM) – seed based algorithm
13 April 2015
Department of Computer Science
74
2009
Structured Lighting
The University of Auckland | Computer Science | New Zealand
PRESENTATION
Approach 2: Structured Lighting.
System Geometry (side view)
Active approach. Depth inferred in the
same manner as stereo.
Augment stereo system with a colour
projector.
Add structure to scene -> break
homogeneity.
Projects 800 x 600 pixel image.
Acer PL111 LCD Projector.
6 of the Gray code projections:
Interested in ‘one shot’ patterns over
Gray code.
Tested algorithms:
Time-multiplexed structured lighting using Gray code
Direct coding - ‘one shot’ colour gradation pattern.
Direct coding - ‘one shot’ colour strip pattern.
13 April 2015
Department of Computer Science
•Add texture to face.
•Used with standard stereo
algorithms.
75
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Photometric Stereo
Approach 3: Photometric Stereo (PSM).
Face viewed under 3 different known lighting
conditions.
Depth by integrating recovered surface orientation
map.
Albedo independent approach used.
System Geometry (top-down view)
Three 150W light sources.
Analysed gradient field integration
techniques.
Tested algorithms:
Frankot-Chellappa Variant (FCV)
Fourier based integration.
Four-Scan method
Local integration paths.
Shapelets
Summation of correlated basis functions.
13 April 2015
Department of Computer Science
76
2009
A Collection of Reconstructions
The University of Auckland | Computer Science | New Zealand
PRESENTATION
Example depth maps:
Ground
truth
Gray
code
FCV
SAD
Structured lighting
SDPS
GC
CM
Binocular Stereo
Photometric Stereo
13 April 2015
Department of Computer Science
77
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Photometric Stereo Results
Reconstruction accuracy:
17 test subjects.
Percentage of errors less than 2 disparity units
Method
P <=2,%
69
54
71
97
Gold standard result for accuracy
13 April 2015
Department of Computer Science
78
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Passive Stereo Results
Reconstruction accuracy:
Method
P<=2,%
89
79
77
GC
80
73
88
97
13 April 2015
Department of Computer Science
79
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Stereo + Gradation Pattern
Reconstruction accuracy:
Method
P<=2,%
90
84
83
GC
85
77
89
97
13 April 2015
Department of Computer Science
80
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Stereo + Strip Pattern
Reconstruction accuracy:
Method
P<=2,%
93
92
92
GC
93
89
92
97
13 April 2015
Department of Computer Science
81
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Improvement to Stereo from Active
Illumination
Addition of the Strip colour pattern.
SAD stereo algorithm:
Strip pattern
Depth map
SAD - without
pattern
SAD - with
strip pattern
P<=2 = 80%
P<=2 = 93%
Pattern colour should avoid skin tones.
13 April 2015
Department of Computer Science
82
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Error map
example
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Gray code approach most accurate.
Slower acquisition time.
Look to alternative ‘one shot’ approaches.
Photometric stereo least accurate.
Our test set has high resolution images and large
disparity ranges.
O(n3) stereo algorithms – GC, BP – inappropriate.
Long processing time.
Parameter setting difficult.
Our results differ from the Middlebury rankings:
http://cat.middlebury.edu/stereo/
13 April 2015
Department of Computer Science
84
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
All results contain errors.
Need post processing to clean up data.
Even for the commercial 3D scanner.
Faces have many unique properties posing a challenge
for 3D reconstruction
Human sensitivity to errors in reconstruction - we see faces
all the time.
For computer vision:
13 April 2015
Specularities.
Anistropic reflectance of hair.
Sub-surface scattering.
Large homogenous regions.
Department of Computer Science
85
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Static reconstruction conclusion
and analysis
Framework and test-bench for active and passive 3-D
acquisition systems designed.
Three computer vision approaches tested.
12 algorithms altogether.
Analysed accuracy of algorithms for 3D face reconstruction.
Data compared to scanner benchmark.
Provided new alternative test set to Middlebury for testing
stereo algorithms
High resolution images of faces.
Passive stereo combined with active illumination a promising
approach.
Want a one shot approach for faces (moving object).
SDPS + Strip pattern.
Leads to real-time spatio-temporal acquisition.
Acquire 3D face performance.
13 April 2015
Department of Computer Science
86
13 April 2015
Department of Computer Science
87
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
A generic face model with an abstract muscle animation
system was designed during my Master’s thesis.
Refined for PhD thesis.
Can be personalised with 3D data and texture information from the
static reconstruction experiment using a custom RBF mapping
procedure.
• Example of muscle contraction
• Generic morphable face with linear
and ellipsoid muscles
13 April 2015
Department of Computer Science
• A biomechanical tissue model
88