Transcript Face Relighting with Radiance Environment Maps

Face Relighting
with Radiance Environment Maps
Zhen Wen1 , Zicheng Liu2 , Thomas Huang1
Beckman Institute1
Microsoft Research2
University of Illinois
One Microsoft Way
Urbana, IL61801, USA
Redmond, WA 98052, USA
{zhenwen, [email protected]}
[email protected]
Problem Statement
• Given a single image of face, modify the lighting effect
– Simulate environment rotation
– Transfer the lighting from another face image
– Interactive lighting editing
Input
Modified
lighting
Related Work
• Inverse rendering – recover reflection
properties from image samples and geometry.
– Recover BRDF [Marschner1998].
– Recover basis
• [Debevec2000], ~2000 basis
• [Georphiades1999], 3 basis for Lambertian reflection.
• Relighting by illumination ratio
– Preserve high frequency texture.
• [Riklin1999], [Stoschek2000] “quotient image”
• [Sato1999]
Assumptions
• Diffuse face surface
• Distant illumination
• Ignore cast shadow
– (as in applications of environment maps)
Radiance Environment Map
• Suppose we capture the lighting using a sphere
(radiance environment map).
I sphere (n)   sphere 
(n )
L(ω)(n  ω)dω
– For face
I face (n)   face 
(n )
L(ω)(n  ω)dω
Approximate Radiance Environment
Map from Image
• Use spherical harmonic representation of lighting:
[Ramamoorthi2001] [Basri2001]
E(n) 
 Aˆ
L Y (n)
lm lm lm
l 2, l ml
• Hypothesis: face albedo has no lower
order (1,2,3,4) coefficients:
 (n)  00  (n)
I (n)   (n) E (n)
 00  Aˆlm LlmYlm (n)  (n)
l  2 ,  l  m l
projection( I (n)) 

 Aˆ
L Y (n)
lm lm lm
l  2 ,  l  m l
ˆ L Y (n)
A
00 lm lm lm
l 2, l ml
• Algorithm:
– Solve for the first 9 harmonics coefficients of I (n)
00 AˆlmLlm : l  2,l  m  l
– REM =
00 E(n)
Relighting
• Step 1: Radiance environment map
• Step 2: To relight a
rotated pixel:
I
Different Relighting Scenario
– Different lighting:
I ' face (n1 ) I 'sphere (n1 )
I 'sphere (n1 )

 I ' face (n1 )  I face (n1 ) 
I face (n1 ) I sphere (n1 )
I sphere (n1 )
• Light transfer:
Estimate I 'sphere from new face image
• Light editing:
Obtain I 'sphere
by editing coefficients
Back Lighting Assumption
• It’s under-constrained to recover all 9
coefficients from a single frontal image
– [Ramamoorthi2002]
• Make assumptions about back lighting
– Symmetric lighting, i.e. lighting in the back is the
same as front
• Good when only frontal lighting matters
• Equivalent to assuming 3 “asymmetric” coefficients to
be zero.
– Assumption based on scene, e.g. dark back lighting
Basic Algorithm
• Align image with generic 3D face model.
• Approximate radiance environment map.
• Synthesis appearance in
– rotated lighting
– different lighting using REM recovered from
images in target lighting.
– light editing by adjusting the 9 coefficients
Dynamic Range of Image
• Ratio-based relighting has large relative error when
pixels values are too low or saturated.
• Use example-based texture synthesis to improve.
– Relight all skin pixels to same normal.
– Detect outliers using robust statistics.
– Use the remaining pixels as examples to synthesis in the
place of outliers
• Use patch-based approach
• Constrain that synthesized patch should be as close as the original
patch.
Results – Rotation Example 1
The middle image is the input
Results – Rotation Example 2
– Ground Truth Comparison
The middle image is the input. The upper row is the ground truth.
Results – Rotation Example 3
The middle image is the input
Results – Low Dynamic Range
RGB
B
Basic
algorithm
Input
With examplebased synthesis
Results – Light Editing
Results – Lighting Transfer
Input
Target
Result
Conclusion
• Efficient technique for modifying, editing
lighting in face images
• Capture variation due to diffuse lighting using
spherical harmonics based environment maps
• Ratio image remove material dependency
Future Work
• More intuitive interface for light editing
• Handle non-Lambertian effects
• Recover personalized 3D geometry model