Transcript Multiple View Geometry in Computer Vision

Shape-from-X
Class 11
Some slides from Shree Nayar. others
3D photography course schedule
(tentative)
Lecture
Exercise
Introduction
-
Oct. 3
Geometry & Camera model
Camera calibration
Oct. 10
Single View Metrology
Measuring in images
Oct. 17
Feature Tracking/matching
Correspondence computation
Sept 26
(Friedrich Fraundorfer)
Oct. 24
Epipolar Geometry
F-matrix computation
Oct. 31
Shape-from-Silhouettes
Visual-hull computation
(Li Guan)
Nov. 7
Stereo matching
Project proposals
Nov. 14
Structured light and
active range sensing
Papers
Nov. 21
Structure from motion
Papers
Nov. 28
Multi-view geometry
and self-calibration
Papers
Dec. 5
Shape-from-X
Papers
Dec. 12
3D modeling and registration
Papers
Dec. 19
Appearance modeling and
image-based rendering
Final project presentations
Shape-from-X
• X =Shading
• X =Multiple Light Sources
(photometric stereo)
• X =Texture
• X =Focus/Defocus
• X =Specularities
• X =Shadows
• X =…
Shape from shading
•
•
Shading as a cue for shape reconstruction
What is the relation between intensity and
shape?
•
Reflectance Map
Reflectance Map
Relates image irradiance I(x,y) to surface
orientation (p,q) for given source direction and
surface reflectance
Lambertian case:
I x, y 
•
•
k : source brightness

: surface albedo (reflectance)
c : constant (optical system)
s vn
i
Image irradiance:


I  kc cos  i  kc n  s


Let

kc  1 then

I  cosi  n s
Reflectance Map
•
Lambertian case
I  cosi  n  s 
 pps  qqs  1
p  q 1 p  q 1
2
2
2
S
2
S
 R p, q 
Reflectance Map
(Lambertian)
Iso-brightness contour
cone of constant
i
Reflectance Map
•
Lambertian case
iso-brightness
contour
q
0 .8
0 .9
R p, q   0.7
1 .0
 pS , qS 
p
i  90
 ppS  qqS 1  0
0 .3
0 .0
Note: R p, q is maximum when  p, q   pS , qS 
Shape from a Single Image?
•
•
Given a single image of an object with known
surface reflectance taken under a known light
source, can we recover the shape of the object?
Given R(p,q) ( (pS,qS) and surface reflectance)
can we determine (p,q) uniquely for each image
point?
q
p
NO
Solution
• Add more constraints
• Shape-from-shading
• Take more images
• Photometric stereo
Stereographic Projection
(f,g)-space z
(p,q)-space (gradient space)
z
S
s
p
sˆ
z 1
1

N
1
q
n
s
f
z 1
nˆ
g
n
y
y
x
x
Problem
(p,q) can be infinite when
1
  90
f 
2p
1 1 p2  q2
Redefine reflectance map as R f , g 
g
2q
1 1 p2  q2
Occluding Boundaries
e
n
v
e
n
n  e, n  v  n  e  v
e and v are known
The n values on the occluding boundary can be used as
the boundary condition for shape-from-shading
Image Irradiance Constraint
• Image irradiance should match the
reflectance map
Minimize
ei 
 I x, y   R f , g  dxdy
2
image
(minimize errors in image irradiance in the image)
Smoothness Constraint
• Used to constrain shape-from-shading
• Relates orientations (f,g) of neighboring
surface points
Minimize
es 
2
2
2
2
f

f

g

g
  x y   x y dxdy
image
 f , g  : surface orientation under stereographic projection

f
f
g
g
fx  , f y  , gx 
, gy 
x
y
x
y
(penalize rapid changes in surface orientation f and g over the image)
Shape-from-Shading
• Find surface orientations (f,g) at all image
points that minimize
weight
e  es  ei
smoothness
constraint
image irradiance
error
Minimize
e
  f
image
2
x
f
2
y
 g
2
x

 g   I x, y   R f , g  dxdy
2
y
2
Results
Results
Solution
• Add more constraints
• Shape-from-shading
• Take more images
• Photometric stereo
Photometric Stereo
q
p , q 
p , q 
2
2
S
S
1
1
S
S
p
p , q 
3
3
S
S
Photometric Stereo
Lambertian case:
s2
s1
•
k

c
: source brightness
: surface albedo (reflectance)
: constant (optical system)

 kc 
I  kc cos  i  n  s   1



n
Image irradiance:
s3
v
We can write this in matrix form:
T


I
s
 1
1
 I    sT n
 2
 2
sT3 
 I 2 
 
I1  n  s1
I 2  n  s 2
I 3  n  s3
Solving the Equations
T

I
s
 1
1
 I   s T
 2
 2
T



I2 
s 3
I
31


 n


S
3 3
~  S 1I
n
~
n
~ n
~
n
n ~ 
n 
~
n
31
inverse
More than Three Light Sources
•
Get better results by using more lights
T
 I1   s 1 
       n
   
 I N  sTN 
•
Least squares solution:
~
I  Sn
~
ST I  ST Sn
N 1  N  33 1
 
1 T
T
~
n S S S I
•
Solve for  , n as before
Moore-Penrose pseudo inverse
Color Images
•
The case of RGB images
•
get three sets of equations, one per color channel:
I R   RSn
I G  GSn
I B   BSn
•
•
Simple solution: first solve for n using one channel
Then substitute known n into above equations to get
 R , G , B 
•
Or combine three channels and solve for
I  I 2R  I 2G  I 2B  Sn
n
Computing light source directions
•
Trick: place a chrome sphere in the scene
•
the location of the highlight tells you the source
direction
Specular Reflection - Recap
•
For a perfect mirror, light is reflected about N
n
s i i r
•
•
v
Ri
Re  
0
We see a highlight when
Then is given as follows:
s
if v  r
otherwise
v r
s  2n  r n  r
Computing the Light Source Direction
Chrome sphere that has a highlight at position h in the image
N
H
h
rN
c
C
sphere in 3D
image plane
•
Can compute N by studying this figure
•
Hints:
•
•
use this equation:
can measure c, h, and r in the image
Depth from Normals
V2
V1
N
•
Get a similar equation for V2
•
•
Each normal gives us two linear constraints on z
compute z values by solving a matrix equation
Limitations
• Big problems
• Doesn’t work for shiny things, semitranslucent things
• Shadows, inter-reflections
• Smaller problems
• Camera and lights have to be distant
• Calibration requirements
•
•
measure light source directions, intensities
camera response function
Trick for Handling Shadows
•
Weight each equation by the pixel brightness:
I i I i   I i n  si 
•
Gives weighted least-squares matrix equation:
 I12   I1sT1 
  

       n
 I N2   I N sTN 
  

•
Solve for
,n
as before
Original Images
Results - Shape
Shallow reconstruction
(effect of interreflections)
Accurate reconstruction
(after removing interreflections)
Results - Albedo
No Shading Information
Original Images
Results - Shape
Results - Albedo
Results
1.
2.
3.
4.
5.
Estimate light source directions
Compute surface normals
Compute albedo values
Estimate depth from surface normals
Relight the object (with original texture and uniform albedo)
Shape from texture
• Obtain normals from texture element (or
statistics) deformations,
Examples from Angie Loh
Shape from texture
• Obtain normals from texture element (or
statistics) deformations,
Examples from Angie Loh
Depth from focus
• Sweep through focus settings
• “most sharp” pixels correspond to depth
(most high frequencies)
Depth from Defocus
• More complicates, but needs less images
• Compare relative sharpness between
images
Shape from shadows
S. Savarese, M. Andreetto, H. Rusmeier, F.
Bernardini, P. Perona, “3D Reconstruction by
Shadow Carving: Theory and Practical Evaluation”,
International Journal of Computer Vision (IJCV) ,
vol 71, no. 3, pp. 305-336, March 2007.
Shape from specularities
Toward a Theory of Shape from Specular Flow
Y. Adato, Y. Vasilyev, O. Ben-Shahar, T. Zickler
Proc. ICCV’07
Next class:
3d modeling and registration