Transcript Computer Vision: Multiview Stereo

CSCE 641 Computer Graphics:
Image-based Modeling (Cont.)
Jinxiang Chai
Image-based Modeling and Rendering
Model
Panoroma
Image
based
modeling
Images
user input range
scans
Images + Depth
Geometry+ Images
Imagebased
rendering
Camera + geometry
Geometry+ Materials
Light field
Kinematics
Dynamics
Etc.
Images
Stereo reconstruction
Given two or more images of the same scene or object,
compute a representation of its shape
known
camera
viewpoints
Stereo reconstruction
Given two or more images of the same scene or object,
compute a representation of its shape
known
camera
viewpoints
How to estimate camera parameters?
- where is the camera?
- where is it pointing?
- what are internal parameters, e.g. focal length?
Spectrum of IBMR
Model
Panoroma
Image
based
modeling
Images
user input range
scans
Images + Depth
Geometry+ Images
Imagebased
rendering
Camera + geometry
Geometry+ Materials
Light field
Kinematics
Dynamics
Etc.
Images
Calibration from 2D motion
Structure from motion (SFM)
- track points over a sequence of images
- estimate for 3D positions and camera positions
- calibrate intrinsic camera parameters before hand
Self-calibration:
- solve for both intrinsic and extrinsic camera parameters
SFM = Holy Grail of 3D Reconstruction
Take movie of object
Reconstruct 3D model
Would be
commercially
highly viable
How to Get Feature Correspondences
Feature-based approach
- good for images
- feature detection (corners or sift features)
- feature matching using RANSAC (epipolar line)
Pixel-based approach
- good for video sequences
- patch based registration with lucas-kanade algorithm
- register features across the entire sequence
Structure from Motion
Two Principal Solutions
• Bundle adjustment (nonlinear optimization)
• Factorization (SVD, through orthographic approximation, affine
geometry)
Projection Matrix
Perspective projection:
ui   f x
v    0
 i 
 1   0
K
R

u0  r1T

v0   r2T
1  r3T
fy
0
T
P
x 
t1   i 
  yi 
t2  
 zi 

t3   
1
2D coordinates are just a nonlinear function of its
3D coordinates and camera parameters:
( f x r1T  r2T  u0 r3T )  P  f x t1  t 2  u0t3
ui 
r3T  P  t3
vi 
( f y r2T  v0 r3T )  P  f y t 2  t3
r  P  t3
T
3
f ( K , R, T ; Pi )
g ( K , R, T ; Pi )
Nonlinear Approach for SFM
What’s the difference between camera calibration and
SFM?
Nonlinear Approach for SFM
What’s the difference between camera calibration and
SFM?
- camera calibration: known 3D and 2D
M
N
arg min  (u
K ,{ R j },{T j }
j 1 i 1
i
j
 f ( K , R j , T j ; Pi ))2  (vij  g ( K , R j , T j ; Pi ))2
Nonlinear Approach for SFM
What’s the difference between camera calibration and
SFM?
- camera calibration: known 3D and 2D
M
N
arg min  (u
K ,{ R j },{T j }
j 1 i 1
 f ( K , R j , T j ; Pi ))2  (vij  g ( K , R j , T j ; Pi ))2
j
i
- SFM: unknown 3D and known 2D
M
N
arg min  (u
{ Pi }, K ,{ R j },{T j } j 1 i 1
i
j
 f ( K , R j , T j , Pi ))2  (vij  g ( K , R j , T j , Pi ))2
Nonlinear Approach for SFM
What’s the difference between camera calibration and
SFM?
- camera calibration: known 3D and 2D
M
N
arg min  (u
K ,{ R j },{T j }
j 1 i 1
 f ( K , R j , T j ; Pi ))2  (vij  g ( K , R j , T j ; Pi ))2
j
i
- SFM: unknown 3D and known 2D
M
N
arg min  (u
{ Pi }, K ,{ R j },{T j } j 1 i 1
i
j
 f ( K , R j , T j , Pi ))2  (vij  g ( K , R j , T j , Pi ))2
- what’s 3D-to-2D registration problem?
Nonlinear Approach for SFM
What’s the difference between camera calibration and
SFM?
- camera calibration: known 3D and 2D
M
N
arg min  (u
K ,{ R j },{T j }
j 1 i 1
 f ( K , R j , T j ; Pi ))2  (vij  g ( K , R j , T j ; Pi ))2
j
i
- SFM: unknown 3D and known 2D
M
N
arg min  (u
{ Pi }, K ,{ R j },{T j } j 1 i 1
i
j
 f ( K , R j , T j , Pi ))2  (vij  g ( K , R j , T j , Pi ))2
- what’s 3D-to-2D registration problem?
SFM: Bundle Adjustment
M
N
arg min  (u
{ Pi }, K ,{ R j },{T j } j 1 i 1
i
j
 f ( K , R j , T j , Pi ))2  (vij  g ( K , R j , T j , Pi ))2
SFM = Nonlinear Least Squares problem
Minimize through
•
•
•
•
Gradient Descent
Conjugate Gradient
Gauss-Newton
Levenberg Marquardt common method
Prone to local minima
Count # Constraints vs #Unknowns
M
N
arg min  (u
{ Pi }, K ,{ R j },{T j } j 1 i 1
i
j
 f ( K , R j , T j , Pi ))2  (vij  g ( K , R j , T j , Pi ))2
M camera poses
N points
2MN point constraints
6M+3N unknowns
Suggests: need 2mn  6m + 3n
But: Can we really recover all parameters???
Count # Constraints vs #Unknowns
M
N
arg min  (u
{ Pi }, K ,{ R j },{T j } j 1 i 1
i
j
 f ( K , R j , T j , Pi ))2  (vij  g ( K , R j , T j , Pi ))2
M camera poses
N points
2MN point constraints
6M+3N unknowns (known intrinsic camera parameters)
Suggests: need 2mn  6m + 3n
But: Can we really recover all parameters???
• Can’t recover origin, orientation (6 params)
• Can’t recover scale (1 param)
Thus, we need 2mn  6m + 3n - 7
Are We Done?
No, bundle adjustment has many local minima.
SFM Using Factorization
Assume an othorgraphic camera
Image
 xi 
 
ui  r1T t1   yi 
v    T t    z 
i
 i  r2 2 
 
1
World
SFM Using Factorization
Assume othorgraphic camera
Image
 xi 
 
ui  r1T t1   yi 
v    T t    z 
i
 i  r2 2 
 
1
Subtract the
mean
World
N


u

i

i 1
ui 
 r T   xi 
N   1  y 

 
N

 r2T   i 
vi 
 zi 


 vi  i 1 
N 

SFM Using Factorization
Stack all the features from the same frame:
 x1 x2 ... x N 
~
~
~
T
u1 u2 ...u N  r1  



y
y
...
y
N
 v~ v~ ... v~   T   1 2
N
 1 2
r2   z z ... z 
N 
 1 2
SFM Using Factorization
Stack all the features from the same frame:
 x1 x2 ... x N 
~
~
~
T
u1 u2 ...u N  r1  



y
y
...
y
N
 v~ v~ ... v~   T   1 2
N
 1 2
r2   z z ... z 
N 
 1 2
Stack all the features from all the images:
T
u~F ,1 u~F , 2 ...u~F , N   r1,1 
 v~ v~
~   r T   x1 x2 ... xN 
v
...
F ,N 
 1, 2  
 F ,1 F , 2



y
y
...
y


N
~ ~
  T   1 2
~
u F ,1 u F , 2 ...u F , N   rF ,1   z1 z 2 ... z N 

 v~ v~ ... v~  r T  
F ,N 
 F ,1 F , 2
 F ,2 
W
SFM Using Factorization
Stack all the features from the same frame:
 x1 x2 ... x N 
~
~
~
T
u1 u2 ...u N  r1  



y
y
...
y
N
 v~ v~ ... v~   T   1 2
N
 1 2
r2   z z ... z 
N 
 1 2
Stack all the features from all the images:
T
u~F ,1 u~F , 2 ...u~F , N   r1,1 
 v~ v~
~   r T   x1 x2 ... xN 
v
...
F ,N 
 1, 2  
 F ,1 F , 2



y
y
...
y


N
~ ~
  T   1 2
~
u F ,1 u F , 2 ...u F , N   rF ,1   z1 z 2 ... z N 

 v~ v~ ... v~  r T  
F ,N 
 F ,1 F , 2
 F ,2 
~
W2FN
W
M 2 F 3 S3N
SFM Using Factorization
Stack all the features from all the images:
T
u~F ,1 u~F , 2 ...u~F , N   r1,1 
 v~ v~
~   r T   x1 x2 ... xN 
v
...
F ,N 
 1, 2  
 F ,1 F , 2



y
y
...
y


N
~ ~
  T   1 2
~
u F ,1 u F , 2 ...u F , N   rF ,1   z1 z 2 ... z N 

 v~ v~ ... v~  r T  
F ,N 
 F ,1 F , 2
 F ,2 
~
W2FN
Factorize the matrix
SVD:
~
~
W2FN
W2 F N  UV T
W
M 2 F 3 S3N
into two matrix using
1
~
M 2 F 3  U 2
1
~
S3N   2V T
SFM Using Factorization
Stack all the features from all the images:
T
u~F ,1 u~F , 2 ...u~F , N   r1,1 
 v~ v~
~   r T   x1 x2 ... xN 
v
...
F ,N 
 1, 2  
 F ,1 F , 2



y
y
...
y


N
~ ~
  T   1 2
~
u F ,1 u F , 2 ...u F , N   rF ,1   z1 z 2 ... z N 

 v~ v~ ... v~  r T  
F ,N 
 F ,1 F , 2
 F ,2 
~
W2FN
Factorize the matrix
SVD:
~
~
W2FN
W2 F N  UV T
M 2 F 3 S3N
into two matrix using
1
~
M 2 F 3  U 2
Is the
Wsolution unique?
1
~
S3N   2V T
SFM Using Factorization
Stack all the features from all the images:
T
u~F ,1 u~F , 2 ...u~F , N   r1,1 
 v~ v~
~   r T   x1 x2 ... xN 
v
...
F ,N 
 1, 2  
 F ,1 F , 2



y
y
...
y


N
~ ~
  T   1 2
~
u F ,1 u F , 2 ...u F , N   rF ,1   z1 z 2 ... z N 

 v~ v~ ... v~  r T  
F ,N 
 F ,1 F , 2
 F ,2 
~
W2FN
Factorize the matrix
SVD:
~
~
W2FN
W2 F N  UV T
M 2 F 3 S3N
into two matrix using
1
~
M 2 F 3  U 2
~
M 2 F3  M 2 F3Q33
1
~
S3N   2V T
S3N  Q33
Is the solution unique? No!
~
S3N
1
SFM Using Factorization
Stack all the features from all the images:
T
u~F ,1 u~F , 2 ...u~F , N   r1,1 
 v~ v~
~   r T   x1 x2 ... xN 
v
...
F ,N 
 1, 2  
 F ,1 F , 2



y
y
...
y


N
~ ~
  T   1 2
~
u F ,1 u F , 2 ...u F , N   rF ,1   z1 z 2 ... z N 

 v~ v~ ... v~  r T  
F ,N 
 F ,1 F , 2
 F ,2 
~
W2FN
Factorize the matrix
SVD:
~
~
W2FN
W2 F N  UV T
M 2 F 3 S3N
into two matrix using
1
~
M 2 F 3  U 2
~
M 2 F3  M 2 F3Q33
How to compute theW
matrix Q33?
1
~
S3N   2V T
S3N  Q33
~
S3N
1
SFM Using Factorization
M is the stack of rotation matrix:
T
M 2 F 3 M 2 F 3
 r1T,1 
 T 
 r1, 2 
     r1,1
 rFT,1 
 T 
rF , 2 
 r1T,1r1,1 r1T,1r1, 2
 T
T
r
r
r
1
,
2
1
,
1
1
, 2 r1, 2

 


 
 


r1, 2  rF , 2






T
 rF ,1rF ,1
 rFT, 2 rF ,1
rF , 2 


 
 
T
rF ,1rF , 2 
rFT, 2 rF , 2 
SFM Using Factorization
M is the stack of rotation matrix:
T
M 2 F 3 M 2 F 3
Orthogonal constraints
from rotation matrix
 r1T,1 
 T 
 r1, 2 
     r1,1
 rFT,1 
 T 
rF , 2 
T
T
 r11
,1r1,1 r10
,1r1, 2
 T
T
r
r
r
0
1
1
,
2
1
,
1
1
, 2 r1, 2

 


 
 


r1, 2  rF , 2






T
 rF ,1r1F ,1
 rFT, 2 r0F ,1
rF , 2 


 
 
T
rF ,0
1rF , 2 

rFT,1
2 rF , 2 
SFM Using Factorization
M is the stack of rotation matrix:
T
M 2 F 3 M 2 F 3
Orthogonal constraints
from rotation matrix
 r1T,1 
 T 
 r1, 2 
     r1,1
 rFT,1 
 T 
rF , 2 
T
T
 r11
,1r1,1 r10
,1r1, 2
 T
T
r
r
r
0
1
1
,
2
1
,
1
1
, 2 r1, 2

 


 
 


r1, 2  rF , 2






T
 rF ,1r1F ,1
 rFT, 2 r0F ,1
~
T ~
T
 M 2F3Q33Q33 M 2F3
rF , 2 


 
 
T
rF ,0
1rF , 2 

rFT,1
2 rF , 2 
SFM Using Factorization
Orthogonal constraints from rotation matrices:
~
T ~
T
M 2F3Q33Q33 M 2F3
T
 r11
0T
,1r1,1 r1,1r1, 2
 T
T
, 2 r1,1 r11
, 2 r1, 2
r10
 


 
 








T
 rF ,1r1F ,1
 rFT, 2 r0F ,1


 
 
T
rF ,0
1rF , 2 

rFT,1
2 rF , 2 
SFM Using Factorization
Orthogonal constraints from rotation matrices:
T
 r11
0T
,1r1,1 r1,1r1, 2
 T
T
, 2 r1,1 r11
, 2 r1, 2
~
r10
T ~
T
M 2F3Q33Q33 M 2F3
 


 
QQ: symmetric 3 by 3 matrix
 








T
 rF ,1r1F ,1
 rFT, 2 r0F ,1


 
 
T
rF ,0
1rF , 2 

rFT,1
2 rF , 2 
SFM Using Factorization
Orthogonal constraints from rotation matrices:
T
 r11
0T
,1r1,1 r1,1r1, 2
 T
T
, 2 r1,1 r11
, 2 r1, 2
~
r10
T ~
T
M 2F3Q33Q33 M 2F3
 


 
QQ: symmetric 3 by 3 matrix
 








T
 rF ,1r1F ,1
 rFT, 2 r0F ,1


 
 
T
rF ,0
1rF , 2 

rFT,1
2 rF , 2 
How to compute QQT?
least square solution
- 4F linear constraints, 9 unknowns (6 independent due to symmetric matrix)
SFM Using Factorization
Orthogonal constraints from rotation matrices:
T
 r11
0T
,1r1,1 r1,1r1, 2
 T
T
, 2 r1,1 r11
, 2 r1, 2
~
r10
T ~
T
M 2F3Q33Q33 M 2F3
 


 
QQ: symmetric 3 by 3 matrix
 








T
 rF ,1r1F ,1
 rFT, 2 r0F ,1


 
 
T
rF ,0
1rF , 2 

rFT,1
2 rF , 2 
How to compute QQT?
least square solution
- 4F linear constraints, 9 unknowns (6 independent due to symmetric matrix)
How to compute Q from QQT:
SVD again: QQ  UV
T
Q  U
1
2
SFM Using Factorization
M is the stack of rotation matrix:
T
M 2 F 3 M 2 F 3
Orthogonal constraints
from rotation matrix
 r1T,1 
 T 
 r1, 2 
     r1,1
 rFT,1 
 T 
rF , 2 
T
T
 r11
,1r1,1 r10
,1r1, 2
 T
T
r
r
r
0
1
1
,
2
1
,
1
1
, 2 r1, 2

 


 
 


r1, 2  rF , 2






T
 rF ,1r1F ,1
 rFT, 2 r0F ,1
~
T ~
T
 M 2F3Q33Q33 M 2F3
QQT: symmetric 3 by 3 matrix
Computing QQT is easy:
- 3F linear equations
- 6 independent unknowns
rF , 2 


 
 
T
rF ,0
1rF , 2 

rFT,1
2 rF , 2 
SFM Using Factorization
~
1. Form the measurement matrix W2FN
~
2. ~Decompose the matrix into two matrices M 2 F3 and
S3N using SVD
3. Compute the matrix Q with least square and SVD
4. Compute the rotation matrix and shape matrix:
~
1 ~
M  M 2F3Q and S  Q S2 F 3
Weak-perspective Projection
Factorization also works for weak-perspective
projection (scaled orthographic projection):
d
z0
 xi 
r1T t1   yi 
ui 
v      T t    z 
 i
r2 2  i
 
1
Factorization for Full-perspective Cameras
[Han and Kanade]
SFM Summary
Bundle adjustment (nonlinear optimization)
- work with perspective camera model
- work with incomplete data
- prone to local minima
Factorization:
- closed-form solution for weak perspective camera
- simple and efficient
- usually need complete data
- becomes complicated for full-perspective camera model
Phil Torr’s structure from motion toolkit in matlab (click here)
Voodoo camera tracker (click here)
All Together Video
Click here
- feature detection
- feature matching (epipolar geometry)
- structure from motion
- stereo reconstruction
- triangulation
- texture mapping
SFM: Recent Development
• Large Scale SFM:
Reconstructing the World from Internet Photos
• Project website:
source code and data sets are available from
http://www.cs.cornell.edu/projects/bigsfm/