Transcript Computational Vision 493.69 – 051

3-D Computer Vision
CSc 83020
Camera Calibration
CSc 83020 3-D Computer Vision – Ioannis Stamos
Camera Calibration



Problem: Estimate camera’s extrinsic &
intrinsic parameters.
Method: Use image(s) of known scene.
Tools:



Geometric camera models.
SVD and constrained least-squares.
Line extraction methods.
CSc 83020 3-D Computer Vision – Ioannis Stamos
Camera Calibration
Perspective
Equations
Feature
Extraction
From Sebastian Thrun and Jana Kosecka
CSc 83020 3-D Computer Vision – Ioannis Stamos
Perspective Projection, Remember?
O
X
Z
x
f
From Sebastian Thrun and Jana Kosecka
CSc 83020 3-D Computer Vision – Ioannis Stamos
X
xf
Z
Intrinsic Camera Parameters


Determine the intrinsic parameters of a
camera (with lens)
What are Intrinsic Parameters?
(can you name 7?)
From Sebastian Thrun and Jana Kosecka
CSc 83020 3-D Computer Vision – Ioannis Stamos
Intrinsic Parameters
pixelsize sx , s y
O
X
Image center(ox, oy)
lens distortionk1 , k2
image centerox , oy
Z
f
focallength f
From Sebastian Thrun and Jana Kosecka
CSc 83020 3-D Computer Vision – Ioannis Stamos
X
xf
Z
Intrinsic Camera Parameters

Intrinsic Parameters:
 Focal Length f
 Pixel size sx , sy
 Image center ox , oy


(Nonlinear radial distortion coefficients k1 , k2…)
Calibration = Determine the intrinsic
parameters of a camera
From Sebastian Thrun and Jana Kosecka
CSc 83020 3-D Computer Vision – Ioannis Stamos
Coordinate Frames
Camera Coordinate Frame
Zc
Pixel Coordinates
Yc
Xc
Extrinsic
Parameters
Zw
Intrinsic
Parameters
Image Coordinate Frame
Yw
World Coordinate Frame
Xw
CSc 83020 3-D Computer Vision – Ioannis Stamos
Why Calibrate?
Image point
Scene ray
Image
Calibration: relates points in the image to rays in the scene
CSc 83020 3-D Computer Vision – Ioannis Stamos
Why Calibrate?
Image point
x
Scene ray
y
z
Image
x
y
Calibration: relates points in the image to rays in the scene
CSc 83020 3-D Computer Vision – Ioannis Stamos
Example Calibration Pattern
From Sebastian Thrun and Jana Kosecka
Calibration Pattern: Object with features of known size/geometry
CSc 83020 3-D Computer Vision – Ioannis Stamos
Harris Corner Detector
From Sebastian Thrun and Jana Kosecka
CSc 83020 3-D Computer Vision – Ioannis Stamos
Perspective Camera
r
(x,y,z)
r’
(X,Y,Z)
Center of
Projection
r =(x,y,z)
r’=(X,Y,Z)
r/f=r’/Z
f: effective focal length:
distance of image plane from O.
x=f * X/Z
y=f * Y/Z
z=f
Extrinsic Parameters
T
Pc=R(Pw-T)
Translation followed by rotation
Extrinsic Parameters (2nd formulation)
R same as before
Pc=R Pw +T
T different
Rotation followed by translation
The Rotation Matrix
R * RT= RT* R = I =>
R-1= RT
Orthonormal Matrix
Degrees of freedom?
I=
100
010
001
CSc 83020 3-D Computer Vision – Ioannis Stamos
Perspective Camera Model

Step 1: Transform into camera coordinates
~
 X C   cos sin  0   cos 0 sin    1
0
~  




 Y C     sin  cos 0   0
1
0   0 cos
 ~ C  
0
1    sin  0 cos   0  sin
 Z   0
0   X W   TX 
 W   
sin   Y    TY 
cos   Z W   TZ 
Step 2: Transform into image coordinates
f
xim  
sx
f
yim  
sy
From Sebastian Thrun and Jana Kosecka
~c
X
~ c  ox
Z
~c
Y
~ c  oy
Z
CSc 83020 3-D Computer Vision – Ioannis Stamos
Intrinsic Parameters
x 
y 
f
Z
f
Z
X
Y
CSc 83020 3-D Computer Vision – Ioannis Stamos
Image and Camera Frames
Zcamera
Ycamera
Ximage
(xim, yim)
Xcamera
(ox,oy)
Yimage
CSc 83020 3-D Computer Vision – Ioannis Stamos
Geometric Model
x=
y=
XcT
ZcT
YcT
ZcT
3D Point in
Camera
Coordinate
Frame
•Transformation from Image •Transformation from
to Camera Frame.
World to Camera Frame.
(ox,oy,sx,sy)
•Perspective projection
•No distortion!
(f, R, T)
Point in Camera Frame
CSc 83020 3-D Computer Vision – Ioannis Stamos
Camera Calibration: Issues

Which parameters need to be estimated.



What kind of accuracy is needed.


Focal length, image center, aspect ratio
Radial distortions
Application dependent
What kind of calibration object is used.


One plane, many planes
Complicated three dimensional object
CSc 83020 3-D Computer Vision – Ioannis Stamos
Camera Calibration
Calibration object
Extracted features
CSc 83020 3-D Computer Vision – Ioannis Stamos
Camera Calibration
Extract centers of circles
CSc 83020 3-D Computer Vision – Ioannis Stamos
Basic Equations
xim
f Xc

 ox
c
sx Z
yim
f Yc

 oy
c
sy Z
X c  r11 X w  r12Y w  r13 Z w  Tx
Y c  r21 X w  r22Y w  r23 Z w  Ty
Z c  r31 X w  r32Y w  r33 Z w  Tz
CSc 83020 3-D Computer Vision – Ioannis Stamos
Basic Equations
Xc
x   fx
 ox
c
Z
Yc
y   fy
 oy
c
Z
X c  r11 X w  r12Y w  r13 Z w  Tx
Y c  r21 X w  r22Y w  r23 Z w  Ty
Z c  r31 X w  r32Y w  r33 Z w  Tz
In the remaining slides
x means xim and y means yim
CSc 83020 3-D Computer Vision – Ioannis Stamos
Basic Equations
Xc
x   fx
 ox
c
Z
fx Y c
y
 oy
c
 Z
X c  r11 X w  r12Y w  r13 Z w  Tx
Y c  r21 X w  r22Y w  r23 Z w  Ty
Z c  r31 X w  r32Y w  r33 Z w  Tz
CSc 83020 3-D Computer Vision – Ioannis Stamos
Basic Equations
Extrinsic Parameters
1) Rotation matrix R (3x3)
2) Translation vector T (3x1)
Intrinsic Parameters
1) fx=f/sx, length in effective horizontal pixel size units.
2) α=sy/sx, aspect ratio.
3) (ox,oy), image center coordinates.
4) Radial distortion coefficients.
Total number of parameters (excluding distortion): ?
CSc 83020 3-D Computer Vision – Ioannis Stamos
Basic Equations
r11 X w  r12Y w  r13 Z w  Tx
x  ox   f x
r31 X w  r32Y w  r33 Z w  Tz
y  oy   f y
r21 X w  r22Y w  r23 Z w  Ty
r31 X w  r32Y w  r33 Z w  Tz
1) Assume that image center is known.
2) Solve for the remaining parameters.
3) Use N image points ( xi , yi ) and their
corresponding
w
w
w T
[
X
,
Y
,
Z
N world points
i
i
i ]
CSc 83020 3-D Computer Vision – Ioannis Stamos
Basic Equations
r11 X w  r12Y w  r13 Z w  Tx
x   fx
r31 X w  r32Y w  r33 Z w  Tz
y   fy
r21 X w  r22Y w  r23 Z w  Ty
r31 X w  r32Y w  r33 Z w  Tz
1) Assume that image center is known.
2) Solve for the remaining parameters.
3) Use N image points ( xi , yi ) and their
corresponding
w
w
w T
[
X
,
Y
,
Z
N world points
i
i
i ]
Here x means xim - ox and y means yim - oy
CSc 83020 3-D Computer Vision – Ioannis Stamos
(1)
Basic Equations
xi f y ( r21 X iw  r22Yi w  r23 Z iw  Ty )

yi f x ( r11 X iw  r12Yi w  r13 Z iw  Tx )
1) Assume that image center is known.
2) Solve for the remaining parameters.
3) Use N image points ( xi , yi ) and their
corresponding
N world points [ X iw ,Yi w , Ziw ]T
CSc 83020 3-D Computer Vision – Ioannis Stamos
(2)
Basic Equations
xi X iw v1  xiYi w v2  xi Z iw v3  xi v4
 yi X iw v5  yiYi w v6  yi Z iw v7  yi v8  0
v1  r21 , v5  r11
v2  r22 , v6  r12
v3  r23 , v7  r13
v4  Ty , v8  Tx
CSc 83020 3-D Computer Vision – Ioannis Stamos
(3)
Basic Equations
xi X iw v1  xiYi w v2  xi Z iw v3  xi v4
 yi X iw v5  yiYi w v6  yi Z iw v7  yi v8  0
v1  r21 , v5  r11
v2  r22 , v6  r12
v3  r23 , v7  r13
v4  Ty , v8  Tx
Av  0
How would we solve this system?
CSc 83020 3-D Computer Vision – Ioannis Stamos
(3)
Basic Equations
 x1 X 1w

w
x
X
2
2

A 
.

.

w

 xN X N
x1Y1w
x1Z1w
x1
 y1 X 1w
 y1Y1w
 y1Z1w
x2Y2w
x2 Z 2w
x2
 y2 X 2w
 y2Y2w
 y2 Z 2w
.
.
.
.
.
.
.
.
.
.
.
.
x N YNw
x N Z Nw
xN
 y N X Nw
 y N YNw
 y N Z Nw
Av  0
(3)
How would we solve this system?
Rank of matrix A?
Solution up to a scale factor.
CSc 83020 3-D Computer Vision – Ioannis Stamos
 y1 

 y2 
. 

. 
 yN 

Singular Value Decomposition
Appendix A.6 (Trucco)
A  UDV
T
A: m x n
U: m x m, columns orthogonal unit vectors.
V: n x n ,
-//D: m x n , diagonal. The diagonal elements
σi are the singular values
σ1>= σ2>= … >= σn >= 0
CSc 83020 3-D Computer Vision – Ioannis Stamos
Singular Value Decomposition
Appendix A.6 (Trucco)
A  UDV
T
Square A non-singular iff σi != 0
For square A C=σ1/σN is the condition number
For rectangular A # of non-zero σi is the rank
For square non-singular A: A1  VD 1U T
For square A, pseudoinverse: A  VD01U T
Singular values of A = square roots of
eigenvalues of AAT and AT A
7. Columns of U, V
Eigenvectors of AAT
AT A
8. Frobenius norm of a matrix
1.
2.
3.
4.
5.
6.
CSc 83020 3-D Computer Vision – Ioannis Stamos
Singular Value Decomposition
Appendix A.6 (Trucco)
A  UDV
T
Av  0
If rank(A)=n-1 (7 in our case) then
the solution is the eigenvector which
corresponds to the ONLY zero eigenvalue.
Solution up to a scale factor.
CSc 83020 3-D Computer Vision – Ioannis Stamos
Solving for v
 x1 X 1w

w
x
X
2
2

A 
.

.

w

 xN X N
x1Y1w
x1Z1w
x1
 y1 X 1w
 y1Y1w
 y1Z1w
x2Y2w
x2 Z 2w
x2
 y2 X 2w
 y2Y2w
 y2 Z 2w
.
.
.
.
.
.
.
.
.
.
.
.
x N YNw
x N Z Nw
xN
 y N X Nw
 y N YNw
 y N Z Nw
Av  0
(3)
 y1 

 y2 
. 

. 
 yN 

How would we solve this system: SVD.
Solution: v   ( r21 , r22 , r23 , Ty ,r11 , r12 ,r13 , Tx )
Uknown scale factor γ=?
Aspect ratio α=?
Solving for Tz and fx?
CSc 83020 3-D Computer Vision – Ioannis Stamos
Solving for Tz and fx?
xi ( r31 X i
w
 r32Yi
 f x ( r11 X i
w
w
 r33 Z i
 r12Yi
w
w
 Tz ) 
 r13 Z i
w
 Tx )
 Tz 
A
 f 
b
 x
How would we solve this system?
CSc 83020 3-D Computer Vision – Ioannis Stamos
Solving for Tz and fx?
xi ( r31 X i
w
 r32Yi
 f x ( r11 X i
w
w
 r33 Z i
 r12Yi
w
w
 Tz ) 
 r13 Z i
w
 Tx )
 Tz 
A
 f 
b
 x
How would we solve this system?
 T^ 
 z   ( AT A)1 AT b
 ^ 
 fx 
Solution in the least squares sense.
CSc 83020 3-D Computer Vision – Ioannis Stamos
Camera Center
Camera Models (linear versions)
x=
3D Point in
World Coordinate
Frame
y=
•Transformation from Image •Transformation from
World to Camera Frame.
to Camera Frame.
•Perspective projection
(ox,oy,sx,sy)
(f, R, T)
•No distortion!
Point in Camera Frame
CSc 83020 3-D Computer Vision – Ioannis Stamos
Camera Models (linear versions)
Elegant decomposition.
No distortion!
Homogeneous
Coordinates
Measured Pixel
(xim, yim)
?
World Point
(Xw, Yw,Zw)
Camera Calibration – Other method
X 
u 
Y 
 v   P 
 
Z 
 w
 
1
u
x
w
v
y
w
Extracted features
Step 1: Estimate P
Step 2: Decompose P into internal and external parameters R,T,C
CSc 83020 3-D Computer Vision – Ioannis Stamos
Camera Calibration: Step 1
X 
u 
Y 
 v   P 
 
Z 
 w
 
1
u
x
w
v
y
w
wx  u
wy  v
w
Extracted features
x p31 X  p32Y  p33 Z  p34   p11 X  p12Y  p13 Z  p14
y p31 X  p32Y  p33 Z  p34   p21 X  p22Y  p23 Z  p24
Each point (x,y) gives us two equations
u
v
Camera Calibration: Step 1
X

0
0  xX
 xY
 xZ
 yX
 yY
 yZ
Y
Z 1
0
0
0
0
0
X
Y
Z 1
0
 p11 
p 
 12 
 p13 
 
 p14 
 p21 
 
 x   p22  0
 

 y   p23  0
 
 p24 
p 
 31 
 p32 
 
 p33 
 p34 
Extracted features
x p31 X  p32Y  p33 Z  p34   p11 X  p12Y  p13 Z  p14
y p31 X  p32Y  p33 Z  p34   p21 X  p22Y  p23 Z  p24
Each corner (x,y) gives us two equations
Camera Calibration: Step 1
2n
X

0





X

0
0  xX
 xY
 xZ
 yX
 yY
 yZ
0  xX
 xY
 xZ
 yX
 yY
 yZ
Y
Z 1
0
0
0
0
0
0
X
Y
Z 1
Y
Z 1
0
0
0
0
0
X
Y
Z 1
0
 p11 
p 
 12 
 
 x   p13 
 p
 y   14  0
 
  p21  0
  p22   
    
  p23  0
 p   
24
 x    0
  p31 
 y   p 
32
 
 p33 
 p34 
A
Extracted features
n points gives us 2n equations
CSc 83020 3-D Computer Vision – Ioannis Stamos
Camera Calibration: Step 1
2n
X

0





X

0
0  xX
 xY
 xZ
 yX
 yY
 yZ
0  xX
 xY
 xZ
 yX
 yY
 yZ
Y
Z 1
0
0
0
0
0
0
X
Y
Z 1
Y
Z 1
0
0
0
0
0
X
Y
Z 1
0
 p11 
p 
 12 
 
 x   p13 
 p
 y   14  0
 
  p21  0
  p22   
    
  p23  0
 p   
24
 x    0
  p31 
 y   p 
32
 
 p33 
 p34 
A
We need to solve
Extracted features
Ap  0
In the presence of noise we need to solve
min
Ap
p
The solution is given by the eigenvector with the smallest eigenvalue of AT A
CSc 83020 3-D Computer Vision – Ioannis Stamos
Camera Calibration: Step 1
The result can be improved through
non-linear minimization.
2
2




ui
vi 

min   xi     yi  
p

wi  
wi  
i


Xi 
 ui 
Y 
 v   P i 
 i
 Zi 
 wi 
 
1
Extracted features
CSc 83020 3-D Computer Vision – Ioannis Stamos
Camera Calibration: Step 1
The result can be improved through
non-linear minimization.
2
2




ui
vi 

min   xi     yi  
p

wi  
wi  
i


Xi 
 ui 
Y 
 v   P i 
 i
 Zi 
 wi 
 
1
Extracted features
Minimize the distance between the
predicted and detected features.
CSc 83020 3-D Computer Vision – Ioannis Stamos