Transcript PPT
Last 4 lectures
Camera Structure
Image Filtering
HDR
Image Transform
Today
Camera Projection
Camera Calibration
Pinhole camera
Pinhole camera model
(X,Y,Z)
P
origin
p
(x,y)
(optical
center)
principal point
• The coordinate system
– We will use the pin-hole model as an approximation
– Put the optical center (Center Of Projection) at the origin
– Put the image plane (Projection Plane) in front of the COP (Why?)
Pinhole camera model
fX
x
Z
fY
y
Z
principal
point
x fX f
y ~ fY 0
1 Z 0
0
f
0
0 X
0 Y
1 Z
Pinhole camera model
(X,Y,Z)
P
origin
p
(x,y)
(optical
center)
principal point
y
x
x fX f
y ~ fY 0
1 Z 0
0
f
0
0 X
0 Y
1 Z
Pinhole camera model
(X,Y,Z)
P
origin
p
(x,y)
(optical
center)
principal point
y
x
x fX f
y ~ fY 0
1 Z 0
0
f
0
x0 X
y0 Y
1 Z
Intrinsic matrix
Is this form of K good enough?
f
K 0
0
0
f
0
xc
yc
1
• non-square pixels (digital video)
fx
K 0
0
0
fy
0
xc
yc
1
Intrinsic matrix
Is this form of K good enough?
f
K 0
0
• non-square pixels (digital video)
• skew
f
x
K 0
0
0
f
0
s
fy
0
xc
yc
1
xc
yc
1
Intrinsic matrix
Is this form of K good enough?
f
K 0
0
• non-square pixels (digital video)
• skew
fx
• radial distortion
K0
0
0
f
0
s
fy
0
xc
yc
1
xc
yc
1
Distortion
• Radial distortion of the image
– Caused by imperfect lenses
– Deviations are most noticeable for rays that pass
through the edge of the lens
Barrel Distortion
No distortion
Wide Angle Lens
Barrel
Pin Cushion Distortion
No distortion
Telephoto lens
Pin cushion
Modeling distortion
Distortion-Free:
fX
x
Z
fY
y
Z
With Distortion:
1. Project (X, Y, Z)
to “normalized”
image coordinates
X
xn
Z
Y
yn
Z
r 2 xn2 yn2
2. Apply radial distortion
yd
3. Apply focal length
translate image center
y 1 r
r
xd xn 1 1r 2 2 r 4
n
x fxd xc
y fyd yc
• To model lens distortion
– Use above projection operation instead of
standard projection matrix multiplication
1
2
4
2
Camera rotation and translation
X '
X
Y ' R 33 Y t
Z'
Z
x f
y ~ 0
1 0
0
f
0
X
xc
Y
yc R t
Z
1
1
x ~ KR t X
extrinsic matrix
Two kinds of parameters
• internal or intrinsic parameters: focal length,
optical center, skew
• external or extrinsic (pose): rotation and
translation:
x f
y ~ 0
1 0
0
f
0
X
a
xc
Y
yc R t e
Z
i
1
1
b
f
j
c
g
k
X
d
Y
h
Z
l
1
Other projection models
Orthographic projection
• Special case of perspective projection
– Distance from the COP to the PP is infinite
Image
World
– Also called “parallel projection”: (x, y, z) → (x, y)
Other types of projections
• Scaled orthographic
– Also called “weak perspective”
• Affine projection
– Also called “paraperspective”
Fun with perspective
Perspective cues
Perspective cues
Fun with perspective
Ames room
Forced perspective in LOTR
Elijah Wood: 5' 6" (1.68 m)
Ian McKellen 5' 11" (1.80 m)
Camera calibration
Camera calibration
x f
y ~ 0
1 0
0
f
0
X
a
xc
Y
yc R t e
Z
i
1
1
b
f
j
c
g
k
X
d
Y
h
Z
l
1
• Estimate both intrinsic and extrinsic
parameters
• Mainly, two categories:
1. Using objects with known geometry as
reference
2. Self calibration (structure from motion)
Camera calibration approaches
• Directly estimate 11
unknowns in the M matrix
using known 3D points
(Xi,Yi,Zi) and measured
feature positions (ui,vi)
x ~ KR t X MX
Linear regression
Linear regression
Xi
0
Yi
Zi
1
0
0
0
0 ui X i
uiYi
0
0
0
Xi
Yi
Zi
1 vi X i
viYi
m00
m
01
m02
m03
m10
ui Z i ui
m11
viYi vi
m
12
m13
m
20
m21
m22
Linear regression
X 1 Y1 Z1 1 0 0 0
0 0 0 0 X 1 Y1 Z1
0
0
X N YN Z N 1 0
0
0
0 0 X N YN Z N
0 u1 X 1
1 v1 X 1
0 uN X N
1 vN X N
m00
m
01
m02
u1Y1 u1Z1
m03
m10
v1Y1 v1Y1
ui
m11 v
u N YN u N Z N i
m12
v N YN v N YN
m13
m
20
m21
m22
Solve for Projection Matrix M using leastsquare techniques
Normal equation (Geometric Interpretation)
Given an overdetermined system
Ax b
the normal equation is that which minimizes
the sum of the square differences between
left and right sides
min Ax b
2
A ( Ax b) 0
1 T
T
x A A A b
T
Normal equation (Differential Interpretation)
E (x) Ax b
2
a11 ... a1m
b1
:
x
:
:
1
:
: : :
x
:
:
m
:
an1 ... anm
bn
nxm, n equations, m variables
Normal equation
E (x) Ax b
2
Ax b Ax b
T
x T A T b T Ax b
x A Ax b Ax x A b b b
T
T
T
T
T
T
T
T
Carl Friedrich Gauss
x A Ax A b x A b x b b
T
T
T
E
T
T
2A Ax 2A b
x
T
T
Who invented Least Square?
Nonlinear optimization
• A probabilistic view of least square
• Feature measurement equations
• Likelihood of M given {(ui,vi)}
Optimal estimation
• Log likelihood of M given {(ui,vi)}
2
m00 X i m01Yi m02 Z i m03
m10 X i m11Yi m12 Z i m13
1
1
2 vi
2 ui
m20 X i m21Yi m22 Z i 1 i
m20 X i m21Yi m22 Z i 1
i 1 i
N
• It is a least square problem (but not
necessarily linear least square)
• How do we minimize C?
2
Nonlinear least square methods
Least square fitting
number of data points
number of parameters
Nonlinear least square fitting
Function minimization
Least square is related to function minimization.
It is very hard to solve in general. Here, we only
consider a simpler problem of finding local minimum.
Function minimization
Quadratic functions
Approximate the function with
a quadratic function within
a small neighborhood
Function minimization
Computing gradient and Hessian
1 m
2
F (x h) f i (x h)
2 i 1
f
1
1 fi
f i (x) i h h T
2 i 1
x
2
x 2
m
T
2
h
2
T
T
2
f
f
f
f
1 m
2
T
i
i
i
i
h
f i (x) 2 f i (x) h h f i (x) 2
2 i 1
x
x
x
x
T
2
m
f
f
f
f
1
1
2
T
i
i
i
i
f i ( x) f i ( x) h h f i ( x) 2
h
2 i 1
x
2 i 1
x
x x
i 1
m
m
T
T
m
f
f
f
1
1
2
T
i
i
i
h
f i (x) f i (x) h h
2 i 1
x
2
x
x
i
1
i
1
m
m
Gradient
T
Hessian
Computing gradient and Hessian
1 m
2
F (x h) f i (x h)
2 i 1
f
1
1 fi
f i (x) i h h T
2 i 1
x
2
x 2
m
T
2
h
2
T
T
2
f
f
f
f
1 m
2
T
i
i
i
i
h
f i (x) 2 f i (x) h h f i (x) 2
2 i 1
x
x
x
x
T
2
m
f
f
f
f
1
1
2
T
i
i
i
i
f i ( x) f i ( x) h h f i ( x) 2
h
2 i 1
x
2 i 1
x
x x
i 1
m
m
T
T
m
f
f
f
1
1
2
T
i
i
i
h
f i (x) f i (x) h h
2 i 1
x
2
x
x
i
1
i
1
m
m
Gradient
T
Hessian
Computing gradient and Hessian
1 m
2
F (x h) f i (x h)
2 i 1
f
1
1 fi
f i (x) i h h T
2 i 1
x
2
x 2
m
T
2
h
2
T
T
2
f
f
f
f
1 m
2
T
i
i
i
i
h
f i (x) 2 f i (x) h h f i (x) 2
2 i 1
x
x
x
x
T
2
m
f
f
f
f
1
1
2
T
i
i
i
i
f i ( x) f i ( x) h h f i ( x) 2
h
2 i 1
x
2 i 1
x
x x
i 1
m
m
T
T
m
f
f
f
1
1
2
T
i
i
i
h
f i (x) f i (x) h h
2 i 1
x
2
x
x
i
1
i
1
m
m
Gradient
T
Hessian
Computing gradient and Hessian
1 m
2
F (x h) f i (x h)
2 i 1
f
1
1 fi
f i (x) i h h T
2 i 1
x
2
x 2
m
T
2
h
2
T
T
2
f
f
f
f
1 m
2
T
i
i
i
i
h
f i (x) 2 f i (x) h h f i (x) 2
2 i 1
x
x
x
x
T
2
m
f
f
f
f
1
1
2
T
i
i
i
i
f i ( x) f i ( x) h h f i ( x) 2
h
2 i 1
x
2 i 1
x
x x
i 1
m
m
T
T
m
f
f
f
1
1
2
T
i
i
i
h
f i (x) f i (x) h h
2 i 1
x
2
x
x
i
1
i
1
m
m
Gradient
T
Hessian
Computing gradient and Hessian
1 m
2
F (x h) f i (x h)
2 i 1
f
1
1 fi
f i (x) i h h T
2 i 1
x
2
x 2
m
T
2
h
2
T
T
2
f
f
f
f
1 m
2
T
i
i
i
i
h
f i (x) 2 f i (x) h h f i (x) 2
2 i 1
x
x
x
x
T
2
m
f
f
f
f
1
1
2
T
i
i
i
i
f i ( x) f i ( x) h h f i ( x) 2
h
2 i 1
x
2 i 1
x
x x
i 1
m
m
T
T
m
f
f
f
1
1
2
T
i
i
i
h
f i (x) f i (x) h h
2 i 1
x
2
x
x
i
1
i
1
m
m
Gradient
T
Hessian
Searching for update h
Gradient
Hessian
1 m
1 T
2
T
F (x h) f i (x) g h h Hh
2 i 1
2
Idea 1: Steepest Descent
Let h g
1 m
2 T
2
T
F ( x h ) f i ( x ) g g
g Hg
2 i 1
2
gTg
T
g Hg
Steepest descent method
isocontour
gradient
Steepest descent method
It has good performance in the initial stage of the iterative
process. Converge very slow with a linear rate.
Searching for update h
Gradient
Hessian
1 m
1 T
2
T
F (x h) f i (x) g h h Hh
2 i 1
2
Idea 2: minimizing the quadric directly
Let
F (x h) g Hh 0
h
h H 1g
Converge faster but needs to solve the linear system
Recap: Calibration
• Directly estimate 11
unknowns in the M matrix
using known 3D points
(Xi,Yi,Zi) and measured
feature positions (ui,vi)
x ~ KR t X MX
Camera Model:
Recap: Calibration
• Directly estimate 11
unknowns in the M matrix
using known 3D points
(Xi,Yi,Zi) and measured
feature positions (ui,vi)
x ~ KR t X MX
Linear Approach:
Recap: Calibration
• Directly estimate 11
unknowns in the M matrix
using known 3D points
(Xi,Yi,Zi) and measured
feature positions (ui,vi)
x ~ KR t X MX
NonLinear Approach:
2
m00 X i m01Yi m02 Z i m03
m10 X i m11Yi m12 Z i m13
ui
vi
m20 X i m21Yi m22 Z i 1
m20 X i m21Yi m22 Z i 1
i 1
N
2
Practical Issue
is hard to make and the 3D feature
positions are difficult to measure!
A popular calibration tool
Multi-plane calibration
Advantage
Images courtesy Jean-Yves Bouguet, Intel Corp.
• Only requires a plane
• Don’t have to know positions/orientations
• Good code available online!
–
Intel’s OpenCV library: http://www.intel.com/research/mrl/research/opencv/
–
Matlab version by Jean-Yves Bouget:
http://www.vision.caltech.edu/bouguetj/calib_doc/index.html
–
Zhengyou Zhang’s web site: http://research.microsoft.com/~zhang/Calib/
Step 1: data acquisition
Step 2: specify corner order
Step 3: corner extraction
Step 3: corner extraction
Step 4: minimize projection error
Step 4: camera calibration
Step 4: camera calibration
Step 5: refinement