Transcript Multiple View Geometry in Computer Vision

Geometric
Computer Vision
Marc Pollefeys
Fall 2009
http://www.inf.ethz.ch/personal/pomarc/courses/gcv/
Geometric Computer Vision course schedule
(tentative)
Lecture
Exercise
Sept 16
Introduction
-
Sept 23
Geometry & Camera model
Camera calibration
Sept 30
Single View Metrology
Measuring in images
(Changchang Wu)
Oct. 7
Feature Tracking/Matching
Correspondence computation
Oct. 14
Epipolar Geometry
F-matrix computation
Oct. 21
Shape-from-Silhouettes
Visual-hull computation
Oct. 28
Multi-view stereo matching
Project proposals
Nov. 4
Structure from motion and visual SLAM
Papers
Nov. 11
Multi-view geometry and
self-calibration
Papers
Nov. 18
Shape-from-X
Papers
Nov. 25
Structured light and active range
sensing
Papers
Dec. 2
3D modeling, registration
and range/depth fusion
Papers
(Christopher Zach?)
Dec. 9
Appearance modeling and imagebased rendering
Papers
Dec. 16
Final project presentations
Final project presentations
Projective Geometry and
Camera model
Class 2
points, lines, planes
conics and quadrics
transformations
camera model
Read tutorial chapter 2 and 3.1
http://www.cs.unc.edu/~marc/tutorial/
Chapter 1, 2 and 5 in Hartley and Zisserman
Homogeneous coordinates
Homogeneous representation of 2D points and lines
ax + by + c = 0
(a,b,c)T (x,y,1) = 0
The point x lies on the line l if and only if
lT x = 0
Note that scale is unimportant for incidence relation
(a,b,c)T ~ k (a,b,c)T ,∀k ≠0
(x, y,1)T ~ k (x, y,1)T ,∀k ≠0
equivalence class of vectors, any vector is representative
Set of all equivalence classes in R3(0,0,0)T forms P2
(
Homogeneous coordinates x1 , x2 , x3
T
Inhomogeneous coordinates (x, y)
)
T
but only 2DOF
Points from lines and vice-versa
Intersections of lines
The intersection of two lines l and l' is x = l× l'
Line joining two points
The line through two points x and x' is l = x× x'
Example
Note:
x × x' = [x]× x'
 x
 
(0,1,-1) y 
1
 
with
y =1
x =1
 x
 
(1,0,-1) y 
1
 
x 
 0 z - y
 - z 0
x 
 y - x 0 
Ideal points and the line at infinity
Intersections of parallel lines
l = (a, b, c)T and l' = (a, b, c')T
Example
l l'  b,a,0
T
b, -a
(a, b)
tangent vector
normal direction
x =1 x = 2
Ideal points
Line at infinity
(x , x ,0)
T
1
2
l∞ = (0,0,1)T
P 2 = R 2 ∪l∞
Note that in P2 there is no distinction
between ideal points and others
3D points and planes
Homogeneous representation of 3D points and planes
π1 X1 + π2 X 2 + π3 X 3 + π4 X 4 = 0
The point X lies on the plane π if and only if
πT X = 0
The plane π goes through the point X if and only if
πT X = 0
Planes from points
Solveπ from X1T π = 0, XT2 π = 0 andX3T π = 0
X1T 
 T
X 2  π  0
X 3T 
 
 X 1T 
 
(solve π as right nullspace of  X T2  )
 X 3T 
 
Points from planes
SolveX from π1T X = 0, πT2 X = 0 and π3T X = 0
 π1T 
 T
π 2  X  0
 π 3T 
 
(solve Xas right nullspace of
Representing a plane by its span
X=Mx
πT M = 0
M  X1X2X3 
 π1T 
 T)
π 2 
 π 3T 
 
Lines
Representing a line by its span
(4dof)
A T 
W   T
B 
λA μB
Dual representation
T


P
*
W   T
Q 
λP μQ
T
W*WT = WW* = 02×2
Example: X-axis
0 0 0 1
W

1
0
0
0


0 0 1 0
W 

0
1
0
0


*
(Alternative: Plücker representation, details see e.g. H&Z)
Points, lines and planes
W
M   T
X 
W* 
M T
π 
W
Mπ =0
X
W*
MX 0
π
Conics
Curve described by 2nd-degree equation in the plane
ax2 + bxy + cy2 + dx + ey + f = 0
or homogenized x  x1 x , y  x2 x
3
3
ax12 + bx1x2 + cx2 2 + dx1x3 + ex2 x3 + fx32 = 0
or in matrix form
b / 2 d / 2
 a

e / 2 
x T C x = 0 with C   b / 2 c
d / 2 e / 2
f 
5DOF:
{a : b : c : d : e : f }
Five points define a conic
For each point the conic passes through
axi2 + bxi yi + cyi2 + dxi + eyi + f = 0
or
(x , x y , y , x , y ,1).c = 0
2
i
i
i
2
i
i
i
stacking constraints yields
 x12
 2
 x2
 x32
 2
 x4
x2
 5
x1 y1
y12
x1
x2 y 2
y22
x2
x3 y3
y32
x3
x4 y 4
y42
x4
x5 y5
y52
x5
y1 1

y2 1
y3 1c  0

y4 1
y5 1
c = (a, b, c, d , e, f )T
Tangent lines to conics
The line l tangent to C at point x on C is given by l=Cx
x
l
C
Dual conics
A line tangent to the conic C satisfies
In general (C full rank):
lT C* l = 0
C*  C-1
Dual conics = line conics = conic envelopes
Degenerate conics
A conic is degenerate if matrix C is not of full rank
m
e.g. two lines (rank 2)
l
C = lmT + mlT
e.g. repeated line (rank 1)
C = llT
l
Degenerate line conics: 2 points (rank 2), double point (rank1)
Note that for degenerate conics
(C* )* ≠C
Quadrics and dual quadrics
XT QX = 0
•
•
•
•
(Q : 4x4 symmetric matrix)
9 d.o.f.
in general 9 points define quadric
det Q=0 ↔ degenerate quadric
tangent plane π = QX


Q
















π T Q*π = 0
•
relation to quadric Q* = Q-1 (non-degenerate)
2D projective transformations
Definition:
A projectivity is an invertible mapping h from P2 to itself
such that three points x1,x2,x3 lie on the same line if and
only if h(x1),h(x2),h(x3) do.
Theorem:
A mapping h:P2P2 is a projectivity if and only if there
exist a non-singular 3x3 matrix H such that for any point
in P2 reprented by a vector x it is true that h(x)=Hx
Definition: Projective transformation
 x'1   h11
  
 x'2   h21
 x'  h
 3   31
h12
h22
h32
h13  x1 
 

h23  x2 
h33  x3 
or
x' = H x
8DOF
projectivity=collineation=projective transformation=homography
Transformation of 2D points, lines
and conics
For a point transformation
x' = H x
Transformation for lines
l' = H -T l
Transformation for conics
C' = H-TC H-1
Transformation for dual conics
C'* = HC*HT
Fixed points and lines
He = λe
H
T
(eigenvectors H =fixed points)
(1=2  pointwise fixed line)
l = λ l (eigenvectors H-T =fixed lines)
Hierarchy of 2D transformations
Projective
8dof
Affine
6dof
Similarity
4dof
Euclidean
3dof
 h11 h12
h
 21 h22
 h31 h32
 a11 a12
a
 21 a22
 0
0
h13 
h23 
h33 
tx 
t y 
1 
 sr11 sr12 t x 
 sr

sr
t
22
y
 21
 0
0
1 
 r11 r12 t x 
r

r
t
21
22
y


 0 0 1 
transformed
squares
invariants
Concurrency, collinearity,
order of contact (intersection,
tangency, inflection, etc.),
cross ratio
Parallellism, ratio of areas,
ratio of lengths on parallel
lines (e.g midpoints), linear
combinations of vectors
(centroids).
The line at infinity l∞
Ratios of lengths, angles.
The circular points I,J
lengths, areas.
The line at infinity
l  H
 A
l   T T
 t A
T
A 
T
0
0  
  0   l
1  
1
The line at infinity l is a fixed line under a projective
transformation H if and only if H is an affinity
Note: not fixed pointwise
Affine properties from images
projection
1 0 0 
H PA   0 1 0  H A
l1 l2 l3 
rectification
l∞  l1 l2 l3  , l3 ≠0
T
Affine rectification
l∞
v1
l1
l2
l3
v2
v 2 = l3 × l 4
l4
v1  l1  l2
l∞  v1  v2
The circular points
1
 
Ii
0
 
 s cos
I  H S I   s sin 
 0
1
 
J  i
0
 
s sin 
s cos
0
t x  1 
1


 

i
t y  i   se  i   I
0
1  0 
 
The circular points I, J are fixed points under the
projective transformation H iff H is a similarity
The circular points
“circular points”
x12 + x22 + dx1x3 + ex2 x3 + fx32 = 0
x3  0
l∞
x12 + x22 = 0
I  1, i,0
T
J  1,-i,0
T
Algebraically, encodes orthogonal directions
T
T
(
)
(
)
I = 1,0,0 + i 0,1,0
Conic dual to the circular points
C*∞  IJT  JI T
1 0 0
C*∞  0 1 0
0 0 0
C*∞  HS C*∞HTS
l∞
I
J
The dual conic C*∞ is fixed conic under the
projective transformation H iff H is a similarity
Note: C*∞has 4DOF
l∞ is the nullvector
Euclidean:
Angles
T
T
(
)
(
)
l = l1 , l2 , l3
m = m1, m2 , m3
cosθ =
Projective: cos  
l1m1 + l2 m2
(l
2
1
l
+ l22 )(m12 + m22 )
l T C* m
T

C* l m T C* m
lT C*∞ m = 0 (orthogonal)

Transformation of 3D points,
planes and quadrics
For a point transformation
X' = H X
(cfr. 2D equivalent)
(x'= H x)
Transformation for lines
π'= H -T π
(l' = H-T l)
Transformation for conics
Q' = H-TQH-1
(C' = H-TCH-1)
Transformation for dual conics
Q'* = HQ*HT
(C'* = HC*HT )
Hierarchy of 3D transformations
Projective
15dof
A
vT

t
v 
Affine
12dof
 A t
0T 1


Similarity
7dof
s R t 
 0T 1


Angles, ratios of length
The absolute conic Ω∞
Euclidean
6dof
 R t
0T 1


Volume
Intersection and tangency
Parallellism of planes,
Volume ratios, centroids,
The plane at infinity π∞
The plane at infinity
 AT
π  H π    T  T
- t A
T
A
0
 
0  0 
    π
1 0
 
1
The plane at infinity π is a fixed plane under a
projective transformation H iff H is an affinity
1.
2.
3.
4.
canonical position π∞ = (0,0,0,1)
contains directions D = (X1 , X 2 , X 3 ,0)T
two planes are parallel  line of intersection in π∞
line // line (or plane)  point of intersection in π∞
T
The absolute conic
The absolute conic Ω∞ is a (point) conic on π.
In a metric frame:
2
2
2
X1  X 2  X 3 
0
X4

or conic for directions:
(with no real points)
X1, X 2 , X 3 IX1, X 2 , X 3 T
The absolute conic Ω∞ is a fixed conic under the
projective transformation H iff H is a similarity
1. Ω∞ is only fixed as a set
2. Circle intersect Ω∞ in two circular points
3. Spheres intersect π∞ in Ω∞
The absolute dual quadric
I
  T
0
*
∞
0
0
The absolute dual quadric Ω*∞ is a fixed conic under
the projective transformation H iff H is a similarity
1. 8 dof
2. plane at infinity π∞ is the nullvector of Ω∞
3. Angles:
π T Ω* π
cos θ =
1
(π
T
1
∞ 2
Ω*∞π1 )(π T2 Ω*∞π 2 )
Camera model
Relation between pixels and rays in space
?
Pinhole camera
Pinhole camera model
T
T
( X , Y , Z )  ( fX / Z , fY / Z )
X
   fX   f
Y    
 Z    fY   
   Z  
1
 
f
X
0  
Y 

0  
Z

1 0 
1
linear projection in homogeneous coordinates!
Pinhole camera model
X
X
 fX   f
 10  0 
  
Y 
Y 




f
 fY x  PX

  01Z  0 Z 
 Z  
1  0 1 0 
  
1
1
P  diag( f , f ,1)I | 0
Principal point offset
T
T
( X , Y , Z )  ( fX / Z + px , fY / Z + py )
T
( px , py )
principal point
X
   fX  Zpx   f
 
Y  
 Z    fY  Zpx   
 
  
Z
 
1
 
f
px
py
1
X
0  
Y 

0  
Z
0 
1
Principal point offset
px
 fX  Zp x   f

 

I | 0fXcam
py
 fY  Zp x x  K


 
Z
1

 
f
K  

f
X
0  
Y 

0  
Z
0 
1
px 
p y  calibration matrix
1 
Camera rotation and translation
~
~ ~
(
Xcam = R X - C)
X
~
~  
R - RC Y  R - RC
X cam  
X
   

1 
1  Z   0
0
1
~
 
x  KR
Ccam
X
KI |I0| -X
~
x = PX
P  KR | t 
t  -RC
 
CCD camera
p x f
mx x
KK
my y p y

11
f
px 
p y 
1 
General projective camera
 x s
K  
x

~
P  KR I | C


px 
p y 
1 
11 dof (5+3+3)
non-singular
P  KR | t 
intrinsic camera parameters
extrinsic camera parameters
Radial distortion
• Due to spherical lenses (cheap)
• Model:
R
R
 x
( x, y)  (1  K1 ( x 2  y 2 )  K 2 ( x 4  y 4 )  ...) 
 y
straight lines are not straight anymore
http://foto.hut.fi/opetus/260/luennot/11/atkinson_6-11_radial_distortion_zoom_lenses.jpg
Camera model
Relation between pixels and rays in space
?
Projector model
Relation between pixels and rays in space
(dual of camera)
?
(main geometric difference is vertical principal point offset
to reduce keystone effect)
Meydenbauer camera
vertical lens shift
to allow direct
ortho-photographs
Affine cameras
Action of projective camera on points
and lines
projection of point
x = PX
forward projection of line
X(μ) = P(A+ μB) = PA + μPB = a + μb
back-projection of line
Π = PTl
 X  l PX
T
T
(lT x = 0; x = PX)
Action of projective camera on conics
and quadrics
back-projection to cone
T
Qco = P CP
x T Cx = XT PT CPX = 0
(x = PX)
projection of quadric
C  PQ P
*
* T
TQ*  lT PQ*PT l  0
(Π = PTl)
Image of absolute conic
A simple calibration device
(i)
compute H for each square
(corners @ (0,0),(1,0),(0,1),(1,1))
(ii) compute the imaged circular points H(1,±i,0)T
(iii) fit a conic to 6 circular points
(iv) compute K from w through cholesky factorization
(≈ Zhang’s calibration method)
Exercises: Camera calibration
Next class:
Single View Metrology
Antonio Criminisi