Transcript Lecture Notes

Lecture 3: Camera Rotations
and Homographies
X 
 
P Y
 
 Z 
Recap: Normalized Camera
A camera is normalized if the
units are chosen so that the
focal length is 1. Then image
coordinates (x,y) = X/Z,Y/Z
In general, mapping from world
points to image points uses
the “calibration matrix”. This
takes a 3D world point X,Y,Z
and returns its 2d
homogeneous pixel
Computer Vision, Robert
coordinates.
Pless
 x
p 
 y
(0,0,0)
Camera Center of Projection
 x   fx
  
 y   0
1   0
  
s
fy
0
x0  X

y 0  Y
1   Z





Complete projection matrix:
 x   fx
  
 y   0
1   0
  
.

.
.

.
KR
.
.Tx 

.T y 
. T z 
Computer Vision, Robert
Pless
s
fy
0
x0  .

y 0  .
1   .
.
R
.
 X grid
. T x 
  Y grid
.T y 
Z grid


.Tz 
1








 x   fx
  
 y   0
1   0
  
s
fy
0
x0  .

y 0  .
1   .
.
R
.
 X grid
. T x 
  Y grid
.T y 
Z grid


.Tz 
1








æ X grid ö
æ x ö æ p1 p2 p3 p4 öç
÷
÷çYgrid ÷
ç ÷ ç
ç y ÷ @ ç p5 p6 p7 p8 ÷ç
÷
÷÷ç Z grid ÷
ç1 ÷ çç
p11 øç
è ø è p9 p10 1
÷
1
è
ø
p1X +p2Y + p3Z + p4
x=
p9X + P10Y + Z + p11
p9Xx + P10Yx + Zx + P11x = p1X + p2Y + p3Z + p4
p1X + p2Y + p3Z + p4 + 0 p5 + 0 p6 + 0 p7 + 0 p8 - Xxp9 -Yxp10 - xP11 = Zx
p1X + p2Y + p3Z + p4 + 0 p5 + 0 p6 + 0 p7 + 0 p8 - Xxp9 -Yxp10 - xP11 = Zx
0 p1+ 0 p2 + 0 p3+ 0 p4 + Xp5 +Yp6 + Zp7 +1p8 - Xyp9 - Yyp10 - yP11 = Zy
æ p1 ö
ç
÷
ç p2 ÷
ç
÷
p3
ç
÷
ç p4 ÷
æ X Y Z 1 0 0 0 0 -Xx -Yx -x öç
÷ Zx
p5
ç
֍
÷
ç 0 0 0 0 X Y Z 1 -Xy -Yy -y ÷ç p6 ÷ = Zy
ç
֍
÷
ç
÷ç p7 ÷
è
øç
p8 ÷
ç
÷
ç p9 ÷
ç p10 ÷
ç
÷
ç p11 ÷
è
ø
 x   fx
  
 y   0
1   0
  
s
fy
0
x0  .

y 0  .
1   .
.
R
.
 X grid
. T x 
  Y grid
.T y 
Z grid


.Tz 
1








• Let’s consider a special case.
Suppose we take two pictures of the
world, from a camera at the same
location. But the camera has rotated
between the two pictures.
 x   fx
  
 y   0
1   0
  
• First image, points
in 3D, measured in  x  
camera coordinate  y   
1  
system.
  
• Second image,
camera has
 x'  
rotated.
x 0  .

y 0  .
1   .
s
fy
0
fx
s
0
fy
0
0
fx
  
 y '   0
1   0
  
s
fy
0
x0  .

y 0  .
1   .
x 0  .

y 0  .
1   .
.
R1
.
.
I
.
.
R
.
 X grid
. T x 
  Y grid
.T y 
Z grid


.Tz 
1








 X grid
. 0 
  Y grid
. 0 
Z grid


.0 
1








 X grid
. 0 
  Y grid
. 0 
Z grid


.0 
1








• …mostly on
chalkboard
• On image 1
p = KP
And image 2,
p’ = KRP
 x   fx
  
p   y   0
1   0
  
 x   fx
  
p   y   0
1   0
  
p = K((KR)-1)p’
= KR-1K-1p’
s
fy
0
s
fy
0
x0  .

y 0  .
1   .
x0  .

y 0  .
1   .
p  KRP
p  KP
p '  KRP
.
R1
.
.
R1
.
 X grid
. 0 
  Y grid
. 0 
Z grid


.0 
1

.   X grid

.   Y grid


.   Z grid














p
-1
-1
=KR K p’,
so what?
• Given a few examples of
corresponding points p,p’, we can
solve for a mapping KR-1K-1 of all
points.
Another Special Case:… Suppose the world is a
plane.
• Projection from the world to the image:
 x   fx
  
 y   0
1   0
  
s
fy
0
x0 

y 0 
1  
R
 X grid
T x 
  Y grid
T y 
Z grid


Tz 
1

Irrelevant







0
• Ignore z-coordinate (it is 0 anyway), drop the 3rd column of the
3x4 matrix, then you get a mapping between the plane and the
image which is an arbitrary 3 x 3 matrix. Given 2 images:
• p = KGP,
• p’ = KG’P, so p’ = KG’(G-1K-1)p… p’ = (some 3x3) p
Computer Vision, Robert
Pless
Homography: (x’,y’,1) ~ H (x,y,1)
• Tells you exactly where the point goes.
• Point (x,y) in one frame corresponds to point (x’,y’) in the
other frame. If we need to think about multiple points, we
may put subscripts on them.
• Being careful about the homogenous coordinate, we write:
x
 wx ' 
 
 wy ' 
 H y


1 
w 
 
Computer Vision, Robert Pless
Homography is a “simple” example of
a 3D to 2D transformation
It is also the “most complicated” linear
2D to 2D transformation.
What other 2D  2D transformations
are there?
Computer Vision, Robert
Pless
Homography is most general,
encompasses other transformations
Projective
8 dof
Affine
6 dof
Similarity
4 dof
 h11

h
 21
 h 31
 a 11

a
 21
 0
 sr11

sr
 21
 0
h12
h 22
h 32
a 12
a 22
0
sr12
sr 22
 r11 r12
Euclidean

r
r22
3 dof
 21
Computer Vision, Robert
0
0
Pless
0
h13 

h 23

h 33 
tx 

ty

1 
tx 

ty

1 
tx 

ty

1 
Views of a plane from different
viewpoints, any view of a scene
from the same viewpoint.
Images of a “far away” object under
any rotation
Camera looking at an assembly line
w/ zoom.
Camera looking at an assembly line.
Invariants…
Projective
8 dof
Affine
6 dof
Similarity
4 dof
Euclidean
3 dof
 h11

h
 21
 h 31
 a 11

a
 21
 0
 sr11

sr
 21
 0
 r11

r
 21
 0
Computer Vision, Robert
Pless
h12
h 22
h 32
a 12
a 22
0
sr12
sr 22
0
r12
r22
0
h13 

h 23

h 33 
tx 

ty

1 
tx 

ty

1 
tx 

ty

1 
Concurrency, collinearity, order of
contact (intersection, tangency,
inflection, etc.), cross ratio
Parallellism, ratio of areas, ratio of
lengths on parallel lines (e.g midpoints).
Ratios of any edge lengths, angles.
Absolute lengths, areas.
Image registration
Determining the 2d transformation that brings one
into alignment (registers it) with another.
Computer Vision, Robert
Pless
image
Image Warping
•
What kind of warps are these?
 0 . 6101

 0 . 1680

  0 . 0011
Computer Vision, Robert
Pless
 0 . 1918
0 . 6000
 0 . 0010
34 . 2411 

26 . 7034


1
How to solve for these
mappings?
Given:
Solve for:
 0 . 6101

 0 . 1680

  0 . 0011
Computer Vision, Robert
Pless
 0 . 1918
0 . 6000
 0 . 0010
34 . 2411 

26 . 7034


1
Unwrapping a matrix.
 x   wx ' 
  

H  y   wy '


1   w 
 
a

d

 g
c   x   wx ' 
    wy ' 
f  y  


 w 


1   1  

b
e
h
Write out the lines of this matrix equation.
ax  by  c  wx
'
dx  ey  f  wy
'
gx  hy  1  w
And remember which variables are unknown.
Computer Vision, Robert
Pless
Unwrapping a matrix.
a

d

 g
b
e
h
c   x   wx ' 
    wy ' 
f  y  


 w 



1 1  

ax  by  c  wx
'
i
dx  ey  f  wy
'
i
gx  hy  1  w
ax  by  c  0 d  0 e  0 f  0 g  0 h  wx i
'
0 a  0 b  0 c  dx  ey  f  0 g  0 h  wy i
'
0 a  0 b  0 c  0 d  0 e  0 f  gx  hy  1  w
ax  by  c  0 d  0 e  0 f  0 g  0 h  wx  0
'
0 a  0 b  0 c  dx  ey  f  0 g  0 h  wy  0
'
Computer Vision, Robert
Pless
0 a  0 b  0 c  0 d  0 e  0 f  gx  hy  w   1
a

d

 g
b
e
h
c   x   wx ' 
    wy ' 
f  y  




1   1   w 
ax  by  c  0 d  0 e  0 f  0 g  0 h  wx  0
'
0 a  0 b  0 c  dx  ey  f  0 g  0 h  wy  0
'
0 a  0 b  0 c  0 d  0 e  0 f  gx  hy  w   1
Looks like another matrix equation:
x

0
0

Computer Vision, Robert
Pless
y
1
0
0
0
0
0
0
0
x
y
1
0
0
0
0
0
0
0
x
y
a 
 
b
 
c 

' 
 xi   d  0 
 
'


 yi  e  0
   
 1   f    1
g 
 
h 
 
w 
a

d

 g
b
e
h
c   x   wx ' 
    wy ' 
f  y  




1   1   w 
ax  by  c  0 d  0 e  0 f  0 g  0 h  wx  0
'
0 a  0 b  0 c  dx  ey  f  0 g  0 h  wy  0
'
0 a  0 b  0 c  0 d  0 e  0 f  gx  hy  w   1
Looks like another matrix equation:
Data from
different points
Computer Vision, Robert
Pless
x

0
0

x

0

0

x
0

0

x

0

0
y
1
0
0
0
0
0
0
0
x
y
1
0
0
0
0
0
0
0
x
y
y
1
0
0
0
0
0
0
0
x
y
1
0
0
0
0
0
0
0
x
y
y
1
0
0
0
0
0
0
0
x
y
1
0
0
0
0
0
0
0
x
y
y
1
0
0
0
0
0
0
0
x
y
1
0
0
0
0
0
0
0
x
y
0  a  0 

'
 y i 0 0 0  b  0 

  

 1 0 0 0  c    1

  
'
0  xi 0 0   d  0 
'

0  yi 0 0 e  0 

  
0
 1 0 0   f    1


'

0 
0 0  xi 0  g

  
'

0 0  yi 0 h  0 

  
0 0
 1 0   w    1
'
0 0 0  xi  w 2  0 

  
'
w
3

0 0 0  yi 
 0 

0 0 0
 1   w 4    1
 xi
'
0
0
a

d

 g
b
e
h
c   x   wx ' 
    wy ' 
f  y  




1   1   w 
Challenges…
• Maybe you have error in finding corresponding points, and
want to use many many corresponding points. Then your
number of unknowns keeps growing…
• Is there a better way?
• At the end of the day, how do we compute a real coordinate
x’?
Computer Vision, Robert
Pless
The game of “finding the linear constraint…”
x’ = wx’ / w
ax  by  c  wx
gx  hy  1  w
a

d

 g
'
x 
'
ax  by  c
b
e
h
c   x   wx ' 

  
f  y    wy ' 



1   1   w 
Non-linear
gx  hy  1
x ( gx  hy  1)  ax  by  c
'
Computer Vision, Robert
Pless
Linear (in a,b,c,g,h)
x ( gx  hy  1)  ax  by  c
'
gxx  hyx  x  ax  by  c
'
'
'
gxx  hyx  ax  by  c   x
'
'
'
 ax  by  c  gxx  hyx   x
'
'
'
 ax  by  c  0 d  0 e  0 f  gxx  hyx   x
'
0 a  0 b  0 c  dx  ey  f  gxy  hyy
'
Computer Vision, Robert
Pless
'
'
 y
'
'
 ax  by  c  0 d  0 e  0 f  gxx i  hyx i   x i
'
'
'
0 a  0 b  0 c  dx  ey  f  gxy i  hyy i   y i
'
 x

0


'
 y
1
0
0
0
xx '
yx '
0
0
x
 y
1
xy '
yy '
And
just add two more rows for each corresponding point
Computer Vision, Robert
Pless
'
a

b

c

 d
 e
 
f
g

 h





   x'
   y '







 x

0


 y
1
0
0
0
xx '
yx '
0
0
x
 y
1
xy '
yy '
a

b

c

 d
 e
 
f
g

 h





   x'
   y '







Ax=b
Matlab: x = A\b
Then make your homography matrix by rearranging x into a 3 x 3 matrix
Computer
Size ofVision,
A? b?Robert
Pless