Transcript presentation.as.pres..

Metric Self Calibration From
Screw-Transform Manifolds
Russell Manning and Charles Dyer
University of Wisconsin -- Madison
Goal
• Find internal camera calibration
• Only input is images (views)
• Camera…
– has fixed internal parameters
– undergoes general motion
Input
assorted views
Output
• Internal calibration
• (Optional) scene reconstruction
reconstruction
Quick Overview
F2
F1
Find pairwise fundamental matrices Fi
Don’t need an initial projective reconstruction
Just need fundamental matrices, which are derived
from view pairs
So related to methods based on the Kruppa constraints
rather than stratified calibration methods like the
modulus constraint
Each fundamental matrix induces
a curved surface (manifold) in R5
every manifold point is a legal internal calibration
F1
There is a unique mutual intersection
point to all these manifolds
intersection point gives calibration
F2
F1
F4
F3
Details
• What is a manifold?
• What is a screw transform?
• Screw-transform manifolds and their
coordinate system
• Finding intersection point: voting algorithm
Manifold
Surface in Rn that has a coordinate system
-2
-1
6
7
5
0
8
4
1
2
3
Manifold
grid paper example
kappa
theta
Screw Transformation
A
B
A
A
B
A
B
A
B
A
B
A
B
A
B
B
Screw Transformation
screw transformation
camera A
the scene
camera B (=camera A in arbitrary new position)
Equivalent to Rising Turntable
rising turntable: just like a normal turntable
except can also translate (rise) parallel to its
rotation axis
*turntable is viewed by a single camera, fixed in position
scene
camera A
rising turntable
Equivalent to Rising Turntable
rising turntable: just like a normal turntable
except can also translate (rise) parallel to its
rotation axis
*turntable is viewed by a single camera, fixed in position
Equivalent to Rising Turntable
rising turntable: just like a normal turntable
except can also translate (rise) parallel to its
rotation axis
*turntable is viewed by a single camera, fixed in position
Equivalent to Rising Turntable
rising turntable: just like a normal turntable
except can also translate (rise) parallel to its
rotation axis
*turntable is viewed by a single camera, fixed in position
Equivalent to Rising Turntable
rising turntable: just like a normal turntable
except can also translate (rise) parallel to its
rotation axis
*turntable is viewed by a single camera, fixed in position
translated by gamma
camera B
rotated by theta
The Two Camera Matrices
camera A...
is located at position (1,0,0)
is tilted by a rotation matrix R
and has internal calibration K
The Two Camera Matrices
camera B...
is exactly like camera A, but...
the world has first undergone a
screw transform S
The Two Camera Matrices
The Two Camera Matrices
translation by gamma parallel to z-axis
rotation by theta around the z-axis (== screw axis)
The Two Camera Matrices
so we can refer to
the column vectors
of the camera
matrices
Fundamental Matrix
screw
rotation
angle
screw translation
H = [ h 1 h2 h3 ] = K R
columns from
camera matrices
Fundamental Matrix
through this representation, we will be able to find K from F by picking
three real numbers: theta, gamma, and kappa !
these real numbers parameterize a “screw-transform manifold”
Screw Transform Manifold
You give me 3 real numbers (kappa, theta, gamma)
and a fundamental matrix
I run the numbers through an algorithm (given in the paper)
to get a legal internal calibration matrix (K)
Set of legal K’s is the screw-transform manifold
(kappa, theta, gamma) are the coordinate system
Where does manifold live?
Screw Transform Manifold
Manifold lives in R5 because K is 3x3 upper-triangular
matrix:
x x x
0 x x
0 0 x
only 5 degrees of freedom
since don’t care about scale
(e.g., make norm 1)
Mutual Intersection Point
Manifold corresponds to fundamental matrix
Manifold is a set of points
Each point is a legal internal calibration matrix
The mutual intersection point of all the manifolds is a
legal internal calibration matrix for every fundamental
matrix
Voting Algorithm
Basic Idea
“sketch” the manifolds within
initial search region
find approximate region
where manifolds seem to
intersect
“zoom in”: repeat the
process starting in the
new, smaller search region
f(t) is shown as a dotted line because we don’t know it’s shape initially
Manifold defined
by function f(t) with
one underlying
real-valued
parameter t
In the limited
search space
shown at left, t
might vary between
0 and 1
Pick t at random,
say t=0.6, and find
f(t)
Continue picking
random values for t
in range [0,1] to
sketch manifold f(t)
To utilize the
randomly-selected
points (“votes”), first
drop a voxel grid
over the search
space
Mark voxels that
receive at least one
vote
Sketch the second
manifold in the
same way
lighter color means higher number of votes
Now consider
tabulating the
votes
For each voxel,
keep track of how
many manifolds
have voted for that
voxel
When some region
receives enough
votes, zoom in on
that region
Now “zoom in” -- start over using the smaller search region
as the original search region
“binary search”
Repeat the sketching and voting procedure.
A winner
emerges.
Determine the
new “zoom in”
region
Voxelize the new search space
No consensus!
Properties of Voting Algorithm
Benefits from using lots of fundamental matrices
Noisy manifolds will contribute votes in empty regions
and thus be ignored
Zoom-in step eliminates bad manifolds
Determines a small region where K is rather than an
exact point
Quick convergence at end when manifolds become
linear
Properties of Voting Algorithm
General technique -- use whenever need to intersect
manifolds
begin
end
(1) What is self calibration?
(2) What is a screw transform?
(3) Fundamental matrix + choice of three screw
transform parameters  internal calibration
matrix.
(4) Internal calibration matrix K lives in a 5D
space (termed “K-space”).
(5) Each fundamental matrix yields a 3D
manifold in K-space, termed “screw-transform
manifold.”
(6) Intersection of 3 or more manifolds gives K
matrix common to all view pairs (i.e., finds
internal calibration).
Map of the Talk
(1) A sample application of self calibration
(2) The “K” matrix (internal calibration matrix).
(3) Relationship between K and “screw transform”
that underlies pairwise views.
(4) What is a manifold?
(5) The screw-transform manifold.
(6) Intersecting screw-transform manifolds.
(7) Algorithm for self calibration.
(8) Voting algorithm to find intersection point.
(pictures of me taking photos of scene…leading
to self calibration and scene reconstruction)
Map of the Talk
(1) A sample application of self calibration
(2) The “K” matrix (internal calibration matrix).
(3) Relationship between K and “screw transform”
that underlies pairwise views.
(4) What is a manifold?
(5) The screw-transform manifold.
(6) Intersecting screw-transform manifolds.
(7) Algorithm for self calibration.
(8) Voting algorithm to find intersection point.
Camera Equation
Consider the equation for how points in 3D
get projected onto image plane of camera…
Camera Equation
p == K [ R | t ] P
position P in space…
…gets translated by t and rotated by R…
…then transformed to camera’s internal
coordinate system by K
p is projected position on image plane
(in homogeneous coords)
Self Calibration
p == K [ R | t ] P
Goal of self calibration is to find K up to a scale factor.
K is an upper triangular, 3x3 matrix:
x x x
0 x x
0 0 x
only 5 degrees of freedom
since don’t care about scale
(e.g., make norm 1)
Map of the Talk
(1) A sample application of self calibration
(2) The “K” matrix (internal calibration matrix).
(3) Relationship between K and “screw transform”
that underlies pairwise views.
(4) What is a manifold?
(5) The screw-transform manifold.
(6) Intersecting screw-transform manifolds.
(7) Algorithm for self calibration.
(8) Voting algorithm to find intersection point.
Screw Transformation
arbitrary movement in space
Screw Transformation
screw axis
Screw Transformation
screw axis
Screw Transformation
screw axis
Screw Transformation
screw rotation
screw axis
Screw Transformation
screw rotation
screw axis
Screw Transformation
screw rotation
screw axis
Screw Transformation
screw rotation
screw axis
Screw Transformation
screw rotation
screw axis
Screw Transformation
screw rotation
screw axis
Screw Transformation
screw transformation
camera A
the scene
camera B (=camera A in arbitrary new position)
Equivalent to Rising Turntable
rising turntable: just like a normal turntable
except can also translate (rise) parallel to its
rotation axis
*is viewed by a single camera, fixed in position
scene
camera A
rising turntable
Equivalent to Rising Turntable
rising turntable: just like a normal turntable
except can also translate (rise) parallel to its
rotation axis
*is viewed by a single camera, fixed in position
Equivalent to Rising Turntable
rising turntable: just like a normal turntable
except can also translate (rise) parallel to its
rotation axis
*is viewed by a single camera, fixed in position
Equivalent to Rising Turntable
rising turntable: just like a normal turntable
except can also translate (rise) parallel to its
rotation axis
*is viewed by a single camera, fixed in position
Equivalent to Rising Turntable
rising turntable: just like a normal turntable
except can also translate (rise) parallel to its
rotation axis
*is viewed by a single camera, fixed in position
translated by gamma
camera B
rotated by theta
The Two Camera Matrices
camera A...
is located at position (1,0,0)
is tilted by a rotation matrix R
and has internal calibration K
The Two Camera Matrices
camera B...
is exactly like camera A, but...
the world has first undergone a
screw transform S
The Two Camera Matrices
The Two Camera Matrices
translation by gamma parallel to z-axis
rotation by theta around the z-axis (== screw axis)
The Two Camera Matrices
so we can refer to
the column vectors
of the camera
matrices
Fundamental Matrix
screw
rotation
angle
screw translation
H = [ h 1 h2 h3 ] = K R
columns from
camera matrices
Fundamental Matrix
through this representation, we will be able to find K from F by picking
three real numbers: theta, gamma, and kappa !
these real numbers parameterize a “screw-transform manifold”
Finding K from F
F and kappa  h3
h3 and theta  h1 and h2 up to a scale factor
gamma  h1 and h2
h1, h2, and h3  K R = [ h1 h2 h3 ]
K R  K since (K R) (K R)T = K R RT KT = K KT,
which gives K by Cholesky factorization
Map of the Talk
(1) A sample application of self calibration
(2) The “K” matrix (internal calibration matrix).
(3) Relationship between K and “screw transform”
that underlies pairwise views.
(4) What is a manifold?
(5) The screw-transform manifold.
(6) Intersecting screw-transform manifolds.
(7) Algorithm for self calibration.
(8) Voting algorithm to find intersection point.
Manifolds
Example 1: A one-dimensional manifold in 2D space
Manifolds
Part of a 2D space
A 1D manifold in the
2D space
Another 1D manifold
Manifold can be seen as
a function f(t) that maps
real numbers R into R2
t “parameterizes” the
manifold
Manifolds
Manifolds are locally flat
That is, locally a
manifold behaves like
its underlying
parameter space
(which is Rn)
In this case, each manifold is
locally like a line
Manifolds
Example 2: A two-dimensional manifold in 3D space
Map of the Talk
(1) A sample application of self calibration
(2) The “K” matrix (internal calibration matrix).
(3) Relationship between K and “screw transform”
that underlies pairwise views.
(4) What is a manifold?
(5) The screw-transform manifold.
(6) Intersecting screw-transform manifolds.
(7) Algorithm for self calibration.
(8) Voting algorithm to find intersection point.
Screw-Transform Manifold
Earlier it was shown how the fundamental matrix
between two views relates to the parameters of the
underlying screw transformation.
Using this formulation, it can be shown that once a
fundamental matrix between two views is known, then
internal calibration K is known up to three parameters (i.e.,
K has three degrees of freedom).
Screw-Transform Manifold
The set of possible K matrices that are consistent with a
given fundamental matrix is a 3D manifold in (5D) K-space.
K-space == space of possible K matrices (of norm 1)
The three underlying parameters:
theta = angle of rotation for the screw transform
gamma = translation parallel to the screw axis
kappa = determines location of screw axis
Map of the Talk
(1) A sample application of self calibration
(2) The “K” matrix (internal calibration matrix).
(3) Relationship between K and “screw transform”
that underlies pairwise views.
(4) What is a manifold?
(5) The screw-transform manifold.
(6) Intersecting screw-transform manifolds.
(7) Algorithm for self calibration.
(8) Voting algorithm to find intersection point.
Mutual Intersection Point
Every point on screw-transform manifold i represents a
K matrix that is consistent with fundamental matrix i.
Thus a point that lies in the intersection of all the screw
transform manifolds is a legal K that is consistent with
every fundamental matrix.
At least one such point exists (since the camera has an
internal calibration matrix).
Others have shown that, under the right conditions, there
is a unique intersection point.
double intersection
all manifolds
intersect here
for self calibration,
we will want this
point of mutual
intersection
Map of the Talk
(1) A sample application of self calibration
(2) The “K” matrix (internal calibration matrix).
(3) Relationship between K and “screw transform”
that underlies pairwise views.
(4) What is a manifold?
(5) The screw-transform manifold.
(6) Intersecting screw-transform manifolds.
(7) Algorithm for self calibration.
(8) Voting algorithm to find intersection point.
(1) Take several pictures of a scene.
(2) Find all pairwise fundamental matrices.
(3) Each fundamental matrix leads to a screwtransform manifold.
F3
F2
F1
(4) Find K by intersecting the manifolds.
Map of the Talk
(1) A sample application of self calibration
(2) The “K” matrix (internal calibration matrix).
(3) Relationship between K and “screw transform”
that underlies pairwise views.
(4) What is a manifold?
(5) The screw-transform manifold.
(6) Intersecting screw-transform manifolds.
(7) Algorithm for self calibration.
(8) Voting algorithm to find intersection point.
Basic Idea
“sketch” the manifolds within
initial search region
find approximate region
where manifolds seem to
intersect
“zoom in”: repeat the
process starting in the
new, smaller search region
f(t) is shown as a dotted line because we don’t know it’s shape initially
Manifold defined
by function f(t) with
one underlying
real-valued
parameter t
In the limited
search space
shown at left, t
might vary between
0 and 1
Pick t at random,
say t=0.6, and find
f(t)
Continue picking
random values for t
in range [0,1] to
sketch manifold f(t)
To utilize the
randomly-selected
points (“votes”), first
drop a voxel grid
over the search
space
Mark voxels that
receive at least one
vote
Sketch the second
manifold in the
same way
lighter color means higher number of votes
Now consider
tabulating the
votes
For each voxel,
keep track of how
many manifolds
have voted for that
voxel
When some region
receives enough
votes, zoom in on
that region
Now “zoom in” -- start over using the smaller search region
as the original search region
“binary search”
Repeat the sketching and voting procedure.
A winner
emerges.
Determine the
new “zoom in”
region
Voxelize the new search space
No consensus!
Experimental Results
Performance
Concluding Remarks
Last Slide
Self Calibration
Voting Algorithm
Tell me again why we want to
find the mutual intersection
point?
Every point in K-space corresponds to a possible K
matrix (where K is the internal calibration that we
wish to find).
Each fundamental matrix F induces a “screw-transform
manifold” in K-space consisting of all possible K
matrices that are consistent with F.
Hence the mutual intersection point is consistent with all
the fundamental matrices and must represent the
internal calibration of the camera.
3 or more fundamental matrices are necessary for
the manifold intersection to be a single point (or
finite set of points).
More on K-space
K-space is 5D; each STM is a 3D manifold in this space.
Screw Transformation
(two photos of me moving
camera to different
positions)
Screw Transformation