Transcript Multiple View Geometry in Computer Vision

Self-calibration and
multi-view geometry
Class 11
Read Chapter 6 and 3.2
Geometric Computer Vision course schedule
(tentative)
Lecture
Exercise
Sept 16
Introduction
-
Sept 23
Geometry & Camera model
Camera calibration
Sept 30
Single View Metrology
Measuring in images
(Changchang Wu)
Oct. 7
Feature Tracking/Matching
Correspondence computation
Oct. 14
Epipolar Geometry
F-matrix computation
Oct. 21
Shape-from-Silhouettes
Visual-hull computation
Oct. 28
Stereo matching
papers
Nov. 4
Stereo matching (continued)
Project proposals
Nov. 11
Structured light and active range sensing
Papers
Nov. 18
Structure from motion
Papers
Nov. 25
Multi-view geometry and self-calibration
Papers
Dec. 2
3D modeling, registration and
range/depth fusion (Christopher Zach?)
Papers
Dec. 9
Shape-from-X and image-based rendering
Papers
Dec. 16
Final project presentations
Final project presentations
Self-calibration
•
•
•
•
Introduction
Self-calibration
Dual Absolute Quadric
Critical Motion Sequences
Motivation
• Avoid explicit calibration procedure
• Complex procedure
• Need for calibration object
• Need to maintain calibration
Motivation
• Allow flexible acquisition
• No prior calibration necessary
• Possibility to vary intrinsics
• Use archive footage
Projective ambiguity
Reconstruction from uncalibrated images
 projective ambiguity on reconstruction
m  P M  (PT 1)(T M)  P´M´
Stratification of geometry
Projective
15 DOF
Affine
Metric
7 DOF
12 DOF
plane at infinity absolute conic
parallelism
angles, rel.dist.
More general
More structure
Constraints ?
Scene constraints
• Parallellism, vanishing points, horizon, ...
• Distances, positions, angles, ...
Unknown scene  no constraints
Camera extrinsics constraints
–Pose, orientation, ...
Unknown camera motion  no constraints
Camera intrinsics constraints
–Focal length, principal point, aspect ratio & skew
Perspective camera model too general
 some constraints
Euclidean projection matrix
Factorization of Euclidean projection matrix
P  K R T
 R T t
fx

Intrinsics: K 


Extrinsics:
R, t 
s
fy
ux 
uy 
1 
(camera geometry)
(camera motion)
Note: every projection matrix can be factorized,
but only meaningful for euclidean projection matrices
Constraints on intrinsic
parameters
 fx
K  

s
fy
ux 
u y 
1 
Constant
e.g. fixed camera:
Known
K1  K2  
e.g. rectangular pixels: s  0
square pixels: fx  fy , s  0
w h 
principal point known: u x , u y    , 
 2 2
Self-calibration
Upgrade from projective structure to metric
structure using constraints on intrinsic
camera parameters
• Constant intrinsics
(Faugeras et al. ECCV´92, Hartley´93,
Triggs´97, Pollefeys et al. PAMI´99, ...)
• Some known intrinsics, others varying
(Heyden&Astrom CVPR´97, Pollefeys et al. ICCV´98,...)
• Constraints on intrincs and restricted motion
(e.g. pure translation, pure rotation, planar motion)
(Moons et al.´94, Hartley ´94, Armstrong ECCV´96, ...)
A counting argument
• To go from projective (15DOF) to metric
(7DOF) at least 8 constraints are needed
• Minimal sequence length should satisfy
m  # known  m 1 # fixed   8
• Independent of algorithm
• Assumes general motion (i.e. not critical)
Outline
•
•
•
•
Introduction
Self-calibration
Dual Absolute Quadric
Critical Motion Sequences
The Dual Absolute Quadric
I
  T
0
*

0
0
The absolute dual quadric Ω*∞ is a fixed conic under
the projective transformation H iff H is a similarity
1. 8 dof
2. plane at infinity π∞ is the nullvector of Ω∞
3. Angles:
π T * π
cos  

1
2
π  π π  π 
T
1
*

1
T
2
*

2
Absolute Dual Quadric and
Self-calibration
Eliminate extrinsics from equation
Equivalent to projection of Dual Abs.Quadric
Dual Abs.Quadric also exists in projective world
KK T  PΩ*PT  (PT 1 )( TΩ*TT )( T TPT )
 P´Ω´* P´T
Transforming world so that Ω´  Ω
reduces ambiguity to similarity
*
*
Absolute Dual Quadric and
Self-calibration
Projection equation:

i

ω  PiΩ P  K iK
T
i
Translate constraints on K
through projection equation
to constraints on *
T
i
*
*
Absolute conic = calibration object which is
always present but can only be observed
through constraints on the intrinsics
Constraints on *
 f x2  s 2  cx2

ω*   sf y  cx c y

cx

condition
sf y  cx c y
f y2  c 2y
constraint
cy
type
cx 

cy 
1 
#constraints
Zero skew
*
*
ω12
ω*33  ω13
ω*23
quadratic
m
Principal point
*
ω13
 ω*23  0
linear
2m
linear
m
Zero skew (& p.p.)
Fixed aspect ratio
(& p.p.& Skew)
ω 0
*
ω ω'*22  ω*22ω'11
*
12
*
11
quadratic
m-1
Known aspect ratio
(& p.p.& Skew)
*
ω11
 ω*22
linear
m
Focal length
(& p.p. & Skew)
*
ω*33  ω11
linear
m
Linear algorithm
(Pollefeys et al.,ICCV´98/IJCV´99)
Assume everything known, except focal length
 fˆ

*
ω  0
0

2
0
fˆ 2
0
0

0  P*PT
1
PΩ P   PΩ P 
PΩ P   0
PΩ P   0
PΩ P   0


T
11
T
22

T
12

T
13

T
23
Yields 4 constraint per image
Note that rank-3 constraint is not enforced
0
Linear algorithm revisited
(Pollefeys et al., ECCV‘02)
Weighted linear equations
1
 T
 T
2
ˆ



P
Ω
P

P
Ω
P 22  0
f

0 0
11
0 .2
1


 T
T
2
ˆ

P
Ω
P 12  0
KK   0 f
0
0.01
1
 T
0


P
Ω
P 13  0
0
1
0 .1


1
 T

P
Ω
P 23  0
0 .1
1
9
1
9
PΩ P   PΩ P 
PΩ P   PΩ P 


T
11


T
fy
c x  0  0.1
c y  0  0.1
0
33  0
33
22
assumptions
log( fˆˆ )  log(1)  log(3)
f
log( ˆx )  log(1)  log(1.1)
T
s0
T
Is Dual Linear Self-Calibration
Artificially Ambiguous?
(Gurdjos et al., ICCV‘09)
Projective to metric
Compute T from
~
~  I 0
* T
-1~ - T
*
I  TΩT or T I T  Ω with I   T 
0 0
using eigenvalue decomposition of Ω*
and then obtain metric reconstruction as
-1
PT and TM
Alternatives:
(Dual) image of absolute conic
• Equivalent to Absolute Dual Quadric
*
* T
ω  PΩP
ω*  Hω*HT
(H  H  ea)
• Practical when H can be computed first
• Pure rotation (Hartley’94, Agapito et al.’98,’99)
• Vanishing points, pure translations, modulus
constraint, …
Note that in the absence of skew the IAC
can be more practical than the DIAC!
 f x2  cx2

*
ω   c x c y
 cx
 f y2
1 
ω  2 2  0
fx f y
 f y2cx
cx c y
f y2  c y2
cy
0
f x2
 f x2c y
cx 

cy 
1


 f y2cx

 f x2c y

f x2 f y2  f y2cx2  f x2c y2 
Kruppa equations
e' ω e'
*

T

 e' Hω H e'  Fω*FT
*

T

T

Limit equations to epipolar geometry
Only 2 independent equations per pair
But independent of plane at infinity
Refinement
• Metric bundle adjustment
Enforce constraints or priors
on intrinsics during minimization
(this is „self-calibration“ for photogrammetrist)
Outline
•
•
•
•
Introduction
Self-calibration
Dual Absolute Quadric
Critical Motion Sequences
Critical motion sequences
(Sturm, CVPR´97, Kahl, ICCV´99, Pollefeys,PhD´99)
• Self-calibration depends on camera motion
• Motion sequence is not always general enough
• Critical Motion Sequences have more than one
potential absolute conic satisfying all
constraints
• Possible to derive classification of CMS
Critical motion sequences:
constant intrinsic parameters
Most important cases for constant intrinsics
Critical motion type
ambiguity
pure translation
affine transformation (5DOF)
pure rotation
arbitrary position for  (3DOF)
orbital motion
proj.distortion along rot. axis (2DOF)
planar motion
scaling axis  plane (1DOF)
Note relation between critical motion sequences and
restricted motion algorithms
Critical motion sequences:
varying focal length
Most important cases for varying focal length
(other parameters known)
Critical motion type
ambiguity
pure rotation
arbitrary position for  (3DOF)
forward motion
proj.distortion along opt. axis (2DOF)
translation and
rot. about opt. axis
hyperbolic and/or
elliptic motion
scaling optical axis (1DOF)
one extra solution
Critical motion sequences:
algorithm dependent
Additional critical motion sequences can exist
for some specific algorithms
• when not all constraints are enforced
(e.g. not imposing rank 3 constraint)
• Kruppa equations/linear algorithm: fixating a
point
Some spheres also project to
circles located in the image and
hence satisfy all the linear/kruppa
self-calibration constraints
Non-ambiguous new views for CMS
(Pollefeys,ICCV´01)
• restrict motion of virtual camera to CMS
• use (wrong) computed camera parameters
Multi-view geometry
Backprojection
• Represent point as intersection of row and column
• Condition for solution?
Useful presentation for deriving and understanding multiple view geometry
(notice 3D planes are linear in 2D point coordinates)
Multi-view geometry
(intersection constraint)
(multi-linearity of determinants)
(= epipolar constraint!)
(counting argument: 11x2-15=7)
Multi-view geometry
(multi-linearity of determinants)
(3x3x3=27 coefficients)
(= trifocal constraint!)
(counting argument: 11x3-15=18)
Multi-view geometry
(multi-linearity of determinants)
(3x3x3x3=81 coefficients)
(= quadrifocal constraint!)
(counting argument: 11x4-15=29)
from perspective to omnidirectional cameras
3 constraints allow to reconstruct 3D point
perspective camera
(2 constraints / feature)
more constraints also tell something
about cameras
l=(y,-x)
(0,0)
(x,y)
radial camera (uncalibrated) multilinear constraints known as epipolar,
trifocal and quadrifocal constraints
(1 constraints / feature)
37
Quadrifocal constraint
38
(x,y)
Radial quadrifocal tensor
• Linearly compute radial quadrifocal tensor Qijkl
from 15 pts in 4 views
(2x2x2x2 tensor)
• Reconstruct 3D scene and use it for calibration
Not easy for real data, hard to avoid degenerate
cases (e.g. 3 optical axes intersect in single point).
However, degenerate case leads to simpler 3 view
algorithm for pure rotation
• Radial trifocal tensor Tijk from 7 points in 3 views
(2x2x2 tensor)
• Reconstruct 2D panorama and use it for calibration
39
Dealing with Wide FOV Camera
(Thirthala and Pollefeys CVPR05)
• Two-step linear approach to compute radial
distortion
• Estimates distortion polynomial of arbitrary
degree
estimated distortion
(4-8 coefficients)
undistorted image
40
Dealing with Wide FOV Camera
(Thirthala and Pollefeys CVPR05)
• Two-step linear approach to compute radial
distortion
• Estimates distortion polynomial of arbitrary
degree
unfolded cubemap
estimated distortion
(4-8 coefficients)
41
Non-parametric distortion calibration
(Thirthala and Pollefeys, ICCV’05)
angle
• Models fish-eye lenses, cata-dioptric
systems, etc.
normalized radius
42
Non-parametric distortion calibration
(Thirthala and Pollefeys, ICCV’05)
• Models fish-eye lenses, cata-dioptric
systems, etc.
• results
angle
90o
normalized radius
43
Synthetic quadrifocal tensor
example
•
•
•
•
44
Perspective
Fish-eye
Spherical mirror
Hyperbolic mirror
Perspective
45
Fish-eye
Spherical mirror
46
Hyperbolic mirror
Next class: 3D reconstruction
Guest lecturer: Dr. Christopher Zach