Transcript Multiple View Geometry in Computer Vision

Computing F
class 13
Multiple View Geometry
Comp 290-089
Marc Pollefeys
Multiple View Geometry course schedule
(subject to change)
Jan. 7, 9
Intro & motivation
Projective 2D Geometry
Jan. 14, 16
(no class)
Projective 2D Geometry
Jan. 21, 23
Projective 3D Geometry
(no class)
Jan. 28, 30
Parameter Estimation
Parameter Estimation
Feb. 4, 6
Algorithm Evaluation
Camera Models
Feb. 11, 13
Camera Calibration
Single View Geometry
Feb. 18, 20
Epipolar Geometry
3D reconstruction
Feb. 25, 27
Fund. Matrix Comp.
Structure Comp.
Planes & Homographies
Trifocal Tensor
Mar. 18, 20
Three View Reconstruction
Multiple View Geometry
Mar. 25, 27
MultipleView Reconstruction
Bundle adjustment
Apr. 1, 3
Auto-Calibration
Papers
Apr. 8, 10
Dynamic SfM
Papers
Apr. 15, 17
Cheirality
Papers
Apr. 22, 24
Duality
Project Demos
Mar. 4, 6
Two-view geometry
Epipolar geometry
3D reconstruction
F-matrix comp.
Structure comp.
Epipolar geometry
Underlying structure
in set of matches for
rigid scenes
πP l
T
1 1
C1
p
L2
m1
lT1 l2
M
l1 l1  e1  x1
e1
L1
mT2 F m1  0
T
e
2 x
T T T
2 l2  0
x 2 P2 P1 [e1 ]x x1  0
m2
l2
Fundamental matrix
(3x3 rank 2 matrix)
l 2  P2 T P1T l1
C2
1. Computable from corresponding
points
Canonical representation:
P  [I | 0] P' [[e'] F  e' vT | λe' ] 2. Simplifies matching
3. Allows to detect wrong matches
4. Related to calibration
The projective reconstruction theorem
If a set of point correspondences in two views determine the
fundamental matrix uniquely, then the scene and cameras
may be reconstructed from these correspondences alone,
and any two such reconstructions from these
correspondences are projectively equivalent
allows reconstruction from pair of uncalibrated images!
Objective
Given two uncalibrated images compute (PM,P‘M,{XMi})
(i.e. within similarity of original scene and cameras)
Algorithm
(i) Compute projective reconstruction (P,P‘,{Xi})
(a) Compute F from xi↔x‘i
(b) Compute P,P‘ from F
(c) Triangulate Xi from xi↔x‘i
(ii) Rectify reconstruction from projective to metric
Direct method: compute H from control points X Ei
PM  PH -1 PM  PH -1 X Mi  HX i
Stratified method:
(a) Affine reconstruction: compute p∞
I | 0

H 
 π 
(b) Metric reconstruction: compute IAC w
-1

A
0
H
 0 1

AA  M ωM
T
T

1
 HX i
Image information
provided
point correspondences
point correspondences
including vanishing points
Points correspondences
and internal camera
calibration
View relations and
projective objects
3-space
objects
reconstruction
ambiguity
projective
F
F,H∞
p∞
affine
F,H∞
w,w’
p∞
W∞
metric
Epipolar geometry: basic equation
x' T Fx  0
x' xf11  x' yf12  x' f13  y' xf21  y' yf 22  y' f 23  xf31  yf 32  f 33  0
separate known from unknown
x' x, x' y, x' , y' x, y' y, y' , x, y,1 f11, f12 , f13, f 21, f 22 , f 23, f31, f32 , f33 T  0
(data)
 x'1 x1
 
 x'n xn

x'1 y1

x 'n yn
(unknowns)
(linear)
x'1

x 'n
y '1 x1

y ' n xn
y '1 y1

y 'n yn
Af  0
y '1

y 'n
x1

xn
y1 1
  f  0
yn 1
the singularity constraint
e'T F  0
Fe  0 detF  0
rank F  2
SVD from linearly computed F matrix (rank 3)
σ1

 V T  U1σ1V1T  U 2σ 2 V2T  U 3σ 3V3T
F  U
σ2


σ
3

Compute closest rank-2 approximation
min F - F' F
σ1

F'  U 
σ 2  V T  U1σ1V1T  U 2σ 2 V2T


0


the minimum case – 7 point correspondences
 x'1 x1
 
 x'7 x7

x'1 y1

x '7 y 7
x'1

x '7
y '1 x1

y '7 x7
y '1 y1

y '7 y 7
A  U7x7 diag σ1 ,..., σ7 ,0,0V9x9
y '1

y '7
y1 1
  f  0
y7 1
x1

x7
T
 A[V8V9 ]  09x2
x i (F1  λF2 )x i  0, i  1...7
T
one parameter family of solutions
but F1+lF2 not automatically rank 2
e.g.V
T
V8  [000000010 ]T

the minimum case – impose rank 2
3
(obtain 1 or 3 solutions)
F7pts F
F1
F2
det( F1  λF2 )  a3λ 3  a2 λ 2  a1λ  a0  0
det( F1  λF2 )  det F2 det( F2-1F1  λI)  0
Compute possible l as eigenvalues of F2-1F1
(only real solutions are potential solutions)
(cubic equation)
the NOT normalized 8-point algorithm
 f11 
f 
 12 
 f13 
 x1 x1´ y1 x1´ x1´ x1 y1´ y1 y1´ y1´ x1 y1 1 

f
x x ´ y x ´ x ´ x y ´ y y ´ y ´ x
  21 
y
1
2 2
2
2 2
2 2
2
2
2
 2 2
 f   0
 






   22 

  f 23 
 xn xn ´ y n xn ´ xn ´ xn y n ´ y n y n ´ y n ´ xn y n 1  f 
 31 
~10000
~100
~10000
~100 ~10000 ~10000 ~100 ~100
1 f 
32


Orders of magnitude difference
f
 33 
!
Between column of data matrix
 least-squares yields poor results
the normalized 8-point algorithm
Transform image to ~[-1,1]x[-1,1]
(0,500)
(0,0)
(700,500)
(700,0)
 2
 700





0
2
500

 1

 1

1

(-1,1)
(1,1)
(0,0)
(-1,-1)
(1,-1)
Least squares yields good results (Hartley, PAMI´97)
algebraic minimization
possible to iteratively minimize algebraic distance
subject to det F=0 (see book if interested)
Geometric distance
Gold standard
Sampson error
Symmetric epipolar distance
Gold standard
Maximum Likelihood Estimation (= least-squares for
Gaussian noise)
2
2
ˆ
ˆ




 d x i , x i  d x' i , x'i
subject to xˆ 'T Fxˆ  0
i
Initialize: normalized 8-point, (P,P‘) from F, reconstruct Xi
Parameterize:
P  [I | 0], P'  [M | t], Xi
xˆ i  PXi , xˆ i  P' Xi
(overparametrized)
Minimize cost using Levenberg-Marquardt
(preferably sparse LM, see book)
Gold standard
Alternative, minimal parametrization (with a=1)
(note (x,y,1) and (x‘,y‘,1) are epipoles)
problems:
• a=0  pick largest of a,b,c,d to fix
• epipole at infinity
 pick largest of x,y,w and of x’,y’,w’
4x3x3=36 parametrizations!
reparametrize at every iteration, to be sure
Zhang&Loop’s approach CVIU’01
First-order geometric error (Sampson error)
 
T
e
 JJ
T 1
eTe
 JJ T
e
1
e
T
 x' F0
xi
0
e   x' T Fx  0

 
(one eq./point
JJT scalar)

JJ T  x' T F 1  x' T F 2  Fx 1  Fx 2
eTe
 JJ T
2
2
2
2
x' T Fx
 x' F  x' F  Fx   Fx 
T
2
1
T
2
2
2
1
2
2
(problem if some x is located at epipole)
advantage: no subsidiary variables required
Symmetric epipolar error
 d x' , Fx 
2
i
i

 d x i , F x' i
T

2
i


1
1


  x' Fx

2
2
2
2
 x' T F  x' T F
Fx 1  Fx 2 
1
2

T

 

Some experiments:
Some experiments:
Some experiments:
Some experiments:
Residual error:
 d x' , Fx 
2
i
i
i
(for all points!)

 d x i , F x' i
T

2
Recommendations:
1. Do not use unnormalized algorithms
2. Quick and easy to implement: 8-point normalized
3. Better: enforce rank-2 constraint during minimization
4. Best: Maximum Likelihood Estimation
(minimal parameterization, sparse implementation)
Special case:
Enforce constraints for optimal results:
Pure translation (2dof),
Planar motion (6dof),
Calibrated case (5dof)
The envelope of epipolar lines
What happens to an epipolar line if there is noise?
Monte Carlo
C  l l T  k 2l
n=10
n=15
n=25
n=50
Other entities?
Lines give no constraint for two view geometry
(but will for three and more views)
Curves and surfaces yield some constraints
related to tangency
Automatic computation of F
(i) Interest points
(ii) Putative correspondences
(iii) RANSAC
(iv) Non-linear re-estimation of F
(v) Guided matching
(repeat (iv) and (v) until stable)
Feature points
• Extract feature points to relate images
• Required properties:
• Well-defined
(i.e. neigboring points should all be different)
• Stable across views
(i.e. same 3D point should be extracted
as feature for neighboring viewpoints)
Feature points
(e.g.Harris&Stephens´88; Shi&Tomasi´94)
Find points that differ as much as possible
from all neighboring points
homogeneous
edge
SSD   M
corner
T


 xI 
M
 I  xI yI wx, y dxdy
W  y 
 
M should have large eigenvalues

Feature = local maxima (subpixel) of F(l1, l 2)
Feature points
Select strongest features (e.g. 1000/image)
Feature matching
Evaluate NCC for all features with
similar coordinates
e.g. x´, y´ x  10w , x  10w  y  10h , y  10h 
Keep mutual best matches
Still many wrong matches!
?
Feature example
3
2
4
1
5
0.96
-0.40
-0.16
-0.39
0.19
-0.05
0.75
-0.47
0.51
0.72
3
2
4
1
-0.18
-0.39
0.73
0.15
-0.75
-0.27
0.49
0.16
0.79
0.21
0.08
0.50
-0.45
0.28
0.99
Gives satisfying results
for small image motions
5
Wide-baseline matching…
• Requirement to cope with larger
variations between images
•
•
•
•

Translation, rotation, scaling
geometric
transformations
Foreshortening
photometric
Non-diffuse reflections
changes
Illumination

Wide-baseline matching…
(Tuytelaars and Van Gool BMVC 2000)
Wide baseline matching for two different
region types
RANSAC
Step 1. Extract features
Step 2. Compute a set of potential matches
Step 3. do
Step 3.1 select minimal sample (i.e. 7 matches)
Step 3.2 compute solution(s) for F
Step 3.3 determine inliers (verify hypothesis)

(generate
hypothesis)
until (#inliers,#samples)<95%
Step 4. Compute F based on all inliers
Step 5. Look for additional matches
Step 6. Refine F based on all correct matches
  1  (1 


# inliers 7 # samples
)
# matches
#inliers
90%
80%
70%
60%
50%
#samples
5
13
35
106
382
Finding more matches
restrict search range to neighborhood of epipolar line
(1.5 pixels)
relax disparity restriction (along epipolar line)
Degenerate cases:
• Degenerate cases
• Planar scene
• Pure rotation
• No unique solution
• Remaining DOF filled by noise
• Use simpler model (e.g. homography)
• Model selection
(Torr et al., ICCV´98, Kanatani, Akaike)
• Compare H and F according to expected residual error
(compensate for model complexity)
More problems:
• Absence of sufficient features (no texture)
• Repeated structure ambiguity
• Robust matcher also finds
support for wrong hypothesis
• solution: detect repetition
(Schaffalitzky and Zisserman,
BMVC‘98)
two-view geometry
geometric relations between two views is fully
described by recovered 3x3 matrix F
Next class:
image pair rectification
reconstructing points and lines