Transcript Multiple View Geometry Marc Pollefeys University of North Carolina at Chapel Hill

Multiple View Geometry
Marc Pollefeys
University of North Carolina at Chapel Hill
Modified by Philippos Mordohai
Tutorial outline
• Introduction
• Visual 3D modeling
– Acquisition of camera motion
– Acquisition of scene structure
– Constructing visual models
• Examples and applications
2
Visual 3D models from images
“Sampling” the real world
• Visualization
– Virtual/augmented/mixed reality, tele-presence,
medicine, simulation, e-commerce, etc.
• Metrology
– Cultural heritage and archaeology, geology, forensics,
robot navigation, etc.
Convergence of computer vision, graphics and photogrammetry
3
Visual 3D models from video
What can be achieved?
unknown scene
unknown
camera
unknown
motion
Scene
camera
(static)
automatic
modelling
Visual model
4
Example: DV video  3D model
accuracy ~1/500 from DV video (i.e. 140kb jpegs 576x720)
5
(Pollefeys et al. ICCV’98;
…
Pollefeys et al.’IJCV04)
6
Outline
• Introduction
• Image formation
• Relating multiple views
7
Perspective projection
Perspective projection
Linear equations (in homogeneous coordinates)
8
Homogeneous coordinates
• 2-D points represented as 3-D vectors
• 3-D points represented as 4-D vectors
• Equality defined up to scale
– (X Y Z 1)T ~ (WX WY WZ W)T
(x y 1)T
(X Y Z 1)T
• Useful for perspective projection  makes equations
linear
C
9
m
M1
M2
Pinhole camera model
T
( X , Y , Z )  ( fX / Z , fY / Z )
X
   fX   f
Y    
 Z    fY   
   Z  
1
 
10
f
T
X
0  
Y 

0  
Z
1 0 
1
linear projection in homogeneous coordinates!
The pinhole camera
fX
x
Z
fY
y
Z
11
Effects of perspective projection
• Colinearity is invariant
• Parallelism is not preserved
12
Principal point offset
( X , Y , Z )  ( fX / Z  px , fY / Z  p y )
T
( p x , p y )T
13
T
principal point
X
   fX  Zp x   f
 
Y  
 Z    fY  Zp x   
 
  
Z
 
1
 
f
px
py
1
X
0  
Y 

0  
Z
0 
1
Principal point offset
px
 fX  Zp x   f

 
py
 fY  Zp x x  KI | 0fXcam

 
Z
1

 
14
f
K  

f
X
0  
Y 

0  
Z
0 
1
px 
p y  calibration matrix
1 
Camera rotation and translation

~
~ ~
Xcam  R X - C
X
~  
~
 R  R C   Y   R  RC 
X cam  
   
X
1  Z   0
1 
0
1
~
 
x  KR
KI |I0|XCcamX
~
x  PX
P  KR | t 
t  RC

15


Intrinsic parameters
 fx
K  

s
fy
cx 
c y 
1 
•
Camera deviates from pinhole
•
Usually:
s: skew
fx ≠ fy: different magnification in x
and y
(cx cy): optical axis does not pierce
image plane exactly at the
center
rectangular pixels:
square pixels:
principal point known:
16
s0
fx  f y
or
af
K  

 γf

K

w h
cx , c y    , 
 2 2
f cos( s) uo 
f
vo 
1 
sf
f
x0 

y0 
1 
Extrinsic parameters
Scene motion
Camera motion
17
R (3x3) t (3x1) 
M

1 
 0(1x3)
R T (3x3) - (R T t) 3x1 
M'  

1
 0(1x3)

Projection matrix
• Includes coordinate transformation and
camera intrinsic parameters
 x   p11 p12
  y    p21 p22
 1  p31 p32
p13
p23
p33
m ~ PM
18
X 
p14   
Y


p24 
Z 
p34   
1
Projection matrix
• Mapping from 2-D to 3-D is a function of
internal and external parameters
X 
 x   f x s cx 
Y 




T
T 

λ y   0 f y cy  R | R t
Z 
 1   0 0 1 
 
1
T
T
λx  K R |  R t X


19
λx  PX


Radial distortion
•In reality, straight lines are not preserved due
to lens distortion
•Estimate polynomial model to correct it
20
Radial distortion
21
Outline
• Introduction
• Image formation
• Relating multiple views
22
3D from images
L2
C1
m1
M?
M
L1
Triangulation
m2
C2
23
l2
- calibration
- correspondences
Comparing image regions
Compare intensities pixel-by-pixel
I(x,y)
I´(x,y)
(Dis)similarity measures
Normalized Cross Correlation
24
Sum of Square Differences
Structure and motion recovery
Feature extraction
Feature matching
Multi-view relation
Structure and
motion recovery
Self-calibration
25
Other cues for depth and geometry
Shading
Texture
26
Shadows, symmetry, silhouette
Focus
Epipolar geometry
Underlying structure in set of matches for rigid scenes
1.
P
L2
C1
m1
4.
M
L1
l1
lT1 l2
e1
m F m1  0
T
2
Fundamental matrix
(3x3 rank 2 matrix)
27
e2
m2
C2
2.
3.
Computable from
corresponding points
Simplifies matching
Allows to detect wrong
matches
Related to calibration
l2
The epipoles
The epipole is the projection of the focal point
of one camera in another image.
e21
e12
C2
C1
Image 1
P1  A1 | b1 
P2  A2 | b2 
28
Image 2
 A11b1 
P1C1  A1 | b1 
0
 1 
 A11b1 
1
P2C1  A2 | b2 

b

A
A
 2
2 1 b1
 1 
Two view geometry computation:
linear algorithm
For every match (m,m´):
m' T Fm  0
x' xf11  x' yf12  x' f13  y' xf21  y' yf 22  y' f 23  xf31  yf 32  f 33  0
separate known from unknown
x' x, x' y, x' , y' x, y' y, y' , x, y,1 f11, f12 , f13 , f 21, f 22 , f 23 , f31, f32 , f33 T  0
(data)
 x'1 x1
 
 x ' n xn

29
x'1 y1

x 'n y n
(unknowns)
(linear)
x'1

x 'n
y '1 x1

y ' n xn
y '1 y1

y 'n yn
Af  0
y '1

y 'n
x1

xn
y1 1
  f  0
yn 1
Benefits from having F
• Given a pixel in one image, the corresponding
pixel has to lie on epipolar line
• Search space reduced from 2-D to 1-D
30
Two view geometry computation:
finding more matches
31
Difficulties in finding corresponding pixels
32
Matching difficulties
•
•
•
•
Occlusion
Absence of sufficient features (no texture)
Smoothness vs. sensitivity
Double nail illusion
33
Two view geometry computation:
more problems
• Repeated structure ambiguity
• Robust matcher also finds
support for wrong hypothesis
• solution: detect repetition
34