Document

Transcript Document

Gebze Institute of Technology Department of Computer Engineering Computer Vision Stereo

Stereo

The ability to infer information on the 3D structure and distance of a scene from two or more images taken from different viewpoints.

Two Problems Of Stereo

• • We need to solve two problems for estimating 3D structure and distance

Correspondence problem

is determining which parts of the left and right (or other) images correspond to each other.

Reconstruction problem

uses the correspondences and the camera geometry to recover the 3D structure and the distance.

O 1

Finding Correspondences

Finding Correspondences:

3D Reconstruction

P P’ 1 P’ 2 O 1 We must solve the correspondence problem first!

Correspondence and 3D reconstruction

A simple stereo system

Simple Stereo Model

1  

f X Z x

2  

f X





1 

Z f B Z





Parameters of a stereo system

P •Intrinsic: •f: focal length of cameras •c l and c r : principal points Z c l x l O l p l T p r x r c r O r •Extrinsic: •T: stereo baseline •Transformation between cameras for a more general configuration © 2005 Yusuf Akgul

The Correspondence Problem

• Basic assumptions: – Most scene points are visible in both images – Corresponding image regions are similar • These assumptions hold if: – The distance of the

fixation point

The Correspondence Problem

• Is a “search” problem: – Given an element in the left image, search for the corresponding element in the right image.

Correspondence Problem

Correlation-based Algorithms

• Elements to be matched by a similarity measure: – Image WINDOWS of fixed size.

What can we use for similarity?

Correlation Based Correspondence

Comparing Windows:

Some possible measures: f ?

“SSD” or “block matching” (Sum of Squared Differences)

It is the most popular.

Cross-Correlation C

Finding the disparity map

CORR_MATCHING Algorithm

• Let p l and p r be pixels on the I l and I r • Let R(p l ) be the search window w x w associated with p l • Let d be the displacement between p l in R(p l ).

CORR_MATCHING Algorithm

• For each pixel p l =[i,j] in I l do: – For each displacement d=[d 1 ,d 2 ] in R(p l ) do: C(d) =  k=-W k=W  k=-W k=W Y (I l (i+k,j+l),I r (i+k-d 1 ,j+l-d 2 )) – The disparity of p l is the vector d that maximizes C(d) over R(p l ) • Output the disparity for each pixel p l © 2005 Yusuf Akgul

• W

How do we set W, R and

w (width of the correlation window):

?

– should be based on the “scale” of the scene.

• R(P l ) and w (search window): – Size should be estimated based on the range of scene distances and the baseline: • Z = fT/d or d = fT/Z – The position of R(Pl) is centered around the same pixel on both images.

Feature-based Methods

• Conceptually very similar to Correlation based methods, but: – They only search for correspondences of a sparse set of image features.

– Correspondences are given by the most similar feature pairs.

– Similarity measure must be adapted to the type of feature used.

Feature-based Methods:

• Features most commonly used: – Corners • Similarity measured in terms of: – surrounding gray values (SSD, Cross-correlation) – location – Edges, Lines • Similarity measured in terms of: – orientation – contrast – coordinates of edge or line’s midpoint – length of line © 2005 Yusuf Akgul

Example: Comparing lines

• • l l q l and l r : line lengths and q r : line orientations • (x l ,y l ) and (x r ,y r ): midpoints • • c l w l and c r : average contrast along lines w q w m w c : weights controlling influence The more similar the lines, the larger S is!

FEATURE_MATCHING Algorithm

• For each feature f l in the left image: – Compute the similarity measure between f l and every feature in the search window R(f l ) – Select the feature in R(f l ) that maximizes the similarity measure.

Which method should we use?

• Correlation methods: – dense maps, good for surface reconstruction – Require textured images – Sensitive to illumination variations – Inadequate for very different viewpoints • Feature methods: – Sparse maps, good for navigation – Require prior knowledge of type of scene – Must find features first © 2005 Yusuf Akgul

Stereo with Parallel Cameras

• Stereo with Parallel Axes – Short baseline • large common FOV • large depth error – Long baseline • small depth error • small common FOV • More occlusion problems

FOV Left right

Stereo with Parallel Cameras

• Stereo with Parallel Axes – Short baseline • large common FOV • large depth error – Long baseline • small depth error • small common FOV • More occlusion problems • Depth Accuracy vs. Depth – Depth Error is proportional to Depth 2 – Nearer the point, better the depth estimation

FOV Left right

Stereo with Converging Cameras

• Two optical axes intersect at the Fixation Point – converging angle q – The common FOV Increases

FOV Fixation point

Left right

Stereo with Converging Cameras

• Disparity properties – Disparity uses angle instead of distance – Zero disparity at fixation point • and the Zero-disparity horopter

Horopter Fixation point

Left

r =

l d

= 0

right r

Stereo with Converging Cameras

Disparity properties – Disparity uses angle instead of distance – Zero disparity at fixation point • and the Zero-disparity horopter – Disparity increases with the distance of objects from the fixation points • >0 : outside of the horopter • <0 : inside the horopter

Horopter Fixation point

q a

r >

l d

> 0 right Left

Stereo with Converging Cameras

Horopter Fixation point

r <

l d

< 0 right Left

Stereo with Converging Cameras

• Disparity properties – Disparity uses angle instead of distance – Zero disparity at fixation point • and the Zero-disparity horopter – Disparity increases with the distance of objects from the fixation points • >0 : outside of the horopter • <0 : inside the horopter

Horopter Fixation point

Left

a) ?

r right

Constraining the Search Space

• We could have used additional constraints in both feature based and correlation based algorithms.

What constraints can we use?

Constraining the Search Space

• We could have used additional constraints in both feature based and correlation based algorithms.

• Motivation: where to search correspondences?

Epipolar Geometry

– Epipolar Plane • A plane going through point P and the centers of projections (COPs) of the two cameras

Epipolar Plane

– Conjugated Epipolar

P l P P r

Lines • Lines where epipolar plane intersects the image planes – Epipoles • The image of the COP of one camera in the other • Epipolar Constraint – Corresponding points must lie on conjugated epipolar lines

O l p l

e l Epipolar Lines Epipoles e r p r O r

Epipolar Geometry

• How do we find the

P l

epipolar lines?

Epipolar Plane

• What do we need to find them?

Epipolar Lines p l P r p r O l e l Epipoles e r

O r

Epipolar Geometry

• Notations – P l =(X l , Y l , Z l ), P r =(X r , Y r , Z r ) • Vectors of the same 3-D point P, in the left and right camera coordinate systems respectively – Extrinsic Parameters • Translation Vector T = (O r -O l ) • Rotation Matrix R

P r



R(P l



T) X l Y l O l p l f l Z l P l R, T

– p l =(x l , y l , z l ), p r =(x r , y r , z r ) • Projections of P on the left and right image plane respectively • For all image points, we have z l =f l , z r =f r



f l Z l

P l P r Z r X r p



f r Z r

p r Y r f r O r

Essential Matrix

• Equation of the epipolar plane – Co-planarity condition of vectors P l , T and P l -T

(P l





P l

 0

P r



R(P l

• Essential Matrix E = RS (

– 3x3 matrix constructed from R and T (extrinsic only) )

 

B T A T

• Rank (E) = 2, two equal nonzero singular values

    

r r r

11 21 31

33    

Rank (R) =3

      0

T z T y



T z

T x



T y T x

0    

Rank (S) =2 P r T EP l

 0



f Z l l

P l p



f Z r r

p r T Ep l

Essential Matrix

• Essential Matrix E = RS

p r T Ep l

 0 – A natural link between the stereo point pair and the extrinsic parameters of the stereo system • One correspondence -> a linear equation of 9 entries • Given 8 pairs of (pl, pr) -> E – Mapping between points and epipolar lines we are looking for • Given p l , E -> p r on the projective line in the right plane • Equation represents the epipolar line of pr (or pl) in the right (or left) image • Note: – pl, pr are in the camera coordinate system, not pixel coordinates that we can measure © 2005 Yusuf Akgul

Fundamental Matrix

• Mapping between points and epipolar lines in the pixel coordinate systems – With no prior knowledge on the stereo system • From Camera to Pixels: Matrices of intrinsic parameters

int       0 0

f x

0 

f y

x o y

   

Rank (M int ) =3

• Questions: – What are fx, fy, ox, oy ?

– How to measure p l in images?

p l p r



M l



1 p l p T Ep r T F l p l



0   0



1 p





T EM

Fundamental Matrix

• Fundamental Matrix – Rank (F) = 2





T EM

 1 – Encodes info on both intrinsic and extrinsic parameters – Enables full reconstruction of the epipolar geometry – In pixel coordinate systems without any knowledge of the intrinsic and extrinsic parameters – Linear equation of the 9 entries of F

p r T F p l

 0 ( (

l x im

) (

y im l

) 1 )    

f f

11 21

f f f

13 23      33   1 (

x im r

) (

y im r

    

33    

     

T z y



T z

T x T y



T x

0    





T EM

 1

Essential and Fundamental Matrix Properties

int       0 0

f x

0 

f y

x o y

How Do We Estimate E and F?

The idea is simple: • Establish correspondences between two stereo images.

• Each correspondence gives us an equation • Solve the linear set of equations to get the Matrix (F) elements.

• How do we solve a homogenous system of linear equations?

• Singular Value Decomposition: – Any mxn matrix can be written as the product of three matrices



UDV

V 1 U 1

      

a a a

11 21

a m

a a

n a mn

          

u u

   

11 21

u m

m u

m u mm

              s 0 0 0 1 0 s 2 s 0 0

           

12 

n v

  Singular values s i  are fully determined D is diagonal: d ij by A =0 if i  j; dii = s i (i=1,2,…,n)  s 1  s 2  …  s N  0 Both U and V are not unique  Columns of each are mutual orthogonal vectors © 2005 Yusuf Akgul

v v n

v n

     

Singular Value Decomposition

• 1. Singularity and Condition Number



UDV

– nxn A is nonsingular IFF all singular values are nonzero – Condition number : degree of singularity of A

 s 1 / s

• A is ill-conditioned if 1/C is comparable to the arithmetic precision of your machine; almost singular • 2. Rank of a square matrix A – Rank (A) = number of nonzero singular values • 3. Inverse of a square Matrix – If A is nonsingular – In general, the pseudo-inverse of A

 

 1 0

• 4. Eigenvalues and Eigenvectors – Eigenvalues of both A T A and AA T are si 2 (si > 0) – The columns of U are the eigenvectors of AA T (mxm) – The columns of V are the eigenvectors of A T A (nxn)

 s  s

i i

2 2

Singular Value Decomposition

• Homogeneous System – m equations for n unknowns

(m >= n-1) – Rank (A) = n-1 (by looking at the SVD of A)



– A non-trivial solution (up to a arbitrary scale) by SVD: – Simply proportional to the eigenvector corresponding to the only zero eigenvalue of A T A (nxn matrix) • Note:

 s

– All the other eigenvalues are positive because

Singular Value Decomposition

• Problem Statements – Numerical estimate of a matrix A whose entries are not independent – Errors introduced by noise alter the estimate to Â • Enforcing Constraints by SVD – Take orthogonal matrix A as an example – Find the closest matrix to Â, which satisfies the constraints exactly • SVD of Â 

UDV

Computing F: The Eight-point Algorithm

• Input: n point correspondences ( n >= 8) – Construct homogeneous system Ax= 0 from

p r T F

• x = (f 11 ,f 12 , ,f 13 , f 21 ,f 22 ,f 23 f 31 ,f 32 , f 33 ) : entries in F • Each correspondence give one equation

p l

 0 • A is a nx9 matrix – Obtain estimate F^ by SVD of A



UDV

• x (up to a scale) is column of V corresponding to the least singular value 

UDV

– Enforce singularity constraint: since Rank (F) = 2 • Compute SVD of F^



• Set the smallest singular value to 0: D -> D’ • Correct estimate of F : • Output: the estimate of the fundamental matrix, F’ • Similarly we can compute E given intrinsic parameters © 2005 Yusuf Akgul

Locating the Epipoles from F

p r T F p l

 0

e l lies on all the epipolar lines of the left image

P l Epipolar Plane P r p r T F e l

 0

For every p r Epipolar Lines pl

F is not identically zero

F e l

 0

O l e l e r Epipoles

• Input: Fundamental Matrix F – Find the SVD of F



UDV

– The epipole e l is the column of V corresponding to the null singular value (as shown above) – The epipole e r is the column of U corresponding to the null singular value • Output: Epipole e l and e © 2005 Yusuf Akgul r

p r O r

Stereo Rectification

 Stereo System with Parallel Optical Axes  Epipoles are at infinity  Horizontal epipolar lines

P l P r Y’ l p’ l Y’ r Z’ l

• Rectification

X’ l O l T X’ r O r

– Given a stereo pair, the intrinsic and extrinsic parameters, find the image transformation to achieve a stereo system of horizontal epipolar lines – A simple algorithm: Assuming calibrated stereo cameras © 2005 Yusuf Akgul

Z’ r

Stereo Rectification

Rectification

Stereo Rectification

• Algorithm – Rotate both left and right camera so that they share the same X axis : O r -O l = T – Define a rotation matrix Rrect for the left camera – Rotation Matrix for the right camera is RrectR T – Rotation can be implemented by image transformation

X’ l X l Y l p l O l T Z l P l R, T P P r X r Z r X l ’ = T, Y l ’ = X l ’xZ l , Z’ l = X l ’xY l ’ p r O r Y r

Stereo Rectification

X’ l X l Y l p l O l T Z l P l R, T P P r X r Z r X l ’ = T, Y l ’ = X l ’xZ l , Z’ l = X l ’xY l ’ p r O r Y r

Stereo Rectification

X’ l Y’ l p’ l O l T Z’ l P l R, T P P r X’ r T’ = (B, 0, 0), P’ r = P’ l – T’ Y’ r Z r O r

Stereo Rectification

• Read your book on how to obtain the rotation matrix for the rectification.

Epipolar Geometry: Summary

• Purpose – where to search correspondences

r T



 0 • Epipolar plane, epipolar lines, and epipoles – known intrinsic (f) and extrinsic (R, T) • co-planarity equation – known intrinsic but unknown extrinsic • essential matrix – unknown intrinsic and extrinsic • fundamental matrix

p r T Ep l

 0 • Rectification

p r T F p l

3D Reconstruction Problem

• What we have done – Correspondences using either correlation or feature based approaches – Epipolar Geometry from at least 8 point correspondences • Three cases of 3D reconstruction depending on the amount of a priori knowledge on the stereo system – Both intrinsic and extrinsic known - > can solve the reconstruction problem unambiguously by triangulation – Only intrinsic known -> recovery structure and extrinsic up to an unknown scaling factor – Only correspondences -> reconstruction only up to an unknown, global projective transformation © 2005 Yusuf Akgul

Reconstruction by Triangulation

• • Assumption and Problem – Under the assumption that both intrinsic and extrinsic parameters are known – Compute the 3-D location from their projections, pl and pr Solution – Triangulation : Two rays are known and the intersection can be computed – Problem: Two rays will not actually intersect in space due to errors in calibration and correspondences, and pixelization – Solution: find a point in space with minimum distance from both rays

O l

p l P p r O r

Reconstruction by Triangulation

How to Get the Stereo Params from Camera Params

Reconstruction up to a Scale Factor

• Assume that intrinsic parameters of both cameras are known • Essential Matrix is known up to a scale factor (for example, estimated from the 8 point algorithm).

Reconstruction up to a Scale Factor

• Assumption and Problem Statement – Under the assumption that only intrinsic parameters and more than 8 point correspondences are given – Compute the 3-D location from their projections, pl and pr, as well as the extrinsic parameters • Solution – Compute the essential matrix E from at least 8 correspondences – Estimate T (up to a scale and a sign) from E (=RS) using the orthogonal constraint of R, and then R • End up with four different estimates of the pair (T, R) – Reconstruct the depth of each point, and pick up the correct sign of R and T.

E EE T

Reconstruction up to a Scale



kSR



SRR T S T



Factor

SS T

    

2   (

T Y

k k

T X



T Y T Z T Z

2 )

2  (

k T X

2 2

T X



T Y T Z

2 

T Y T Z

) 

T X T Z k

2  (

k T X

2 2

T Y



T Z T Y

2 )     

Trace



EE T E k t

 sgn   2

2 (

T X

2 

T Y

2 

T Z

2 )  2

t S R

 sgn



      0

T z T y



T z

T x



T y T x

0     ˆ ˆ



     1   

X T

Y T

 1 

 ˆ

Y X T T

Y T

ˆ ˆ

Z Y

X T



Z T

2     We can get the components of T from this matrix easily.

Reconstruction up to a Scale Factor

     ˆ 1

2 ˆ

3    

    

T R

    Let

w i



E i



ˆ ,

 It can be proved that

R R

1 3   

2  

3 

3 

1 

Reconstruction up to a Scale Factor

We have two choices of

, (

t +

and

t )

because of sign ambiguity and two choices of

(E + and E

This gives us four pairs of translation vectors and rotation matrices.

Reconstruction up to a Scale Factor

ˆ 1. Construct the vectors

, and compute R 2. Reconstruct the Z and Z’ for each point 3. If the signs of Z and Z’ of the reconstructed points are a) both negative for some point, change the sign of

ˆ and go to step 2.

b) c) different for some point, change the sign of each entry

ˆ both positive for all points, exit.



( (



3 



3   

f f



1  

1  ) )

T T p t Z

  

 ( (

3  

1 ) ) ( )

T p

Document

Transcript Document

Gebze Institute of Technology Department of Computer Engineering Computer Vision Stereo

Stereo

Two Problems Of Stereo

Finding Correspondences

Finding Correspondences:

3D Reconstruction

Correspondence and 3D reconstruction

A simple stereo system

A simple stereo system

Simple Stereo Model

Parameters of a stereo system

The Correspondence Problem

The Correspondence Problem

Correspondence Problem

Correlation-based Algorithms

Correlation Based Correspondence

Comparing Windows:

“SSD” or “block matching” (Sum of Squared Differences)

Cross-Correlation C

Finding the disparity map

CORR_MATCHING Algorithm

CORR_MATCHING Algorithm

How do we set W, R and

?

Feature-based Methods

Feature-based Methods:

Example: Comparing lines

FEATURE_MATCHING Algorithm

FEATURE_MATCHING Algorithm

Which method should we use?

Stereo with Parallel Cameras

Stereo with Parallel Cameras

Stereo with Converging Cameras

Stereo with Converging Cameras

Stereo with Converging Cameras

Stereo with Converging Cameras

Stereo with Converging Cameras

Constraining the Search Space

Constraining the Search Space

Epipolar Geometry

Epipolar Geometry

Epipolar Geometry

Essential Matrix

Essential Matrix

Fundamental Matrix

Fundamental Matrix

Essential and Fundamental Matrix Properties

How Do We Estimate E and F?

Singular Value Decomposition

Singular Value Decomposition

Singular Value Decomposition

Computing F: The Eight-point Algorithm

Locating the Epipoles from F

Stereo Rectification

Stereo Rectification

Rectification

Stereo Rectification

Stereo Rectification

Stereo Rectification

Stereo Rectification

Epipolar Geometry: Summary

3D Reconstruction Problem

Reconstruction by Triangulation

Reconstruction by Triangulation

How to Get the Stereo Params from Camera Params

Reconstruction up to a Scale Factor

Reconstruction up to a Scale Factor

Reconstruction up to a Scale

Factor

Reconstruction up to a Scale Factor

Reconstruction up to a Scale Factor

Reconstruction up to a Scale Factor

No Parameters Known

Directory