Joint Disparity Map Estimation and Object Segmentation Cevahir Çığla and A. Aydın Alatan

Overview  Segment-Based Disparity Map(DM) Estimation • •

Assumptions Algorithm

•

Simulation Results

 Graph-Based Image Segmentation • •

Algorithm Simulation Results

 Joint DM Estimation and Object Segmentation  Conclusions and Future Work

Segment-Based DM Estimation

Assumptions

Disadvantages of pixel-based DM Estimation methods :    Problems at untextured regions.

Local intensity variation may not be efficiently utilized.

Object boundaries and edges may disappear.

Utilizing segments instead of pixels provide :     Object boundaries to be preserved.

Untextured regions to be handled.

Strict smoothness among the pixels in a segment.

Planarity assumption of the scene.

Segment-Based DM Estimation

Assumptions

Supposition for Segment-Based DM Estimation :     Locally same colored regions have the same disparity values.

Within each segment, disparity is constant [ 1 ].

Each pixel has unique match.

Depth discontinuities only at segment boundaries.

Extra constraints on Estimated DM :   DM should be smooth.

Reconstruction should be satisfactory.

[1]WEI Yichen,Quan Long, Region-Based Progressive Stereo Matching, cvpr 2004

Segment-Based DM Estimation

Algorithm

      Two rectified images are over-segmented.

Initial DM is estimated by segment matching. The second image is reconstructed by the initial DM.

Disparities are updated by iterative warping.

Visibility, smoothness and intensity matches are considered. Final DM is estimated when the updates converge.

Segment-Based DM Estimation

Algorithm

step1

  Over-segments have same local intensity properties.

Constant disparity assumption holds for small segments.

Teddy image sequence [2] [2] Middlebury stereo evaluation webpage

Segment-Based DM Estimation

Algorithm

step2

  For each segment, disparities are assigned by a search in the disparity space.

The cost function is defined as: C(j,di) = ∑

| I

L (x,y) –

I

R (x+di,y)

|

(x,y) є Sj

Segment-Based DM Estimation

Algorithm

step2

   Left-Right consistency check.

A consistent segment has 80% consistent pixels No disparity is assigned to inconsistent segments.

Segment-Based DM Estimation

Algorithm

step2

  Block matching for the pixels in the inconsistent regions.

The dominant disparity determines the segment disparity.

Segment-Based DM Estimation

Algorithm

step3

During the reconstruction of the second image :     The segments are shifted by the disparity value.

The texture values are determined by the visible pixels [ 3 ].

Visibility increases with an increase in the disparity.

Cost function is defined for each segment.

   Smoothness Overlapping area Intensity match [3] Bleyer M., Gelautz M. A layered stereo algorithm using Segmentation and global visibility constraint

Segment-Based DM Estimation

Algorithm

step3



Smoothness

 The neighboring segment disparities are compared.

 C S, j = Σ N i |d j d | i 

Overlapping

 Invisible pixels are penalized .

 C O, j = λ o .( # of invisible pixels ) 

Intensity match

 Intensity difference for the visible pixels .

Segment-Based DM Estimation

Algorithm

step4

Iteration step :     Initial costs for different segments are sorted.

Starting form the minimum cost  Check different disparities.

   Determine the best improvement in the cost function.

Update the segment disparity and the cost.

For all segments Resort the list and update segments.

Iteration stops when the updates converge.

Segment-Based DM Estimation

Simulation Results

250 200 150 100 50 0 0

Convergence Curve

2 4 6 8 iteration number 10 12 14 Ground truth

Segment-Based DM Estimation

Simulation Results

Ground truth

Graph-Based Image Segmentation

Algorithm

Algorithm properties:       Normalized Cut segmentation [ 4 ] is implemented.

Segments are utilized instead of pixels.

Modifications to overcome irregular segment distribution.

Recursive Two-Way Ncut [ 4 ] algorithm to bipartition the graph.

Automatic segmentation is achieved.

From the whole picture to downward.

[4] J. Shi and J. Malik. Normalized cuts and image segmentation. In Proc. IEEE Conf. Computer Vision and Pattern Recognition

Graph-Based Image Segmentation

Algorithm

Steps :  The image is oversegmented.

 A graph (G) is constructed by the segments.

   Each node represents the segments.

Edges are formed between neighboring segments.

Edge weights : function of similarity between node pair.

exp( -|X(i)-X(j)|* |X(i) X(j)|/σ ) if |X(i)-X(j)| 2 < d W i,j = 0 otherwise  Second smallest eigenvector of the generalized eigensystem.

Graph-Based Image Segmentation

Algorithm

 Irregular segment distribution     Irregular graph structure.

Different neighbor numbers for each segment.

Favors segments with more neighbors to unite.

Decreases the weight effect 

Strong weighted nodes may not unite

.

 Modifications :  Secondary neighboring is utilized.

 Max. Neighbor number is limited.

Graph-Based Image Segmentation

Simulation Results

Proposed algorithm Normalized-Cut Algorithm

Graph-Based Image Segmentation

Simulation Results

Proposed algorithm Normalized-Cut Algorithm

Graph-Based Image Segmentation

Simulation Results

 The proposed algorithm:     Performs faster.

Keeps object details better.

Utilize local intensity variation.

Results in promissing segmentation.

Joint DM Estimation and Object Segmentation  How to combine depth and intensity information?

 Depth similarity is defined between segments.

exp( -|D(i)-D(j)|* |D(i) D(j)|/ λ ) if |X(i)-X(j)| 2 < d V i,j = 0 otherwise  Edge weights are updated with Depth similarity.

W i,j = W i,j * V i,j  Bipartition the new Graph.

Joint DM Estimation and Object Segmentation Color and depth information Only color information

Conclusions and Future Work

      Satisfactory results for DM are obtained.

Comparitive tests with different segmentation algorithms Depth information usage can be modified DM Estimation algorithm should be adapted to:  Forshortening of segments.

 Multiview images.

Global optimization for cost minimization.

Disparity refinement in pixel resolution.

Conclusions and Future Work

 Currently, collaboration between METU and UIL   Refinement of the algorithm.

Comparison of view synthesis algorithms.

Thank you for listening…