Transcript Document

Graph Cuts based
segmentation
methods:
Daniel Heilper,
CS Department, Haifa University
Topics:





Segmentation as Graph cuts problems.
MIN-CUT and Normalized CUT approaches
Examples of Cost/Energy functions
Performance issues – feasibility and
approximations
Relations to Clustering and other methods
Motivation:

Human perception takes into account
several perceptual aspects and “blends”
them together:
Proximity:
Human perception

Proximity:

Closure:
Human perception

Similarity

Continuation
Early segmentation methods





Boundary based methods
- based on local information and cannot
guarantee continuous closed edge contours
(Derivatives kernels, Harris, Canny)
Region based methods
+ statistics inside the region.
- often generates irregular boundaries and small
Holes.
Boundary based methods

Original text

Canny:
Boundary based methods

Derivatives:
Region based methods
Global optimization approaches based
on energy functions

Construct a heuristic energy function which
take into account both the local information
and the global information.
Graph Representation:


The set of points in an arbitrary feature
space are represented as a weighted
undirected
graph G = (V,E,w) where the nodes of the
graph are the points in the feature space,
and an edge is formed between every pair
of nodes. The weight on each edge, Wij is
a function of the similarity between nodes i
and j.
Graph Representation:
Energy cost functions:


Energy Function is heuristic for
quantization of a combination of Data
Features on an N-D Image.
Simple Examples: Distance metric – Image
is sent as a binary image, graph is
represented as points in the plane
Energy functions for Graph Cuts


In some cases, energy minimization is
NP-Hard, or has high polynomial order
which make the method impractical
Choosing a good energy function is not
“Pure” science
Energy functions for Graph Cuts

The most Natural Energy form is:

V - Boundary Penalty caused by choosing
labels for 2 pixels
D – Regional Penalty for assigning a pixel
to some label.

First approach: Max-Flow Min-Cut
Refresher: Flow Networks:
Network flow graphs





Given A flow Networks N  G, s, t , c  where:
G  (V , E ) is a directed graph
s  V is a source (not necessary belongs to G)
t  V is a sink (not necessary belongs to G)
c : E  R  a non-negative capacity function on the
edges (if (u, v)  E we can assume c(u, v)  0 )
Network flow graphs



Function is a valid flow function if holds:
Capacity constraints: f u, v  c u, v, u , v E
counter symmetry: for each , v, u V
f u , v    f v ,u 

flow conservation:  f (e)   f (e), where N  (v)
, are the edge set that leave V.
Similarly N  (v) are the sedge set that
enters V v V \ s, t
eN  ( v )
eN  (v )
Network flow graphs

The value of the flow function is
 f s,v   f (e)   f (e)
vV
eN  ( s )
eN  ( s )
Network flow graphs
Network flow graphs
Network flow graphs


Max-Flow – Min-Cut Theorem:
The maximum amount of flow is equal to
the capacity of a minimum cut
Interactive Segmentation application
using Graph Cuts:


Motivation: for some applications: Fully
automatic segmentation does not give
sufficient results.
for example, in medical diagnostics,
Doctors handlings a medical scan images
where soft tissues, blood vessels, and
bones are “blended” together , are very
difficult to segment.
Above: CAT scan of Pancreas.
Interactive Segmentation application
using Graph Cuts

In a nutshell:

User marks some pixels as seeds of “Background” and
“Object”, and runs the application several times until a
satisfactory result achieved.

Interactive inputs:



“seeds” pixels
the ratio between regional importance VS boundary
importance
Energy function (Intensities, distances, textures, etc..)
Seeds in the flow graph
Interactive Segmentation application
using Graph Cuts




First run, application fully solves max-flow task
Each time the application outputs an insufficient
result, the user adds more “seeds”, (changes the
ratio
) and rerun the application.
The segmentation is improved until satisfactory by
user.
Only the first run is computationally heavy, while
the consecutive runs are fast, and can be seen as
an addition to capacities, and hence finding
additional “augmenting paths” in the flow function
Past Interactive tools for segmentation:




intelligent scissors
live wire
“shortest path” between certain pixels using
some energy function
Difficulties:


the user inputs have to be very accurately
positioned at the desired boundary
can not be easily generalized to 3D data.
Energy function:

General form of the energy function:

Where
Energy function:


R(A) stands for the sum of the regional
penalties of a given labeling
Similarly B(A) is the sum of the boundry
penalties of the labeling
3D - Data

Video sequence is treated as a 3D volume
image:
Second Method: Normalized Cut:


The normalized cut criterion measures both
the total dissimilarity between the different
groups as well as the total similarity within
the groups.
The key point is that image partitioning is to
be done from the big picture downward,
rather like a painter first marking out the
major areas and then filling in the details.
Normalized Cut

Unfortunately, minimizing normalized cut
exactly is NP-complete, even for the
special case of graphs on
grids(Papadimitriou). However, an
approximate discrete solution can be found
efficiently
MINCut problem


Why minimum cut isn’t always favorable?
the minimum cut criteria favors cutting
small sets of isolated nodes in the graph.
(for example when cost function is inverse
to the distance between nodes)
A case where minimum cut gives a
bad partition
.
Definition: Normalized cut

where assoc(A,V) is the total connection
from nodes in A to all nodes in the graph
Normalized cut

It is easy to see that single nodded cuts
produce MINCut value of at least 1, so
minimum normal cut is not likely to be
single nodded.
Definition: Normalized Association:

In the same spirit, we can define a measure
for total normalized association within
groups for a given partition:
NAssoc vs NCut

A direct conclusion:



NCut(A,B) = 2 – Nassoc(A,B)
Ncut is normalized measure of
disassociation
minimizing Ncut is equivalent to maximizing
association
Computing the Optimal Partition

An approximate discrete solution can be
found efficiently by converting the problem
into a generalized eigenvector problem
NCut as Eigenvector system




Matrix representation
W is the cost Matrix
D is the sum of the costs from node i:
The Partition is a binary matrix size |P|XK where
|P| is the number of pixels and K is the number of
labels
NCut as Eigenvector system

The Segmentation matrix y:
Each column is a label vector

NCut can be written as:
Recursive two-way Ncut grouping
algorithm
1.
2.
Given an image or image sequence, set
up a weighted graph G=(V, E) and set the
weight on the edge connection two nodes
to be a measure of the similarity between
the two nodes.
Solve
for eigenvectors
with the smallest eigenvalues
Recursive two-way Ncut grouping
algorithm
3.
4.
Use the eigenvector with the second
smallest eigenvalue to bipartition the
graph
Decide if the current partition should be
subdivided and recursively repartition the
segmented parts if necessary.
Experiments on energy functions:


where X(i) is the spatial location of node i, and F(i)
is a feature vector based on intensity, color, or
texture information
If can separate the regional weight from the
boundary weight using log operation



(a) A synthetic image showing a noisy ªstepº image. Intensity
varies from 0 to 1, and Gaussian noise with sigma = 0.2 is
added.
(b) eigenvector with the second smallest eigenvalue
(c) resulting partition.
Performance and optimality



Solving an eigenvector system takes at
least (|P|^3) time
Using iterative power method we can find
the second lowest eigenvector in a
O(|P|*|n|) where |P| is #pixels, and |n| is
#iterative steps
The NCut at the end of the algorithm has
proven constant proximity to the MIN NCut
Finding the Minimum Normalized Cut
VS interactive MINCut finding:

Min NCut Pros:



Avoids segmenting isolated small sections (VS
“blind” calculation of MINCut)
This NP-Hard can be well approximated
Min NCut cons:

Difficult to control and analyse
Finding the Minimum Normalized Cut
VS interactive MINCut finding

interactive MINCut Pros:


Allows a fast correction of the result in an
interactively manner, and converges
Easy to control
Energy Minimization Techniques for
Graph Cuts



Goal: giving a faster algorithm to the MINCut problem or a general K-Labeling
partition of an Image
Some energy functions makes the MinCut
problem to be NP-Hard
Other functions reach optimal solution in
impractical polynomial time
Known algorithms for energy
minimization:



Simmulated Anealing – converges well but for
small temperature delta’s is exponential in time,
and for large deltas it converges to local
minimum which may be far from the minimum
Combinatorial Max flow – Min cut algorithms.
Combinatorial algorithms solves the problem in
at least O(|P|^3) time (Tarjan & Goldberg).
dynamic programming – restricted to 1-D energy
function
linear time algorithm for finding the
Max-Flow-Min-Cut/


Updating the labeling function:
two types of large moves: α-expansion and
α-β-swap
α-Exapnsion


Take a subset of pixels which are not
labeled to have that label
Works for a metric energy function:
α-β swap


Changing a label α of some pixels to label
β
Works for semimetric Energy functions:
α-Exapnsion
α-Exapnsion and {α, β}–Swap
α-Exapnsion and {α, β}–Swap


these moves allow large number of pixels
to change their labels simultaneously
Note that a move which assigns a given
label α to a single pixel is both an α-β swap
and an α-expansion.
(α-β swap can be seen as a private case of
α-Exapnsion)
{ α} Expansion



1. Start with an arbitrary labeling f
2. Set success := 0
3. For each label α belongs L




3.1. Find ˆ f = argminE(f0) among f0 within one
α-expansion of f
3.2. If E(f*) < E(f), set f := f* and success := 1
4. If success = 1 goto 2
5. Return f
{α, β} – Swap algorithm



1. Start with an arbitrary labeling f
2. Set success := 0

3. For each pair of labels {α, β} belongs to L





3.1. Find f* = argminE(f’) among f’ within one α-β swap
of f
3.2. If E(f*) < E(f), set f := f* and success := 1
4. If success = 1 goto 2
5. Return f
Algorithms summary



A cycle is successful if a strictly better
labeling is found at any iteration.
The algorithms stop after the first
unsuccessful cycle since no further
improvement is possible.
α-Exapnsion much faster than α-β swap :
|L| VS o(|L|^2) iterations (L is the labeling)
Benchmark results

(Results of running the algorithms on the previous
image)
References:



[1] J. Shi and J.Malik,
“Normalized Cuts and Image Segmentation,” Proc. CVPR 1997.
[2] Boykov, Y., Jolly, M., " Interactive graph cuts for optimal
boundary and region
segmentation of objects in N-D images." In: International
Conference on Computer Vision, Vancouver , BC. (2001) 105–112
[3] Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy
minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell.
23 (2001) 1222–1239

[4] Goro Obinata, Ashish Dutta: Vision Systems: Segmentation and
Pattern Recognition, I-Tech Education and Publishing, Vienna,
Austria