Transcript Fast Interactive Image Segmentation by Discriminative Clustering Dingding Liu * Xiong † Kari Pulli † Linda Shapiro * Yingen † Nokia Research Center, Palo Alto, CA 94304,

Fast Interactive Image
Segmentation by Discriminative
Clustering
Dingding Liu *
Xiong †
Kari Pulli † Linda Shapiro *
Yingen
† Nokia Research Center, Palo Alto, CA 94304, USA
*Dept. Elect. Eng., University of Washington, WA 98095, USA
1
Nokia Research Center
Research Aim
• Cut out an object from its background fast
• Computation time – so can quickly iterate
• With as few strokes as possible
2
Nokia Research Center
Overview
• Introduction
• Motivation
• Related work
• Algorithm
• Pre-segmentation by the Mean-Shift algorithm
• Merge regions by discriminative clustering
• Local neighborhood region classification and pruning
• Experiments and Results
• Conclusions and Future Work
3
Nokia Research Center
Introduction
• Motivation: Image editing on mobile devices
• Convenience
• Anytime, anywhere
• Challenges
• Limited computational resources
• Smaller screens and imprecise input
4
Nokia Research Center
Related Work Interactive Image Segmentation
• Lazy Snapping
• Li et al., ACM Transactions on Graphics 2004
5
Nokia Research Center
Related Work Interactive Image Segmentation
• Interactive Image Segmentation by
Maximal Similarity Based Region Merging
• Ning et al., Pattern Recognition 2010
Insufficient
user inputs
Sufficient
user inputs
6
Nokia Research Center
Algorithm: Summary
1. Pre-segmentation by the Mean-Shift algorithm
2. Merge regions by discriminative clustering
3. Local neighborhood region classification and pruning
7
Nokia Research Center
Background: Mean-Shift Segmentation
http://www.caip.rutgers.edu/~comanici/MSPAMI/msPamiResults.html
http://robots.stanford.edu/cs223b04/CS%2520223-B%2520L11%2520Segmentation.ppt
8
Nokia Research Center
The Basic Mean-Shift Algorithm
1. Choose a search window size
2. Choose the initial location of the search window
3. Compute the mean (centroid of the data) within the search window
4. Center the search window at that mean location
5. Repeat 3 and 4 until convergence
The mean shift algorithm seeks the
“mode” or point of highest density
of a data distribution
9
Nokia Research Center
Mean-Shift Segmentation
1. Convert the image into tokens (via color, gradients, texture measures, etc.)
2. Choose initial search window locations uniformly in the data
3. Compute the mean shift window location for each initial position
4. Merge windows that end up on the same “peak” or mode
5. Repeat 3 and 4 until convergence
10
Nokia Research Center
Mean-Shift Segmentation Results
11
Nokia Research Center
Algorithm: Pre-segmentation using MeanShift
• Three reasons for choosing the
Mean-Shift algorithm:
1. It preserves the boundaries better
than other methods
Pre-segmentation can be done
either before or after the user
input
12
Nokia Research Center
2. Its speed has been improved
significantly in recent years
3. Fewer parameters to tune
Algorithm: Merge non-ambiguous regions
df > dthresh + db, background
df + dthresh < db, foreground
Otherwise, ambiguous regions
Only consider color, not location
Create two kd-trees in CIELab color space
• One for the marked foreground, another for the background regions
For each unmarked region, find the color difference to
• the most similar marked background db and foreground region df
Choice of dthresh :
• Min difference of mean colors between the marked foreground and background
• that is higher than a minimum allowed distance (we chose 2)
13
Nokia Research Center
Algorithm: Assign ambiguous regions
Now use also location information
Each of the remaining ambiguous regions is assigned
• the label of the neighboring region that has the most similar mean color
If the most similar neighboring region is also an unmarked region
• merge them to a new unmarked region, repeat the process
If there is a tie in the mean color for assignment to foreground and background
• the label of the region that has the most similar color variance is used
14
Nokia Research Center
Algorithm: Prune / flip isolated regions
Find isolated foreground or background regions (use connected components)
Regions are changed to the opposite label when all of the following hold:
(a) The region is not marked by the user
(b) The region is not the biggest region with that label
(c) The region is smaller than its surrounding regions
15
Nokia Research Center
Results: Segmentation time – in numbers
16
Nokia Research Center
Results: Segmentation time – as a graph
80
70
60
proposed
algorithm
time(s)
50
MSRM
40
graph cut
30
20
10
0
carsten
cheetah
babyp
cow
mushroom
bird
goat (failure case)
different images
17
Nokia Research Center
Why are we faster?
• Two main reasons
• No iterative steps in the first stage
• and not too many in the second or third
• do the easy choices quickly
• fast nearest-neighbor lookups with kd-trees
• graph-cut on many regions is slow, MSRM iterates unnecessarily much
• Merging the region descriptor is fast
• only mean and standard deviation of colors
• MSRM has complicated 4K bin color histograms to merge
18
Nokia Research Center
Results: Segmentation time on phone
19
Nokia Research Center
Results: The best segmentation quality
(a) Input
image
20
Nokia Research Center
(b) Graphcut over
regions
(c) Maximal
Similar
Region
Merging
(d) Proposed
method
Results: The best segmentation quality
(a) Input image
21
Nokia Research Center
(b) Graph-cut
over regions
(c) Maximal
Similar Region
Merging
(d) Proposed
method
Results: The best segmentation quality
(a) Input image
22
Nokia Research Center
(b) Graph-cut
over regions
(c) Maximal
Similar Region
Merging
(d) Proposed
method
Results: Video Demo
23
Nokia Research Center
Conclusions and Future Work
• A new region-based interactive image segmentation algorithm
• Significantly increases the speed of segmentation
• by avoiding global optimization and long iterations
• Does not compromise the segmentation quality
• Uses a region mean color instead of a single pixel color
• Future Work
• Further decrease the users input
• Combine the individual pixel information to further improve the algorithm
24
Nokia Research Center
Thank you!
Questions?
Nokia Research Center