Transcript Слайд 1 - Center For Machine Perception (Cmp)

Curvature Prior for MRF-based
Segmentation and Shape Inpainting
This work was supported bu EU projects FP7-ICT-247870
NIFTi and FP7-ICT-247525 HUMAVIPS and the Czech project
1M0567 CAK
DAGM-OAGM 2012
Alexander Shekhovtsov, Pushmeet Kohli and Carsten Rother
Czech Technical
University in Prague
Motivation
2/21
 Would like to have a model tailored for the specific shape class
Looked at higher-order MRFs and Field of Experts
Experts
Pixels
 Focus on the curvature cost as a simple example of a shape model
A.Shekhovtsov, P. Kohli, C. Rother
Motivation
3/21
 How can we model shapes with higher-order models?
Black and Roth. (2009)
Field of Experts
- nonlinear function of linear filters
- continuous variables
hard pattern
Komodakis and Paragios (2009)
E h (x) =
Pattern-based Higher Order Potentials
Rother et al. (2009)
Sparse Higher Order Potentials
½
Ãi ; if x = pi
Ãm ax ; ot herwise
E h (x) = min hx; wy i + cy
y
expert state
soft pattern
A.Shekhovtsov, P. Kohli, C. Rother
Curvature in Discrete Setting
4/21
 Most of the works go for explicit edge representation (discrete setting)
Brukstain (2001) approximation
Cell-complex
Schoenemann et al. (2009)
Schoenemann, Kahl ,et al. (2011)
Schoenemann, Kuang, et al. (2011)
Strandmark and Kahl (2011)
straight on a large scale, but highly penalized

Convex relaxations in the continuous setting: Bredies et al. (2012), Goldluecke and
Cremers (2011)
A.Shekhovtsov, P. Kohli, C. Rother
The Model
5/21
 Keep the segmentation pixel-wise but assess curvature from a local window
window of the higher-order model
think of the curve with the lowest possible
curvature consistent with discretization
lager windows have a better chance of a more accurate estimate
A.Shekhovtsov, P. Kohli, C. Rother
You would never thought of this curve,
unless you know something
The Model
6/21
Rother et al. (2009)

x – pixel-wise segmentation
densely, at every pixel location,
there is a higher-order term
E (x) =
Energy
Higher-order term:
P
h2 U
E h (Vh (x))
restriction to the window
window locations
E h (x) = min hx; wy i + cy
y
for fixed y a modular (linear) functions of x
lower envelope of the modular functions of x
A.Shekhovtsov, P. Kohli, C. Rother
The Model
7/21
E h (x) = min hx; wy i + cy
y
 What this model can do?
w=
x=
¡ B +B
w1 =
¡ B +B
w=
¡ B 0+B
w2 =
A.Shekhovtsov, P. Kohli, C. Rother
1
¡ B
in the minimum
0
x=
+B
x=
1
0
or
1
0
1
0
Minimization
8/21
 Good news: minimization reduces to pairwise model:
X
X
¡
¢
min
min hx Vh ; wy i + cy = min
hx Vh ; wy i + cy
x
h2 U
y
x ;( y h j h )
-join optimization in segmentation
and latent variables y
h
X X
f h i (x i ; yh )
expands as
h
i
can combine with standard MRF models
 Bad news: still hard to optimize 
- BP-S/TRW-S (Kolmogorov, 2006) implementation saving
a factor of NP (number of patterns) memory
A.Shekhovtsov, P. Kohli, C. Rother
(lazy asymmetric message handling)
BP Schedule dependence
9/21
Input (inpaint the gray area)
A.Shekhovtsov, P. Kohli, C. Rother
Solution by BP-S (max-product)
(swep from left to right,
from top to bottom)
Learning
11/21
 For the case of curvature model, we have a simpler learning problem – we
can learn the model locally.
Generate smooth curves:
Discretize:
true curvature cost
(analytic)
Fit the lower envelope model
K-means like algorithm, needs good initialization
A.Shekhovtsov, P. Kohli, C. Rother
Learning
12/21
cost function to learn
learned patterns
example
(circle radius = model cost)
size 8x8
96 in total
A.Shekhovtsov, P. Kohli, C. Rother
predefined patterns:
assign 0 cost to off-boundary locations
Learning
13/21
 Approximation Error
Testing shape samples (analytic)
A.Shekhovtsov, P. Kohli, C. Rother
discrete approximation vs.
exact contour integral
(overestimating)
Shape Inpainting
14/21
area for inpainting
known segmentation
inpainted segmentation
A.Shekhovtsov, P. Kohli, C. Rother
Shape Inpainting
15/21
A.Shekhovtsov, P. Kohli, C. Rother
Shape Inpainting
16/21
A.Shekhovtsov, P. Kohli, C. Rother
Segmentation
17/21
Input with user seeds
X
E (x) =
X
E v (x v ) + ¸
v
E h (Vh (x))
h2 U
(saturation)
curvature strength
A.Shekhovtsov, P. Kohli, C. Rother
Segmentation (skip)
18/21
Input with user seeds
Standard length regularization
regularization strength
A.Shekhovtsov, P. Kohli, C. Rother
Segmentation (more)
19/21
 Extending the model: we added artificially an ear pattern.
 its cost was tuned manually
before
A.Shekhovtsov, P. Kohli, C. Rother
after
Curvature and Length Regularization
20/21
only curvature
A.Shekhovtsov, P. Kohli, C. Rother
curvature + length
curvature + more length
Towards Object Inpainting
21/21
area for completion
our shape inpainting
interactive segmentation
Texture added automatically
thanks to Barnes et al. (2009)
A.Shekhovtsov, P. Kohli, C. Rother