Transcript 3D Geometry for Computer Graphics
Gaussian Mixture Models and Expectation-Maximization Algorithm
The RGB Domain A regular image 2
The RGB Domain Image pixels in RGB space 3
Pixel Clusters Suppose we cluster the points for 2 clusters 4
Pixel Clusters The result in image space 5
Normal Distribution (1D Gaussian) 1 2 exp
x
2 2 2 ,
mean
std
6
2D Gaussians
d
1 exp
x
T
1
x
2 d x = 2 = random data point (2D vector) = mean value (2D vector) = covariance matrix (2D matrix) The same equation holds for a 3D Gaussian 7
2D Gaussians
d
1 exp
x
T
1
x
2 8
Exploring Covariance Matrix
x i
1
N i N
1
x i i
, )
i
x i
T
w
2 1 is symmetric 2 has eigendecomposition (svd) ...
d
h
2 *
T
9
Covariance Matrix Geometry
a b
a
v
1*
b
v
2* * 1 2
T
10
3D Gaussians
x i
1
N i N
1
x i
x i
T
r
2
g
2
b
2 11
GMMs – Gaussian Mixture Models Suppose we have 1000 data points in 2D space (w,h)
W H
12
GMMs – Gaussian Mixture Models Assume each data point is normally distributed Obviously, there are 5 sets of underlying gaussians
W H
13
The GMM assumption There are K components (Gaussians) Each k is specified with three parameters: weight, mean, covariance matrix The total density function is:
j K
1
j
1
d
det(
j
) exp
x
j
T
j
1
x
j
2 {
j j
,
j
}
K j
1
weight
j
0
j j K
1
j
1 14
The EM algorithm (Dempster, Laird and Rubin, 1977)
i
i
Raw data GMMs (K = 6) Total Density Function 15
EM Basics Objective:
f x
1
x N
) Algorithm: 1. Guess initial 2. Perform E step (expectation) 3. Perform M step (maximization) Based on data points clustering, maximize 4. Repeat 2-3 until convergence (~tens iterations) 16
EM Details E-Step (estimate probability that point t associated to gaussian j) :
w
j
i K
1
i t
j
,
j
)
t
i
,
i
)
j
1,...,
K t
1,...,
N
M-Step (estimate new parameters) :
j new
new j
new j
1
N t N
1
w
t t N N
1
t N
1 1
w x t w w
(
x t
N
t
1
j new w
)(
x t
j new T
) 17
EM Example
data point t blue: w t,j Gaussian j
18
EM Example 19
EM Example 20
EM Example 21
EM Example 22
EM Example 23
EM Example 24
EM Example 25
Back to Clustering We want to label “close” pixels with the same label Proposed metric: label pixels from the same gaussian with same label Label according to max probability: arg max(
w j
) Number of labels = K 26
Graph-Cut Optimization
Motivation for Graph-Cuts Let ’s recall the car example 28
Motivation for Graph-Cuts Suppose we have two clusters in color-space Each pixel is colored by it ’s associated gaussian index 29
Motivation for Graph-Cuts A Problem: Noise Why? Pixel labeling is done independently for each pixel, ignoring the spatial relationships between pixels!
30
Formalizing a New Labeling Problem Previous model for labeling:
f p
arg max(
w j
)
j
1,...,
p
(
image pixels
) ) A new model for labeling. Minimize E:
E data
E smooth
f = Labeling function, assigns label f p E data = Data Term E smooth = Smooth Term Lamda is a free parameter for each pixel p 31
The Energy Function
E data
E smooth
Labels Set: { j=1, …,K } E data : Penalize disagreement between pixel and the GMM
D p
(
f p
)
D p
(
f p w p
E smooth : Penalize disagreement between two pixels, unless it ’s a natural edge in the image 0 ,
neighbors V
(
f p
,
f q
)
V
(
f p
,
f q
) 1
f p ow f q
dist(p,q) = normalized color-distance between p,q 32
Minimizing the Energy Solving Min(E) is NP-hard It is possible to approximate the solution using iterative methods Graph-Cuts based methods approximate the global solution (up to constant factor) in polynomial time Read: “Fast Approximate Energy Minimization via Graph Cuts ”, Y. Boykov, O. Veksler and R. Zabih, PAMI 2001 33
α-expansion moves When using iterative methods, each iteration some of the pixels change their labeling
Current Labeling One Pixel Move α-β-swap Move α-expansion Move
Given a label α, a move from partition P (labeling f) to a new partition P ’ (labeling f’) is called an
α-expansion
move if:
P
P
'
P l
'
P l
34
Algorithm for Minimizing E(f) 1. Start with an arbitrary labeling 2. Set success = 0 3. For each label j 3.1 Find f ’ = argmin(E(f’)) among f’ within one α-expansion of f 3.2 If E(f ’) < E(f), set f = f’ and success = 1 4. If (success == 1) Goto 2 5. Return f
How to find argmin(E(f ’)) ?
35
A Reminder: min-cut / max-flow Given two terminal nodes α and β in
G
=(
V
,
E
), a cut is a set of edges α from β in G’=(V,E\C) Also, no proper subset of C separates α from β in
G
’.
The cost of a cut is defined as the sum of all the edge weights in the cut. The
minimum-cut
of G is the cut C with the lowest cost.
The
minimum-cut
time.
problem is solvable in practically linear 36
Finding the Optimal Expansion Move Problem: Find f ’ = argmin(E(f’)) among f’ within one α-expansion of f Solution: Translate the problem to a min-cut problem on an appropriately defined graph.
37
Graph Structure for Optimal Expansion Move
Terminal α
f p
f p if node t
p
if node t p
C
C
Cut C Terminal not( α)
1-1 correspondence between cut and labeling E(f) is minimized!
38
A Closer Look Each pixel gets a node
P1 P2 Pα
39
A Closer Look
P1 P2 Pα
Add auxiliary nodes between pixel with different labels 40
A Closer Look
P1 P2 Pα
Add two terminal nodes for α and not(α) 41
A Closer Look
P1 P2 Pα
42
A Closer Look
P1 P2 Pα
43
A Closer Look
P1 P2 Pα
44
A Closer Look
P1 P2 Pα
45
A Closer Look
P1 P2 Pα
46
Implementation Notes Neighboring system can be 4-connected pixels, 8-connected and even more.
Lamda allows to determine the ratio between the data term and the smooth term.
Solving Min(E) is simpler and possible in polynomial time if only two labels involved (see “Interactive Graph Cuts for Optimal Boundary & Region Segmentation of Objects in N-D Images ”, Y. Boykov and M-P. Jolly 2001) There is a ready-to-use package for solving max-flow (see http://www.cs.cornell.edu/People/vnk/software/maxflow v2.2.src.tar.gz) 47
Final Project Optimized Color Transfer
www.cs.tau.ac.il/~gamliela/color_transfer_project/color_transfer_project.htm