Transcript Solving MRF using SOCP Relaxations

Solving Markov Random Fields
using
Second Order Cone Programming
Relaxations
M. Pawan Kumar
Philip Torr
Andrew Zisserman
Aim
• Accurate MAP estimation of pairwise Markov random fields
Label ‘1’
2
0
1
Label ‘0’
5
V1
2
1
0
0
4
2
V2
6
4
3
1
1
3
1
0
V3
Random Variables V = {V1,..,V4}
Label Set L = {0,1}
Labelling m = {1, 0, 0, 1}
3
7
V4
Aim
• Accurate MAP estimation of pairwise Markov random fields
Label ‘1’
2
0
1
Label ‘0’
5
V1
Cost(m) = 2
2
1
0
0
4
2
V2
6
4
3
1
1
3
V3
3
1
0
7
V4
Aim
• Accurate MAP estimation of pairwise Markov random fields
Label ‘1’
2
0
1
Label ‘0’
5
0
4
2
1
0
V1
Cost(m) = 2 + 1
2
V2
6
4
3
1
1
3
V3
3
1
0
7
V4
Aim
• Accurate MAP estimation of pairwise Markov random fields
Label ‘1’
2
0
1
Label ‘0’
5
V1
2
1
0
0
4
2
V2
Cost(m) = 2 + 1 + 2
6
4
3
1
1
3
V3
3
1
0
7
V4
Aim
• Accurate MAP estimation of pairwise Markov random fields
Label ‘1’
2
0
1
Label ‘0’
5
V1
2
1
0
0
4
2
V2
6
4
3
1
Cost(m) = 2 + 1 + 2 + 1
1
3
V3
3
1
0
7
V4
Aim
• Accurate MAP estimation of pairwise Markov random fields
Label ‘1’
2
0
1
Label ‘0’
5
V1
2
1
0
0
4
2
V2
6
4
3
1
1
3
V3
Cost(m) = 2 + 1 + 2 + 1 + 3
3
1
0
7
V4
Aim
• Accurate MAP estimation of pairwise Markov random fields
Label ‘1’
2
0
1
Label ‘0’
5
V1
2
1
0
0
4
2
V2
6
4
3
1
1
3
V3
Cost(m) = 2 + 1 + 2 + 1 + 3 + 1
3
1
0
7
V4
Aim
• Accurate MAP estimation of pairwise Markov random fields
Label ‘1’
2
0
1
Label ‘0’
5
V1
2
1
0
0
4
2
V2
6
4
3
1
1
3
3
1
0
V3
Cost(m) = 2 + 1 + 2 + 1 + 3 + 1 + 3
7
V4
Aim
• Accurate MAP estimation of pairwise Markov random fields
Label ‘1’
2
0
1
Label ‘0’
5
V1
2
1
0
0
4
2
V2
6
4
3
1
1
3
3
1
0
7
V3
V4
Cost(m) = 2 + 1 + 2 + 1 + 3 + 1 + 3 = 13
Pr(m)  exp(-Cost(m))
Minimum Cost Labelling = MAP estimate
Aim
• Accurate MAP estimation of pairwise Markov random fields
Label ‘1’
2
0
1
Label ‘0’
5
V1
2
1
0
0
4
2
V2
6
4
3
1
1
3
3
1
0
V3
Objectives
• Applicable for all neighbourhood relationships
• Applicable for all forms of pairwise costs
• Guaranteed to converge
7
V4
Motivation
Subgraph Matching - Torr - 2003, Schellewald et al - 2005
G1
G2
D
C
B
A
D
C
B
A
D
C
B
A
V1
V2
V3
MRF
Unary costs are uniform
Motivation
Subgraph Matching - Torr - 2003, Schellewald et al - 2005
G1
Pairwise Costs
| d(mi,mj) - d(Vi,Vj) | < 
YES
NO
1
2
G2
Potts Model
Motivation
Subgraph Matching - Torr - 2003, Schellewald et al - 2005
D
C
B
A
D
C
B
A
D
C
B
A
V1
V2
V3
MRF
Motivation
Subgraph Matching - Torr - 2003, Schellewald et al - 2005
D
C
B
A
D
C
B
A
D
C
B
A
V1
V2
V3
MRF
Motivation
Matching Pictorial Structures - Felzenszwalb et al - 2001
Outline
Texture
Part likelihood
Spatial Prior
P2
(x,y,,)
P1
Image
P3
MRF
Motivation
Matching Pictorial Structures - Felzenszwalb et al - 2001
• Unary potentials are negative log likelihoods
Valid pairwise configuration
YES
NO
1
2
Potts Model
P2
(x,y,,)
P1
Image
P3
MRF
Motivation
Matching Pictorial Structures - Felzenszwalb et al - 2001
• Unary potentials are negative log likelihoods
Valid pairwise configuration
YES
NO
1
2
Potts Model
P2
(x,y,,)
P1
Image
P3
Pr(Cow)
Outline
• Integer Programming Formulation
• Previous Work
• Our Approach
– Second Order Cone Programming (SOCP)
– SOCP Relaxation
– Robust Truncated Model
• Applications
– Subgraph Matching
– Pictorial Structures
Integer Programming Formulation
Unary Cost
2
Label ‘1’
0
1
Label ‘0’
5
Unary Cost Vector u = [ 5
3
0
V1
Labelling m = {1 , 0}
2
; 2
Cost of
Cost
V1of
=V
01=1
4
4 ]
2
V2
Integer Programming Formulation
Unary Cost
2
Label ‘1’
0
1
Label ‘0’
5
4
3
0
V1
Labelling m = {1 , 0}
2
V2
4 ]T
Unary Cost Vector u = [ 5
2
; 2
Label vector x = [ -1
1
; 1 -1 ]T
Recall that the aimVis1 toV0find
1 = 1the optimal x
Integer Programming Formulation
Unary Cost
2
Label ‘1’
0
1
Label ‘0’
5
4
3
0
V1
Labelling m = {1 , 0}
4 ]T
Unary Cost Vector u = [ 5
2
; 2
Label vector x = [ -1
1
; 1 -1 ]T
Sum of Unary Costs =
1 ∑ u (1 + x )
i i
i
2
2
V2
Integer Programming Formulation
Pairwise Cost
Label ‘1’
2
0
1
Label ‘0’
Labelling m = {1 , 0}
5
V1
4
3
0
2
V2
Pairwise Cost Matrix P
0
0
3
Cost of V1 = 0 and V1 = 0
0 0
1
0
Cost of V1 = 0 and V2 = 0
0 1
0 0
Cost of V1 = 0 and V2 = 1
3 0
0 0
0
Integer Programming Formulation
Pairwise Cost
Label ‘1’
2
0
1
Label ‘0’
Labelling m = {1 , 0}
Pairwise Cost Matrix P
0
0
0
3
0 0
1
0
0 1
0 0
3 0
0 0
5
V1
4
3
0
2
V2
Sum of Pairwise Costs
1 ∑ P (1 + x )(1+x )
ij ij
i
j
4
Integer Programming Formulation
Pairwise Cost
Label ‘1’
2
0
1
Label ‘0’
Labelling m = {1 , 0}
Pairwise Cost Matrix P
0
0
0
3
0 0
1
0
0 1
0 0
3 0
0 0
5
4
3
0
V1
2
V2
Sum of Pairwise Costs
1 ∑ P (1 + x +x + x x )
ij ij
i
j
i j
4
= 1 ∑ij Pij (1 + xi + xj + Xij)
4
X = x xT
Xij = xi xj
Integer Programming Formulation
Constraints
• Each variable should be assigned a unique label
∑ xi = 2 - |L|
i  Va
• Marginalization constraint
∑ Xij = (2 - |L|) xi
j  Vb
Integer Programming Formulation
Chekuri et al. , SODA 2001
x* = argmin
1 ∑ u (1 + x ) + 1 ∑ P (1 + x + x + X )
ij
i
j
ij
i
i
4
2
∑ xi = 2 - |L|
i  Va
∑ Xij = (2 - |L|) xi
j  Vb
Convex
xi {-1,1}
X = x xT
Non-Convex
Outline
• Integer Programming Formulation
• Previous Work
• Our Approach
– Second Order Cone Programming (SOCP)
– SOCP Relaxation
– Robust Truncated Model
• Applications
– Subgraph Matching
– Pictorial Structures
Linear Programming Formulation
Chekuri et al. , SODA 2001
x* = argmin
Retain Convex Part
1 ∑ u (1 + x ) + 1 ∑ P (1 + x + x + X )
ij
i
j
ij
i
i
4
2
∑ xi = 2 - |L|
i  Va
∑ Xij = (2 - |L|) xi
j  Vb
xi {-1,1}
X = x xT
Relax Non-convex
Constraint
Linear Programming Formulation
Chekuri et al. , SODA 2001
x* = argmin
Retain Convex Part
1 ∑ u (1 + x ) + 1 ∑ P (1 + x + x + X )
ij
i
j
ij
i
i
4
2
∑ xi = 2 - |L|
i  Va
∑ Xij = (2 - |L|) xi
j  Vb
xi [-1,1]
X = x xT
Relax Non-convex
Constraint
Linear Programming Formulation
Chekuri et al. , SODA 2001
x* = argmin
Retain Convex Part
1 ∑ u (1 + x ) + 1 ∑ P (1 + x + x + X )
ij
i
j
ij
i
i
4
2
∑ xi = 2 - |L|
i  Va
∑ Xij = (2 - |L|) xi
j  Vb
xi [-1,1]
Linear Programming Formulation
Feasible Region (IP)
x  {-1,1}, X = x2
Linear Programming Formulation
Feasible Region (IP)
Feasible Region (Relaxation 1)
x  {-1,1}, X = x2
x  [-1,1], X = x2
Linear Programming Formulation
Feasible Region (IP)
Feasible Region (Relaxation 1)
Feasible Region (Relaxation 2)
x  {-1,1}, X = x2
x  [-1,1], X = x2
x  [-1,1]
Linear Programming Formulation
• Bounded algorithms proposed by Chekuri et al, SODA 2001
• -expansion - Komodakis and Tziritas, ICCV 2005
• TRW - Wainwright et al., NIPS 2002
• TRW-S - Kolmogorov, AISTATS 2005
• Efficient because it uses Linear Programming
• Not accurate
Semidefinite Programming Formulation
Lovasz and Schrijver, SIAM Optimization, 1990
Retain Convex Part
1 ∑ P (1 + x + x + X )
1
x* = argmin
∑ ui (1 + xi) +
ij
i
j
ij
4
2
∑ xi = 2 - |L|
i  Va
∑ Xij = (2 - |L|) xi
j  Vb
xi {-1,1}
X = x xT
Relax Non-convex
Constraint
Semidefinite Programming Formulation
Lovasz and Schrijver, SIAM Optimization, 1990
Retain Convex Part
1 ∑ P (1 + x + x + X )
1
x* = argmin
∑ ui (1 + xi) +
ij
i
j
ij
4
2
∑ xi = 2 - |L|
i  Va
∑ Xij = (2 - |L|) xi
j  Vb
xi [-1,1]
X = x xT
Relax Non-convex
Constraint
Semidefinite Programming Formulation
1
x1
x2
1 x1 x2
...
xn
..
.
=
xn
1
xT
x
X
Xii = 1
Convex
Non-Convex
Positive Semidefinite
Rank = 1
Semidefinite Programming Formulation
1
x1
x2
1 x1 x2
...
xn
..
.
=
xn
1
xT
x
X
Xii = 1
Convex
Positive Semidefinite
Schur’s Complement
=
I
A
B
BT
C
0
A
BTA-1 I
0
A
0
0
C - BTA-1B
C -BTA-1B
0
I
A-1B
0
I
0
Semidefinite Programming Formulation
=
1
xT
x
X
1
0
1
x
I
0
0
X - xxT
Schur’s Complement
X - xxT
0
I
xT
0
1
Semidefinite Programming Formulation
Lovasz and Schrijver, SIAM Optimization, 1990
Retain Convex Part
1 ∑ P (1 + x + x + X )
1
x* = argmin
∑ ui (1 + xi) +
ij
i
j
ij
4
2
∑ xi = 2 - |L|
i  Va
∑ Xij = (2 - |L|) xi
j  Vb
xi [-1,1]
X = x xT
Relax Non-convex
Constraint
Semidefinite Programming Formulation
Lovasz and Schrijver, SIAM Optimization, 1990
Retain Convex Part
1 ∑ P (1 + x + x + X )
1
x* = argmin
∑ ui (1 + xi) +
ij
i
j
ij
4
2
∑ xi = 2 - |L|
i  Va
∑ Xij = (2 - |L|) xi
j  Vb
xi [-1,1]
Xii = 1
X - xxT
0
Semidefinite Programming Formulation
Feasible Region (IP)
x  {-1,1}, X = x2
Semidefinite Programming Formulation
Feasible Region (IP)
Feasible Region (Relaxation 1)
x  {-1,1}, X = x2
x  [-1,1], X = x2
Semidefinite Programming Formulation
Feasible Region (IP)
Feasible Region (Relaxation 1)
Feasible Region (Relaxation 2)
x  {-1,1}, X = x2
x  [-1,1], X = x2
x  [-1,1], X  x2
Semidefinite Programming Formulation
• Formulated by Lovasz and Schrijver, 1990
• Finds a full X matrix
• Max-cut - Goemans and Williamson, JACM 1995
• Max-k-cut - de Klerk et al, 2000
• Accurate
• Not efficient because of Semidefinite Programming
Previous Work - Overview
LP
Examples
SDP
TRW-S,
Max-k-Cut
-expansion
Accuracy
Low
High
Efficiency
High
Low
Is there a Middle Path ???
Outline
• Integer Programming Formulation
• Previous Work
• Our Approach
– Second Order Cone Programming (SOCP)
– SOCP Relaxation
– Robust Truncated Model
• Applications
– Subgraph Matching
– Pictorial Structures
Second Order Cone Programming
Second Order Cone
x2 + y 2  z2
|| v ||  t OR || v ||2  st
Second Order Cone Programming
Minimize
fTx
Linear Objective Function
Subject to || Ai x+ bi || <= ciT x + di
i = 1, … , L
Affine mapping of Second Order Cone (SOC)
Constraints are SOC of ni dimensions
Feasible regions are intersections of conic regions
Second Order Cone Programming
LP  SOCP  SDP
|| v ||  t
=

1
0
tI
vT
I
0
tI
v
vT
t
0
t2 - vTv
Schur’s Complement
0
I
v
0
1
Outline
• Integer Programming Formulation
• Previous Work
• Our Approach
– Second Order Cone Programming (SOCP)
– SOCP Relaxation
– Robust Truncated Model
• Applications
– Subgraph Matching
– Pictorial Structures
Matrix Dot Product
A
B =
∑ij Aij Bij
A11
A12
B11
B12
A21
A22
B21
B22
= A11 B11 + A12 B12 + A21 B21 + A22 B22
SDP Relaxation
Derive SOCP relaxation from the SDP relaxation
x* = argmin
1 ∑ u (1 + x ) + 1 ∑ P (1 + x + x + X )
ij
i
j
ij
i
i
4
2
∑ xi = 2 - |L|
i  Va
∑ Xij = (2 - |L|) xi
j  Vb
xi [-1,1]
Xii = 1
X-
xxT
0
Further
Relaxation
1-D Example
X - xxT
0
For two semidefinite matrices, the dot product is non-negative
A
X - x2 ≥
0
x2  X
SOC of the form || v ||2  st
A0
SOCP Relaxation
Same as the SDP formulation
Feasible Region (IP)
Feasible Region (Relaxation 1)
Feasible Region (Relaxation 2)
x  {-1,1}, X = x2
x  [-1,1], X = x2
x  [-1,1], X  x2
2-D Example
X
=
xxT =
X11
X12
X21
X22
x1x1
x1x2
x2x1
x2x2
=
=
1
X12
X12
1
x12
x1x2
x1x2
x22
.
C1
2-D Example
(X - xxT)
1
0
0
0
0
.
1 - x12
X12-x1x2
x1  1
-1  x1  1
2
C1
X12-x1x2
1 - x22
0
0
.
C2
2-D Example
(X - xxT)
0
0
0
1
0
.
1 - x12
X12-x1x2
x2  1
-1  x2  1
C2
X12-x1x2
0
0
1 - x22
2
LP Relaxation
.
C3
2-D Example
(X - xxT)
1
1
1
1
C3
0
.
1 - x12
X12-x1x2
X12-x1x2
1 - x22
(x1 + x2  2 + 2X12
2
)
SOC of the form || v ||2  st
0
0
.
C4
2-D Example
(X - xxT)
1
-1
-1
1
C4
0
.
1 - x12
X12-x1x2
X12-x1x2
1 - x22
(x1 - x2  2 - 2X12
2
)
SOC of the form || v ||2  st
0
0
SOCP Relaxation
. .
Kim and Kojima, 2000
Consider a matrix C1 = UUT
C1
0
(X - xxT)  0
||UTx ||2  X
C1
SOC of the form || v ||2  st
Continue for C2, C3, … , Cn
SOCP Relaxation
How many constraints for SOCP = SDP ?
Infinite. For all C
0
We specify constraints similar to the 2-D example
SOCP Relaxation
Muramatsu and Suzuki, 2001
1
0
0
0
0
0
0
1
1
1
1
-1
1
1
-1
1
Constraints hold for the above semidefinite matrices
SOCP Relaxation
Muramatsu and Suzuki, 2001
a
1
0
0
0
+b
0
0
a0
0
1
b0
c0
+c
1
1
1
1
+d
1
-1
-1
1
Constraints hold for the linear combination
d0
SOCP Relaxation
Muramatsu and Suzuki, 2001
a0
a+c+d
c-d
c-d
b0
b+c+d
c0
d0
Includes all semidefinite matrices where
Diagonal elements  Off-diagonal elements
SOCP Relaxation - A
x* = argmin
1 ∑ u (1 + x ) + 1 ∑ P (1 + x + x + X )
ij
i
j
ij
i
i
4
2
∑ xi = 2 - |L|
i  Va
∑ Xij = (2 - |L|) xi
j  Vb
xi [-1,1]
Xii = 1
X - xxT
0
SOCP Relaxation - A
x* = argmin
1 ∑ u (1 + x ) + 1 ∑ P (1 + x + x + X )
ij
i
j
ij
i
i
4
2
∑ xi = 2 - |L|
i  Va
∑ Xij = (2 - |L|) xi
j  Vb
xi [-1,1]
(xi + xj)2  2 + 2Xij
(xi - xj)2  2 - 2Xij
Specified only when Pij  0
Triangular Inequality
• At least two of xi, xj and xk have the same sign
• At least one of Xij, Xjk, Xik is equal to one
Xij + Xjk + Xik 
Xij - Xjk - Xik 
-Xij - Xjk + Xik 
-Xij + Xjk - Xik 
-1
-1
-1
-1
• SOCP-B = SOCP-A + Triangular Inequalities
Outline
• Integer Programming Formulation
• Previous Work
• Our Approach
– Second Order Cone Programming (SOCP)
– SOCP Relaxation
– Robust Truncated Model
• Applications
– Subgraph Matching
– Pictorial Structures
Robust Truncated Model
Pairwise cost of incompatible labels is truncated
Truncated Linear
Model
Potts Model
Truncated Quadratic
Model
• Robust to noise
• Widely used in Computer Vision - Segmentation, Stereo
Robust Truncated Model
Pairwise Cost Matrix can be made sparse
P = [0.5 0.5 0.3 0.3 0.5]
Reparameterization
Sparse Q matrix
 Fewer constraints
Q = [0 0 -0.2 -0.2 0]
Compatibility Constraint
Q(ma, mb) < 0 for variables Va and Vb
Relaxation ∑ Qij (1 + xi + xj + Xij) < 0
SOCP-C = SOCP-B + Compatibility Constraints
SOCP Relaxation
• More accurate than LP
• More efficient than SDP
• Time complexity - O( |V|3 |L|3)
• Same as LP
• Approximate algorithms exist for LP relaxation
• We use |V|  10 and |L|  200
Outline
• Integer Programming Formulation
• Previous Work
• Our Approach
– Second Order Cone Programming (SOCP)
– SOCP Relaxation
– Robust Truncated Model
• Applications
– Subgraph Matching
– Pictorial Structures
Subgraph Matching
Subgraph Matching - Torr - 2003, Schellewald et al - 2005
G1
D
C
B
A
D
C
B
A
G2
D
C
B
A
V1
V2
V3
MRF
Unary costs are uniform
Pairwise costs form a Potts model
Subgraph Matching
• 1000 pairs of graphs G1 and G2
• #vertices in G2 - between 20 and 30
• #vertices in G1 - 0.25 * #vertices in G2
• 5% noise to the position of vertices
• NP-hard problem
Subgraph Matching
Time (sec)
Accuracy (%)
LP
0.85
6.64
SDP-A
35.0
93.11
SOCP-A
3.0
92.01
SOCP-B
4.5
94.79
SOCP-C
4.8
96.18
Method
Outline
• Integer Programming Formulation
• Previous Work
• Our Approach
– Second Order Cone Programming (SOCP)
– SOCP Relaxation
– Robust Truncated Model
• Applications
– Subgraph Matching
– Pictorial Structures
Pictorial Structures
Matching Pictorial Structures - Felzenszwalb et al - 2001
Outline
Texture
P2
(x,y,,)
P1
Image
P3
MRF
Pictorial Structures
Unary costs are negative log likelihoods
Pairwise costs form a Potts model
| V | = 10
| L | = 200
P2
(x,y,,)
P1
Image
P3
MRF
Pictorial Structures
LBP
GBP
SOCP
Pictorial Structures
LBP
GBP
SOCP
Pictorial Structures
LBP
GBP
SOCP
Pictorial Structures
LBP and GBP do not converge
LBP
GBP
SOCP
Pictorial Structures
ROC Curves for 450 +ve and 2400 -ve images
Pictorial Structures
ROC Curves for 450 +ve and 2400 -ve images
Conclusions
• We presented an SOCP relaxation to solve MRF
• More efficient than SDP
• More accurate than LP, LBP, GBP
• #variables can be reduced for Robust Truncated Model
• Provides excellent results for subgraph matching and
pictorial structures
Future Work
• Quality of solution
– Additive bounds exist
– Multiplicative bounds for special cases ??
• Message passing algorithm ??
– Similar to TRW-S or -expansion
– To handle image sized MRF