Transcript Diverse M-Best Solutions in Markov Random Fields , , , Dhruv Batra Payman Yadollahpour Abner Guzman-Rivera Greg Shakhnarovich TTI-Chicago / Virginia Tech TTI-Chicago UIUC TTI-Chicago.

Diverse M-Best Solutions
in Markov Random Fields
,
,
,
Dhruv Batra
Payman Yadollahpour
Abner Guzman-Rivera
Greg Shakhnarovich
TTI-Chicago /
Virginia Tech
TTI-Chicago
UIUC
TTI-Chicago
Local Ambiguity
• Graphical Models
Hat
x1
x2
MAP
…
Inference
xn
Cat
Most Likely Assignment
(C) Dhruv Batra
2
Problems with MAP
Model-Class is Wrong!
-- Approximation Error
Human Body ≠ Tree
(C) Dhruv Batra
Figure Courtesy: [Yang & Ramanan ICCV ‘11]
3
Problems with MAP
Model-Class
is Wrong!
Not
Enough Training
Data!
-- Approximation Error
-- Estimation Error
(C) Dhruv Batra
4
Problems with MAP
Model-Class
is Wrong!
Not
Enough
Training
Data!
MAP
is
NP-Hard
-- Approximation Error
-- Estimation Error
-- Optimization Error
(C) Dhruv Batra
5
Problems with MAP
Model-Class
is Wrong!
Not
Enough
Training
Data!
MAP
is
NP-Hard
--Inherent
ApproximationAmbiguity
Error
-- Estimation Error
-- Optimization Error
-- Bayes Error
?
?
Rotating clockwise /
Old Lady looking left /
anti-clockwise? Young woman looking away?
(C) Dhruv Batra
One instance /
Two instances?
6
Problems with MAP
Model-Class
is
Wrong!
Single
Prediction
= Uncertainty
Mismanagement
Not
Enough
Training
Data!
MAP
is NP-Hard
--Inherent
Approximation
Error
Ambiguity
-- Estimation Error
-- Optimization Error
Make
Predictions!
-- BayesMultiple
Error
(C) Dhruv Batra
7
Multiple Predictions
xxx
x
xx
xxx
xxxx
Sampling
Porway & Zhu, 2011
TU & Zhu, 2002
Rich History
(C) Dhruv Batra
8
Multiple Predictions
Sampling
M-Best MAP
Porway & Zhu, 2011
TU & Zhu, 2002
Rich History
Flerova et al., 2011
Fromer et al., 2009
Yanover et al., 2003
(C) Dhruv Batra
Ideally:
✓
M-Best Modes
9
Multiple Predictions
This Paper: Diverse M-Best in MRFs
Sampling
M-Best MAP
-
Porway & Zhu, 2011
TU & Zhu, 2002
Rich History
(C) Dhruv Batra
Don’t hope for diversity. Explicitly encode it. Ideally:
Flerova et al., 2011
Fromer et
Not guaranteed
toal.,
be2009
modes.
Yanover et al., 2003
✓
M-Best Modes
10
MAP Integer Program
kx1
(C) Dhruv Batra
11
MAP Integer Program
1
0
0
0
kx1
(C) Dhruv Batra
12
MAP Integer Program
0
1
0
0
kx1
(C) Dhruv Batra
13
MAP Integer Program
0
0
1
0
kx1
(C) Dhruv Batra
14
MAP Integer Program
0
0
0
1
kx1
(C) Dhruv Batra
15
MAP Integer Program
0
0
0
1
kx1
k2x1
(C) Dhruv Batra
16
MAP Integer Program
0
0
0
1
kx1
k2x1
(C) Dhruv Batra
17
MAP Integer Program
Graphcuts, BP, Expansion, etc
(C) Dhruv Batra
18
Diverse 2nd-Best
Diversity
MAP
(C) Dhruv Batra
19
Diverse M-Best
(C) Dhruv Batra
20
Diverse 2nd-Best
Q1: How do we solve DivMBest?
Q2: What kind of diversity functions are allowed?
Q3: How much diversity?
See Paper for Details
(C) Dhruv Batra
21
Diverse 2nd-Best
Diversity-Augmented Energy
• Lagrangian Relaxation
Many ways to solve:
Primal 1.
upergradient Ascent.
Optimal. Slow.
See Paper for Details
2. Binary Search.
Optimal for M=2. Faster.
3. Grid-search on lambda.
Sub-optimal. Fastest.Dualize
Dual
Div2Best energy
Concave (Non-smooth)
Lower-Bound on Div2Best En.
(C) Dhruv Batra
22
Diverse 2nd-Best
Q1: How do we solve Div2Best?
Q2: What kind of diversity functions are allowed?
Q3: How much diversity?
See Paper for Details
(C) Dhruv Batra
23
Diversity
• [Special Case] 0-1 Diversity
–
M-Best MAP
[Yanover NIPS03; Fromer NIPS09; Flerova Soft11]
• [Special Case] Max Diversity
[Park & Ramanan ICCV11]
• Hamming Diversity
• Cardinality Diversity
See Paper for Details
• Any Diversity
(C) Dhruv Batra
24
Hamming Diversity
0
0
(C) Dhruv Batra
1
0
0
1
0
1 0
0
0
1
0
0
0
0
25
Hamming Diversity
• Diversity Augmented Inference:
(C) Dhruv Batra
26
Hamming Diversity
• Diversity Augmented Inference:
Unchanged.
Can still use graph-cuts!
Simply edit node-terms. Reuse MAP machinery!
(C) Dhruv Batra
27
Experiments
• 3 Applications
– Interactive Segmentation: Hamming, Cardinality (in paper)
– Pose Estimation: Hamming
– Semantic Segmentation: Hamming
• Baselines:
– M-Best MAP
– Confidence-Based Perturbation
(No Diversity)
(No Optimization)
• Metrics
– Oracle Accuracies
• User-in-the-loop; Upper-Bound
– Re-ranked Accuracies
(C) Dhruv Batra
28
Experiment #1
• Interactive Segmentation
– Model: Color/Texture + Potts Grid CRF
– Inference: Graph-cuts
– Dataset: 50 train/val/test images
Image + Scribbles
MAP
2nd Best MAP
1-2 Nodes Flipped
(C) Dhruv Batra
Diverse 2nd Best
100-500 Nodes Flipped
29
Experiment #1
96%
+3.62%
95%
94%
+1.61%
93%
+0.05%
92%
91%
90%
89%
MAP
M-Best-MAP
(Oracle)
Confidence
(Oracle)
DivMBest
(Oracle)
M=6
(C) Dhruv Batra
30
Experiment #2
• Pose Tracking
– Model: Mixture of Parts from [Park & Ramanan, ICCV ‘11]
– Inference: Dynamic Programming
– Dataset: 4 videos, 585 frames
(C) Dhruv Batra
Image Credit: [Yang & Ramanan, ICCV ‘11]
31
Experiment #2
• Pose Tracking w/ Chain CRF
M Best
Solutions
(C) Dhruv Batra
Image Credit: [Yang & Ramanan, ICCV ‘11]
32
Experiment #2
MAP
(C) Dhruv Batra
DivMBest + Viterbi
33
Experiment #2
85%
Better
DivMBest (Re-ranked)
80%
PCP Accuracy
13% Gain
75%
Same Features
Same Model
[Park & Ramanan, ICCV ‘11] (Re-ranked)
70%
65%
60%
Confidence-based Perturbation (Re-ranked)
55%
50%
45%
1
51
101
151
201
251
301
#Solutions / Frame
(C) Dhruv Batra
34
Experiment #3
• Semantic Segmentation
– Model: Hierarchical CRF [Ladicky et al. ECCV ’10, ICCV ‘09]
– Inference: Alpha-expansion
– Dataset: Pascal Segmentation Challenge (VOC 2010)
• 20 categories + background; 964 train/val/test images
(C) Dhruv Batra
Image Credit: [Ladicky et al. ECCV ’10, ICCV ’09]
35
Experiment #3
Input
(C) Dhruv Batra
MAP
Best of 10-Div
36
Experiment #3
50%
DivMBest (Oracle)
Better
PACAL Accuracy
45%
22%-gain possible
40%
Same Features
Same Model
35%
DivMBest (Re-ranked) [Yadollahpour et al.]
30%
Confidence-based Perturbation (Oracle)
MAP
25%
20%
1
3
5
7
9
11
13
15
17
19
21
23
25
27
29
31
#Solutions / Image
(C) Dhruv Batra
37
Summary
• All models are wrong
• Some beliefs are useful
• DivMBest
– First principled formulation for Diverse M-Best in MRFs
– Efficient algorithm. Re-uses MAP machinery.
– Big impact possible on many applications!
(C) Dhruv Batra
38
Thank you!
• Think about YOUR problem.
• Are you or a loved one, tired of a single solution?
• If yes, then DivMBest might be right for you!*
* DivMBest is not suited for everyone. People with perfect models, and love of continuous
variables should not use DivMBest. Consult your local optimization expert before starting
DivMBest. Please do not drive or operate heavy machinery while on DivMBest.
(C) Dhruv Batra
39