CVPR 2013 Diversity Tutorial Diverse M-Best Solutions in Markov Random Fields Dhruv Batra Virginia Tech Joint work with: Students: Payman Yadollahpour (TTIC), Abner Guzman-Rivera (UIUC) Colleagues: Chris.
Download ReportTranscript CVPR 2013 Diversity Tutorial Diverse M-Best Solutions in Markov Random Fields Dhruv Batra Virginia Tech Joint work with: Students: Payman Yadollahpour (TTIC), Abner Guzman-Rivera (UIUC) Colleagues: Chris.
CVPR 2013 Diversity Tutorial Diverse M-Best Solutions in Markov Random Fields Dhruv Batra Virginia Tech Joint work with: Students: Payman Yadollahpour (TTIC), Abner Guzman-Rivera (UIUC) Colleagues: Chris Dyer (CMU), Greg Shakhnarovich (TTIC), Pushmeet Kohli (MSRC), Kevin Gimpel (TTIC)
CVPR 2013 Diversity Tutorial (C) Dhruv Batra 2
CVPR 2013 Diversity Tutorial Ambiguity Ambiguity Ambiguity (C) Dhruv Batra ?
?
One instance / Two instances?
3
CVPR 2013 Diversity Tutorial Problems with MAP Single Prediction = Uncertainty Mismanagement MAP is NP-Hard Make Multiple Predictions!
(C) Dhruv Batra 4
CVPR 2013 Diversity Tutorial Multiple Predictions x x xx x x x Sampling Porway & Zhu, 2011 TU & Zhu, 2002 Rich History (C) Dhruv Batra 5
CVPR 2013 Diversity Tutorial Multiple Predictions Sampling Porway & Zhu, 2011 TU & Zhu, 2002 Rich History (C) Dhruv Batra M-Best MAP Flerova et al., 2011 Fromer et al., 2009 Yanover et al., 2003 Ideally: M-Best Modes ✓ 6
CVPR 2013 Diversity Tutorial Multiple Predictions Sampling Porway & Zhu, 2011 TU & Zhu, 2002 Our work: Diverse M Best in MRFs [ECCV ‘12] M-Best MAP Don’t
hope
for diversity. Explicitly encode it.
Flerova et al., 2011 Ideally: M-Best Modes ✓ Rich History Yanover et al., 2003 (C) Dhruv Batra 7
CVPR 2013 Diversity Tutorial Example Result (C) Dhruv Batra 8
CVPR 2013 Diversity Tutorial Example Result (C) Dhruv Batra Discriminative Re-ranking of Diverse Segmentation [Yadollahpour et al., CVPR13, Wednesday Poster] 9
CVPR 2013 Diversity Tutorial MAP Integer Program kx1 (C) Dhruv Batra 10
CVPR 2013 Diversity Tutorial MAP Integer Program 1 0 0 0 kx1 (C) Dhruv Batra 11
CVPR 2013 Diversity Tutorial MAP Integer Program 0 1 0 0 kx1 (C) Dhruv Batra 12
CVPR 2013 Diversity Tutorial MAP Integer Program 0 0 1 0 kx1 (C) Dhruv Batra 13
CVPR 2013 Diversity Tutorial MAP Integer Program 0 0 0 1 kx1 (C) Dhruv Batra 14
CVPR 2013 Diversity Tutorial MAP Integer Program 0 0 0 1 kx1 k 2 x1 (C) Dhruv Batra 15
CVPR 2013 Diversity Tutorial MAP Integer Program 0 0 0 1 kx1 k 2 x1 (C) Dhruv Batra 16
CVPR 2013 Diversity Tutorial MAP Integer Program Graphcuts, BP, Expansion, etc (C) Dhruv Batra 17
CVPR 2013 Diversity Tutorial Diverse 2 nd -Best Diversity MAP (C) Dhruv Batra 18
CVPR 2013 Diversity Tutorial Diverse M-Best (C) Dhruv Batra 19
CVPR 2013 Diversity Tutorial Diverse 2 nd -Best (C) Dhruv Batra Q1: How do we solve DivMBest?
Q2: What kind of diversity functions are allowed?
Q3: How much diversity?
20
CVPR 2013 Diversity Tutorial Diverse 2 nd -Best Diversity-Augmented Score Primal Dualize (C) Dhruv Batra 21
CVPR 2013 Diversity Tutorial Diverse 2 nd -Best • Lagrangian Relaxation Diversity-Augmented Score Dual Subgradient Descent Concave (Non-smooth) Upper-Bound on Div2Best Score Div2Best score 22 (C) Dhruv Batra
CVPR 2013 Diversity Tutorial Diverse 2 nd -Best • Lagrangian Relaxation Diversity-Augmented Energy 1.
Subgradient Ascent. 2. Binary Search. 3. Grid-search on lambda. Optimal. Slow.
Optimal for M=2. Faster.
(C) Dhruv Batra 23
CVPR 2013 Diversity Tutorial Theorem Statement • Theorem [Batra et al ’12]: Lagrangian Dual corresponds to solving the Relaxed Primal: • Based on result from [Geoffrion ‘74] Dual Relaxed Primal (C) Dhruv Batra 24
CVPR 2013 Diversity Tutorial Effect of Lagrangian Relaxation (C) Dhruv Batra 25
CVPR 2013 Diversity Tutorial Effect of Lagrangian Relaxation (C) Dhruv Batra 26
CVPR 2013 Diversity Tutorial Effect of Lagrangian Relaxation • [Mezuman et al. UAI13] Pairwise Potential Strength (C) Dhruv Batra Pairwise Potential Strength 27
CVPR 2013 Diversity Tutorial Diverse 2 nd -Best (C) Dhruv Batra Q1: How do we solve DivMBest?
Q2: What kind of diversity functions are allowed?
Q3: How much diversity?
28
CVPR 2013 Diversity Tutorial Diversity • [Special Case] 0-1 Diversity M-Best MAP – [Yanover NIPS03; Fromer NIPS09; Flerova Soft11] • [Special Case] Max Diversity [Park & Ramanan ICCV11] • Hamming Diversity • Cardinality Diversity • Any Diversity (C) Dhruv Batra 29
CVPR 2013 Diversity Tutorial Hamming Diversity 0 1 0 0 0 1 0 0 1 0 0 0 0 1 0 0 (C) Dhruv Batra 30
CVPR 2013 Diversity Tutorial Hamming Diversity • Diversity Augmented Inference: (C) Dhruv Batra 31
CVPR 2013 Diversity Tutorial Hamming Diversity • Diversity Augmented Inference: Unchanged. Can still use graph-cuts!
(C) Dhruv Batra Simply edit node-terms. Reuse MAP machinery!
32
CVPR 2013 Diversity Tutorial Diverse 2 nd -Best (C) Dhruv Batra Q1: How do we solve DivMBest?
Q2: What kind of diversity functions are allowed?
Q3: How much diversity?
33
CVPR 2013 Diversity Tutorial How Much Diversity?
• Empirical Solution: Cross-Val for • More Efficient: Cross-Val for (C) Dhruv Batra 34
CVPR 2013 Diversity Tutorial Experiments • 3 Applications – Interactive Segmentation: Hamming, Cardinality (in paper) – Pose Estimation: Hamming – Semantic Segmentation: Hamming • Baselines: – M-Best MAP (No Diversity) – Confidence-Based Perturbation (No Optimization) (C) Dhruv Batra 35
CVPR 2013 Diversity Tutorial Interactive Segmentation • Setup – Model: Color/Texture + Potts Grid CRF – Inference: Graph-cuts – Dataset: 50 train/val/test images Image + Scribbles MAP 2 nd Best MAP Diverse 2 nd Best 1-2 Nodes Flipped 100-500 Nodes Flipped 36 (C) Dhruv Batra
CVPR 2013 Diversity Tutorial Pose Tracking • Setup – Model: Mixture of Parts from [Park & Ramanan, ICCV ‘11] – Inference: Dynamic Programming – Dataset: 4 videos, 585 frames (C) Dhruv Batra Image Credit: [Yang & Ramanan , ICCV ‘11] 37
CVPR 2013 Diversity Tutorial (C) Dhruv Batra 38
CVPR 2013 Diversity Tutorial Pose Tracking • Chain CRF with M states at each time M Best Solutions (C) Dhruv Batra Image Credit: [Yang & Ramanan , ICCV ‘11] 39
CVPR 2013 Diversity Tutorial Pose Tracking (C) Dhruv Batra MAP DivMBest + Viterbi 40
CVPR 2013 Diversity Tutorial Better 85% 80% 75% 70% 65% 60% 55% 50% 45% 1 (C) Dhruv Batra Pose Tracking DivMBest (Re-ranked) 13% Gain Same Features [Park & Ramanan , ICCV ‘11] (Re-ranked) Confidence-based Perturbation (Re-ranked) 51 101 151 201 #Solutions / Frame 251 301 41
CVPR 2013 Diversity Tutorial Machine Translation
Input:
Die Regierung will die Folter Broschüre heraus von “Hexen” unterbinden und gab eine
MAP Translation:
The government wants the torture of ‘witch’ and gave out a booklet (C) Dhruv Batra 42
CVPR 2013 Diversity Tutorial Machine Translation
Input:
Die Regierung will die Folter Broschüre heraus von “Hexen” unterbinden und gab eine
5-Best Translations:
The government wants the torture of ‘witch’ and gave out a booklet The government wants the torture of “witch” and gave out a booklet The government wants the torture of ‘witch’ and gave out a brochure The government wants the torture of ‘witch’ and gave out a leaflet The government wants the torture of “witch” and gave out a brochure (C) Dhruv Batra 43
CVPR 2013 Diversity Tutorial Machine Translation
Input:
Die Regierung will die Folter Broschüre heraus von “Hexen” unterbinden und gab eine
Diverse 5-Best Translations:
The government wants the torture of ‘witch’ and gave out a booklet The government wants to stop torture of “witch” and issued a leaflet issued The government wants to “stop the torture of” witches and gave out a brochure The government intends to the torture of “witchcraft” and were issued a leaflet The government is the torture of “witches” stamp out and gave a brochure (C) Dhruv Batra 44
CVPR 2013 Diversity Tutorial Machine Translation
Input:
Die Regierung will die Folter Broschüre heraus von “Hexen” unterbinden und gab eine
Diverse 5-Best Translations:
The government wants the torture of ‘witch’ and gave out a booklet The government wants to stop torture of “witch” and issued a leaflet issued The government wants to “stop the torture of” witches and gave out a brochure The government intends to the torture of “witchcraft” and were issued a leaflet The government is the torture of “witches” stamp out and gave a brochure
Correct Translation:
The government wants to limit the torture of “witches,” a brochure was released 45