CVPR 2013 Diversity Tutorial Diverse M-Best Solutions in Markov Random Fields Dhruv Batra Virginia Tech Joint work with: Students: Payman Yadollahpour (TTIC), Abner Guzman-Rivera (UIUC) Colleagues: Chris Dyer (CMU), Greg Shakhnarovich (TTIC), Pushmeet Kohli (MSRC), Kevin Gimpel (TTIC)

CVPR 2013 Diversity Tutorial (C) Dhruv Batra 2

CVPR 2013 Diversity Tutorial Ambiguity Ambiguity Ambiguity (C) Dhruv Batra ?

?

One instance / Two instances?

3

CVPR 2013 Diversity Tutorial Problems with MAP Single Prediction = Uncertainty Mismanagement MAP is NP-Hard Make Multiple Predictions!

(C) Dhruv Batra 4

CVPR 2013 Diversity Tutorial Multiple Predictions x x xx x x x Sampling Porway & Zhu, 2011 TU & Zhu, 2002 Rich History (C) Dhruv Batra 5

CVPR 2013 Diversity Tutorial Multiple Predictions Sampling Porway & Zhu, 2011 TU & Zhu, 2002 Rich History (C) Dhruv Batra M-Best MAP Flerova et al., 2011 Fromer et al., 2009 Yanover et al., 2003 Ideally: M-Best Modes ✓ 6

CVPR 2013 Diversity Tutorial Multiple Predictions Sampling Porway & Zhu, 2011 TU & Zhu, 2002 Our work: Diverse M Best in MRFs [ECCV ‘12] M-Best MAP Don’t

hope

for diversity. Explicitly encode it.

Flerova et al., 2011 Ideally: M-Best Modes ✓ Rich History Yanover et al., 2003 (C) Dhruv Batra 7

CVPR 2013 Diversity Tutorial Example Result (C) Dhruv Batra 8

CVPR 2013 Diversity Tutorial Example Result (C) Dhruv Batra Discriminative Re-ranking of Diverse Segmentation [Yadollahpour et al., CVPR13, Wednesday Poster] 9

CVPR 2013 Diversity Tutorial MAP Integer Program kx1 (C) Dhruv Batra 10

CVPR 2013 Diversity Tutorial MAP Integer Program 1 0 0 0 kx1 (C) Dhruv Batra 11

CVPR 2013 Diversity Tutorial MAP Integer Program 0 1 0 0 kx1 (C) Dhruv Batra 12

CVPR 2013 Diversity Tutorial MAP Integer Program 0 0 1 0 kx1 (C) Dhruv Batra 13

CVPR 2013 Diversity Tutorial MAP Integer Program 0 0 0 1 kx1 (C) Dhruv Batra 14

CVPR 2013 Diversity Tutorial MAP Integer Program 0 0 0 1 kx1 k 2 x1 (C) Dhruv Batra 15

CVPR 2013 Diversity Tutorial MAP Integer Program 0 0 0 1 kx1 k 2 x1 (C) Dhruv Batra 16

CVPR 2013 Diversity Tutorial MAP Integer Program Graphcuts, BP, Expansion, etc (C) Dhruv Batra 17

CVPR 2013 Diversity Tutorial Diverse 2 nd -Best Diversity MAP (C) Dhruv Batra 18

CVPR 2013 Diversity Tutorial Diverse M-Best (C) Dhruv Batra 19

CVPR 2013 Diversity Tutorial Diverse 2 nd -Best (C) Dhruv Batra Q1: How do we solve DivMBest?

Q2: What kind of diversity functions are allowed?

Q3: How much diversity?

20

CVPR 2013 Diversity Tutorial Diverse 2 nd -Best Diversity-Augmented Score Primal Dualize (C) Dhruv Batra 21

CVPR 2013 Diversity Tutorial Diverse 2 nd -Best • Lagrangian Relaxation Diversity-Augmented Score Dual Subgradient Descent Concave (Non-smooth) Upper-Bound on Div2Best Score Div2Best score 22 (C) Dhruv Batra

CVPR 2013 Diversity Tutorial Diverse 2 nd -Best • Lagrangian Relaxation Diversity-Augmented Energy 1.

Subgradient Ascent. 2. Binary Search. 3. Grid-search on lambda. Optimal. Slow.

Optimal for M=2. Faster.

(C) Dhruv Batra 23

CVPR 2013 Diversity Tutorial Theorem Statement • Theorem [Batra et al ’12]: Lagrangian Dual corresponds to solving the Relaxed Primal: • Based on result from [Geoffrion ‘74] Dual Relaxed Primal (C) Dhruv Batra 24

CVPR 2013 Diversity Tutorial Effect of Lagrangian Relaxation (C) Dhruv Batra 25

CVPR 2013 Diversity Tutorial Effect of Lagrangian Relaxation (C) Dhruv Batra 26

CVPR 2013 Diversity Tutorial Effect of Lagrangian Relaxation • [Mezuman et al. UAI13] Pairwise Potential Strength (C) Dhruv Batra Pairwise Potential Strength 27

CVPR 2013 Diversity Tutorial Diverse 2 nd -Best (C) Dhruv Batra Q1: How do we solve DivMBest?

Q2: What kind of diversity functions are allowed?

Q3: How much diversity?

28

CVPR 2013 Diversity Tutorial Diversity • [Special Case] 0-1 Diversity M-Best MAP – [Yanover NIPS03; Fromer NIPS09; Flerova Soft11] • [Special Case] Max Diversity [Park & Ramanan ICCV11] • Hamming Diversity • Cardinality Diversity • Any Diversity (C) Dhruv Batra 29

CVPR 2013 Diversity Tutorial Hamming Diversity 0 1 0 0 0 1 0 0 1 0 0 0 0 1 0 0 (C) Dhruv Batra 30

CVPR 2013 Diversity Tutorial Hamming Diversity • Diversity Augmented Inference: (C) Dhruv Batra 31

CVPR 2013 Diversity Tutorial Hamming Diversity • Diversity Augmented Inference: Unchanged. Can still use graph-cuts!

(C) Dhruv Batra Simply edit node-terms. Reuse MAP machinery!

32

CVPR 2013 Diversity Tutorial Diverse 2 nd -Best (C) Dhruv Batra Q1: How do we solve DivMBest?

Q2: What kind of diversity functions are allowed?

Q3: How much diversity?

33

CVPR 2013 Diversity Tutorial How Much Diversity?

• Empirical Solution: Cross-Val for • More Efficient: Cross-Val for (C) Dhruv Batra 34

CVPR 2013 Diversity Tutorial Experiments • 3 Applications – Interactive Segmentation: Hamming, Cardinality (in paper) – Pose Estimation: Hamming – Semantic Segmentation: Hamming • Baselines: – M-Best MAP (No Diversity) – Confidence-Based Perturbation (No Optimization) (C) Dhruv Batra 35

CVPR 2013 Diversity Tutorial Interactive Segmentation • Setup – Model: Color/Texture + Potts Grid CRF – Inference: Graph-cuts – Dataset: 50 train/val/test images Image + Scribbles MAP 2 nd Best MAP Diverse 2 nd Best 1-2 Nodes Flipped 100-500 Nodes Flipped 36 (C) Dhruv Batra

CVPR 2013 Diversity Tutorial Pose Tracking • Setup – Model: Mixture of Parts from [Park & Ramanan, ICCV ‘11] – Inference: Dynamic Programming – Dataset: 4 videos, 585 frames (C) Dhruv Batra Image Credit: [Yang & Ramanan , ICCV ‘11] 37

CVPR 2013 Diversity Tutorial (C) Dhruv Batra 38

CVPR 2013 Diversity Tutorial Pose Tracking • Chain CRF with M states at each time M Best Solutions (C) Dhruv Batra Image Credit: [Yang & Ramanan , ICCV ‘11] 39

CVPR 2013 Diversity Tutorial Pose Tracking (C) Dhruv Batra MAP DivMBest + Viterbi 40

CVPR 2013 Diversity Tutorial Better 85% 80% 75% 70% 65% 60% 55% 50% 45% 1 (C) Dhruv Batra Pose Tracking DivMBest (Re-ranked) 13% Gain Same Features [Park & Ramanan , ICCV ‘11] (Re-ranked) Confidence-based Perturbation (Re-ranked) 51 101 151 201 #Solutions / Frame 251 301 41

CVPR 2013 Diversity Tutorial Machine Translation

Input:

Die Regierung will die Folter Broschüre heraus von “Hexen” unterbinden und gab eine

MAP Translation:

The government wants the torture of ‘witch’ and gave out a booklet (C) Dhruv Batra 42

CVPR 2013 Diversity Tutorial Machine Translation

Input:

Die Regierung will die Folter Broschüre heraus von “Hexen” unterbinden und gab eine

5-Best Translations:

The government wants the torture of ‘witch’ and gave out a booklet The government wants the torture of “witch” and gave out a booklet The government wants the torture of ‘witch’ and gave out a brochure The government wants the torture of ‘witch’ and gave out a leaflet The government wants the torture of “witch” and gave out a brochure (C) Dhruv Batra 43

CVPR 2013 Diversity Tutorial Machine Translation

Input:

Die Regierung will die Folter Broschüre heraus von “Hexen” unterbinden und gab eine

Diverse 5-Best Translations:

The government wants the torture of ‘witch’ and gave out a booklet The government wants to stop torture of “witch” and issued a leaflet issued The government wants to “stop the torture of” witches and gave out a brochure The government intends to the torture of “witchcraft” and were issued a leaflet The government is the torture of “witches” stamp out and gave a brochure (C) Dhruv Batra 44

CVPR 2013 Diversity Tutorial Machine Translation

Input:

Die Regierung will die Folter Broschüre heraus von “Hexen” unterbinden und gab eine

Diverse 5-Best Translations:

The government wants the torture of ‘witch’ and gave out a booklet The government wants to stop torture of “witch” and issued a leaflet issued The government wants to “stop the torture of” witches and gave out a brochure The government intends to the torture of “witchcraft” and were issued a leaflet The government is the torture of “witches” stamp out and gave a brochure

Correct Translation:

The government wants to limit the torture of “witches,” a brochure was released 45