Transcript Parallel Double Greedy Submodular Maxmization Xinghao Pan, Stefanie Jegelka, Joseph Gonzalez, Joseph Bradley, Michael I.

Parallel Double Greedy
Submodular Maxmization
Xinghao Pan, Stefanie Jegelka, Joseph Gonzalez, Joseph Bradley, Michael
I. Jordan
Submodular Set Functions
Diminishing Returns Property
F: 2V → R, such that for all
and
Submodular Examples
Sensing
(Krause & Guestrin 2005)
Document
Summarization
Network Analysis
(Kempe, Kleinberg, Tardos 2003,
Mossel & Roch 2007)
Graph Algorithms
S
clustering
graph partitioning
Max / Min
Cut
(Lin & Bilmes 2011)
Submodular Maximization
Parallel /
Distributed
Sequential
max F(A), A ⊆ V
Monotone (increasing) functions
[ Positive marginal gains ]
Non-monotone functions
Greedy (Nemhauser et al, 1978)
(1-1/e) - approximation
Optimal polytime
Double Greedy (Buchbinder et al, 2012)
½ - approximation
Optimal polytime
GREEDI (Mirzasoleiman et al, 2013)
(1-1/e)2 / p – approximation
1 MapReduce round
Concurrency Control Double Greedy
Optimal ½ - approximation
Bounded overhead
(Kumar et al, 2013)
1 / (2+ε) – approximation
O(1/ε) MapReduce rounds
Coordination Free Double Greedy
Bounded error
Minimal overhead
?
Double Greedy Algorithm
u
Set
A
Set
B
u
v
w
x
y
z
v
w
x
y
z
Double Greedy Algorithm
u
Set
A
Set
B
u
v
w
x
y
z
v
w
x
y
z
Double Greedy Algorithm
v
Set
A
Set
B
u
v
w
x
y
z
u
w
x
y
z
Marginal gains
Δ+(u|A) = F(A ∪ u) – F(A),
Δ-(u|B) = F(B \ u) – F(B).
p( u | A, B ) = + A
-B
0
1
rand
Double Greedy Algorithm
v
Set
A
Set
B
u
v
w
x
y
z
u
w
x
y
z
Marginal gains
Δ+(u|A) = F(A ∪ u) – F(A),
Δ-(u|B) = F(B \ u) – F(B).
p( u | A, B ) = + A
-B
0
1
rand
Double Greedy Algorithm
v
Set
A
u
Set
B
u
v
w
x
y
z
v
w
x
y
z
Marginal gains
Δ+(v|A) = F(A ∪ v) – F(A),
Δ-(v|B) = F(B \ v) – F(B).
p( v | A, B ) = + A
-B
0
1
rand
Double Greedy Algorithm
w
Set
A
u
Set
B
u
w
x
y
z
Return A
x
y
z
Parallel Double Greedy
Algorithm
u
Set
A
Set
B
v
CPU 1
u
v
w
x
y
z
CPU 2
w
x
y
z
Parallel Double Greedy
Algorithm
Set
A
Set
B
?
?
u
v
CPU 1
u
w
y
Δ+(u | ?) = ?
Δ-(u | ?) = ?
u
v
w
x
y
z
CPU 2
v
x
z
Δ+(v | ?) = ?
Δ-(v | ?) = ?
Concurrency Control Double
Greedy
Maintain bounds on A, B  Enable threads to make decisions locally
Set
A
Set
B
CPU 1
u
w
y
v
x
z
u
u
v
w
x
CPU 2
v
y
z
Concurrency Control Double
Greedy
Maintain bounds on A, B  Enable threads to make decisions locally
Set
A
u
v
Set
B
CPU 1
u
u
w
y
Δ+(u|A) ∈ [Δ+min(u|A), Δ+max(u|A)]
Δ-(u|B) ∈ [Δ-min(u|B), Δ-max(u|B) ]
p( u | A, B ) = + A
v
Uncertaint
y
-B
0
1
w
x
CPU 2
v
y
z
x
z
Δ+(v|A) ∈ [Δ+min(v|A), Δ+max(v|A)]
Δ-(v|B) ∈ [Δ-min(v|B), Δ-max(v|B) ]
p( v | A, B ) = + A
0
Uncertaint
y
-B
1
Concurrency Control Double
Greedy
Maintain bounds on A, B  Enable threads to make decisions locally
Set
A
u
v
Set
B
CPU 1
u
u
w
y
Δ+(u|A) ∈ [Δ+min(u|A), Δ+max(u|A)]
Δ-(u|B) ∈ [Δ-min(u|B), Δ-max(u|B) ]
Uncertaint
y
p( u | A, B ) = + A
v
-B
0
1
rand
w
x
CPU 2
v
y
z
x
z
Δ+(v|A) ∈ [Δ+min(v|A), Δ+max(v|A)]
Δ-(v|B) ∈ [Δ-min(v|B), Δ-max(v|B) ]
p( v | A, B ) = + A
0
Uncertaint
y
-B
1
Concurrency Control Double
Greedy
Maintain bounds on A, B  Enable threads to make decisions locally
Set
A
u
v
Set
B
CPU 1
w
y
Δ+(u|A) ∈ [Δ+min(u|A), Δ+max(u|A)]
Δ-(u|B) ∈ [Δ-min(u|B), Δ-max(u|B) ]
u
Uncertaint
y
p( u | A, B ) = + A
v
Common,
Fast
-B
0
1
rand
w
x
CPU 2
v
y
z
x
Rare,
Slow
z
Δ+(v|A) =
∈ F(A
[Δ+min
∪(v|A),
v) – F(A)
Δ+max(v|A)]
Δ-(v|B) =
∈ F(B
[Δ-min(v|B),
\ v) – F(B)
Δ-max(v|B) ]
p( v | A, B ) = + A
Uncertaint
y
0
-B
1
rand
Theorem: CC double greedy is serializable.
Corollary: CC double greedy preserves optimal
approximation guarantee of ½OPT.
Lemma: CC has bounded overhead.
Expected number of blocked elements
Concurren
cy
Correctness
Properties of CC Double
Greedy
set cover with costs:
< 2τ
sparse max cut:
< 2τ |E| / |V|
τ
Set A
Set B
u
u
w
w
x
x
D
y
y
E
z
Empirical Validation
Multicore up to 16 threads
Set cover, Max graph cut
Real and synthetic graphs
Graph
Vertices
Edges
IT-2004
Italian web-graph
41 Million
1.1 Billion
UK-2005
UK web-graph
39 Million
0.9 Billion
Arabic-2005
Arabic web-graph
22 Million
0.6 Billion
Friendster
Social sub-network
10 Million
0.6 Billion
Erdos-Renyi
Synthetic random
20 Million
2.0 Billion
ZigZag
Synthetic expander
25 Million
2.0 Billion
CC Double Greedy
Coordination
Only 0.01%
needs
blocking!
% elements failed + blocked
Increase in Coordination
Runtime and Strong-Scaling
Speedup for Max Graph Cut
3
15
2.5
2
Concurrency Ctrl.
Sequential
1.5
1
Speedup
Runtime relative to sequential
Runtime, relative to sequential
10
5
0.5
0
0
5
10
# threads
15
0
0
5
10
# threads
15
Conclusion
Scalability
Approximation
Sequential
Double Greedy
Always slow
Always optimal
Concurrency
Control
Double Greedy
Usually fast
Always optimal
Coordination Free
Double Greedy
Always fast
Near optimal
Paper @ NIPS 2014:
Parallel Double Greedy Submodular Maximization.
BACKUP SLIDES
Coordination Free
Theorem: serializable.
preserves optimal
approximation bound
½OPT.
τ
Lemma:
approximation
bound ½OPT error
Set A
Set B
u
u
w
w
x
x
D
y
y
E
set cover with costs:
sparse maxcut:
Correctness
Correctness
Concurrency Control
Lemma:
bounded overhead.
set cover with costs:
sparse maxcut:
from same
dependencies
(uncertainty region)
no
overhead
Concurren
Concurren
cy
z
Adversarial Setting
Processing order
Overlapping covers
 Increased coordination
Adversarial Setting – Ring Set
Cover
Faster
Larger Error
Coord
Free
Conc
Ctrl
Always fast
Possibly wrong
Possibly slow
Always optimal