Transcript Download Here

Difference of Convex (DC)
Decomposition of
Nonconvex Polynomials with
Algebraic Techniques
Georgina Hall
Princeton, ORFE
Joint work with
Amir Ali Ahmadi
Princeton, ORFE
7/26/2016
INFORMS 2015
1
DC Decomposition of Nonconvex Polynomials with Algebraic Techniques
Difference of Convex (DC) programming
• Problems of the form
min 𝑓0 (𝑥)
𝑠. 𝑡. 𝑓𝑖 𝑥 ≤ 0
where:
• 𝑓𝑖 𝑥 ≔ 𝑔𝑖 𝑥 − ℎ𝑖 𝑥 , 𝑖 = 0, … , 𝑚,
• 𝑔𝑖 : ℝ𝑛 → ℝ, ℎ𝑖 : ℝ𝑛 → ℝ are convex.
• Appears in:
• Machine learning and statistics
• Statistical physics
• Communications and networks, etc.
2
DC Decomposition of Nonconvex Polynomials with Algebraic Techniques
Convex-Concave Procedure (CCP)
• Heuristic for minimizing DC programming problems.
• Idea:
Input
𝑘≔0
x𝑥0 , initial point
𝑓𝑖 = 𝑔𝑖 − ℎ𝑖 ,
𝑖 = 0, … , 𝑚
Convexify by linearizing 𝒉
x𝒇𝒌𝒊 𝒙 = 𝑔𝑖 𝑥 − (ℎ𝑖 𝑥𝑘 + 𝛻ℎ𝑖 𝑥𝑘
convex convex
𝒇𝒌𝒊 𝒙
𝑇
Solve convex subproblem
𝑥 − 𝑥𝑘 )
affine
𝑘 ≔𝑘+1
Take 𝑥𝑘+1 to be the solution of
min 𝑓0𝑘 𝑥
𝑠. 𝑡. 𝑓𝑖𝑘 𝑥 ≤ 0, 𝑖 = 1, … , 𝑚
𝒇𝒊 (𝒙)
3
DC Decomposition of Nonconvex Polynomials with Algebraic Techniques
Convex-Concave Procedure (CCP)
• Toy example: min 𝑓 𝑥 , where 𝑓 𝑥 ≔ 𝑔 𝑥 − ℎ(𝑥)
𝑥
Convexify 𝑓 𝑥 to
obtain 𝑓 0 (𝑥)
Initial point: 𝑥0 = 2
Minimize 𝑓 0 (𝑥) and
obtain 𝑥1
Reiterate
𝑥𝑥𝑥∞43𝑥𝑥2 1
𝑥0
4
DC Decomposition of Nonconvex Polynomials with Algebraic Techniques
CCP for nonconvex polynomial optimization problems (1/2)
CCP relies on the input functions being given as the difference of
convex functions.
• Any 𝑪𝟐 function can be written as a difference of convex functions.
• In fact, infinite number of difference of convex decompositions.
Initial decomposition
x𝑓 𝑥 = 𝑔 𝑥 − ℎ(𝑥)
Alternative decompositions
𝑓 𝑥 = 𝑔 𝑥 +𝑝 𝑥 − ℎ 𝑥 +𝑝 𝑥
𝑝(𝑥) convex
How can we find such a decomposition?
Is there a way of optimizing over the set of possible decompositions?
(maybe to get a better one for CCP?)
5
DC Decomposition of Nonconvex Polynomials with Algebraic Techniques
CCP for nonconvex polynomial optimization problems (2/2)
We consider these questions for polynomials.
Definition: A polynomial 𝑔 is a dcd of 𝑓 if
• 𝑔 is convex
• ℎ ≔ 𝑔 − 𝑓 is convex
Theorem: Any polynomial has a (polynomial) dcd of the same degree.
Proof by Wang, Schwing and Urtasun. Alternative proof given later in this
presentation, as corollary of stronger theorem.
Theorem: Given two polynomials 𝑓and 𝑔 of degree ≥ 4, it is strongly
NP-hard to test if 𝑔 is a dcd of 𝑓.
6
DC Decomposition of Nonconvex Polynomials with Algebraic Techniques
Picking the “best” decomposition for CCP (1/3)
Algorithm
Linearize 𝒉 𝒙 around a point 𝑥𝑘 to obtain convexified version of 𝒇(𝒙)
Idea
Pick ℎ 𝑥 such that it is as close as possible to affine around 𝑥𝑘
Mathematical translation
Minimize curvature of ℎ at 𝑥𝑘
Worst-case curvature*
Average curvature*
min 𝜆𝑚𝑎𝑥 (𝐻ℎ 𝑥𝑘 )
min 𝑇𝑟 𝐻ℎ (𝑥𝑘 )
s.t. 𝑓 = 𝑔 − ℎ
𝑔, ℎ convex
s.t. 𝑓 = 𝑔 − ℎ,
𝑔, ℎ convex
g,h
*𝜆𝑚𝑎𝑥 𝐻ℎ 𝑥𝑘 = max
𝑦 𝑇 𝐻ℎ 𝑥𝑘 𝑦
𝑛−1
𝑦∈𝑆
𝑔,ℎ
* 𝑇𝑟 𝐻ℎ 𝑥𝑘 =
𝑦∈𝑆 𝑛−1
𝑦 𝑇 𝐻ℎ 𝑥𝑘 𝑦 𝑑𝜎
7
DC Decomposition of Nonconvex Polynomials with Algebraic Techniques
Picking the “best” decomposition for CCP (2/3)
Definition: A dcd 𝑔 of 𝑓 is undominated if no other dcd of
𝑓 can be obtained by subtracting a convex function from 𝑔.
𝒇 𝒙 = 𝒙𝟒 − 𝟑𝒙𝟐 + 𝟐𝒙 − 𝟐
𝒈 𝒙 = 𝒙𝟒 + 𝒙𝟐 ,
𝒉 𝒙 = 𝟒𝒙𝟐 + 𝟐𝒙 − 𝟐
Convexify around 𝑥0 = 2 to
get 𝒇𝟎 𝒙
DOMINATED BY
𝒈′ 𝒙 = 𝒙𝟒 ,
𝒉′ 𝒙 = 𝟑𝒙𝟐 + 𝟐𝒙 − 𝟐
Convexify around 𝑥0 = 2 to
get 𝒇𝟎′ 𝒙
If 𝒈′ dominates 𝒈 then the next iterate in CCP obtained using 𝒈′ always beats the one obtained using 𝒈.
8
DC Decomposition of Nonconvex Polynomials with Algebraic Techniques
Picking the “best” decomposition for CCP (3/3)
Theorem: Given a polynomial 𝑓, consider
(⋆)
min
𝑔
1
𝐴𝑛
𝑆 𝑛−1
𝑇𝑟 𝐻𝑔 𝑑𝜎, (where 𝐴𝑛 =
2𝜋𝑛/2
)
Γ(𝑛/2)
s.t. 𝑔 convex, 𝑔 − 𝑓 convex
Any optimal solution is an undominated dcd of 𝑓 (and an optimal
solution always exists).
Theorem: If 𝑓 has degree 4, it is strongly NP-hard to solve (⋆).
How can we get around this?
9
DC Decomposition of Nonconvex Polynomials with Algebraic Techniques
Convex relaxations for DC decompositions (1/5)
SOS, DSOS, SDSOS polynomials (Ahmadi, Majumdar)
• Families of nonnegative polynomials.
Type
Characterization
Testing
membership
Sum of squares (sos)
∃𝑞𝑖 , polynomials, s.t. 𝑝(𝑥) = ∑𝑞𝑖2 (𝑥)
SDP
2
⇓
Scaled diagonally dominant
sum of squares (sdsos)
p= ∑𝑖 𝛼𝑖 𝑚𝑖2 + ∑𝑖,𝑗 𝛽𝑖+ 𝑚𝑖 + 𝛾𝑗+ 𝑚𝑗 + 𝛽𝑖− 𝑚𝑖 − 𝛾𝑗− 𝑚𝑗
𝑚𝑖 , 𝑚𝑗 monomials, 𝛼𝑖 ≥ 0
Diagonally dominant
sum of squares (dsos)
+
−
p= ∑𝑖 𝛼𝑖 𝑚𝑖2 + ∑𝑖,𝑗 𝛽𝑖,𝑗
𝑚𝑖 + 𝑚𝑗 + 𝛽𝑖,𝑗
𝑚𝑖 − 𝑚𝑗
+,−
𝑚𝑖 , 𝑚𝑗 monomials, 𝛼𝑖 , 𝛽𝑖𝑗
≥0
⇓
See talk by Sanjeeb Dash on Tuesday Nov 03, 13:30 - 15:00 (TC11).
2
2
SOCP
2
LP
10
DC Decomposition of Nonconvex Polynomials with Algebraic Techniques
Convex relaxations for DC decompositions (2/5)
DSOS-convex, SDSOS-convex, SOS-convex polynomials
𝑝(𝑥)
convex
⇔ 𝐻𝑝 𝑥 ≽ 0, ∀𝑥 ⇔
𝑦 𝑇 𝐻𝑝 𝑥 𝑦 ≥ 0,
∀𝑥, 𝑦 ∈ ℝ𝑛
⇐
𝑦 𝑇 𝐻𝑝 𝑥 𝑦
sos/sdsos/dsos
Definitions:
LP
• 𝑝 is dsos-convex if 𝑦 𝑇 𝐻𝑝 𝑥 𝑦 is dsos.
• 𝑝 is sdsos-convex if 𝑦 𝑇 𝐻𝑝 𝑥 𝑦 is sdsos. SOCP
• 𝑝 is sos-convex if 𝑦 𝑇 𝐻𝑝 𝑥 𝑦 is sos.
SDP
Condition: 𝑓 = 𝑔 − ℎ,
𝑔, ℎ convex
Condition: 𝑓 = 𝑔 − ℎ,
𝑔, ℎ s/d/sos-convex
11
DC Decomposition of Nonconvex Polynomials with Algebraic Techniques
Convex relaxations for DC decompositions (3/5)
Comparison of these sets on a parametric family of polynomials:
𝑝 𝑥1 , 𝑥2 = 2𝑥14 + 2𝑥24 + 𝑎𝑥13 𝑥2 + 𝑏𝑥12 𝑥22 + 𝑐𝑥1 𝑥23
𝑐 = −0.5
𝑐=0
𝑐=1
𝑏
𝑏
𝑏
𝑎
dsos-convex
𝑎
sdsos-convex
𝑎
sos-convex=convex
12
DC Decomposition of Nonconvex Polynomials with Algebraic Techniques
Convex relaxations for DC decompositions (4/5)
Can any polynomial be written as the difference of two dsos/sdsos/sos
convex polynomials?
Theorem: Any polynomial can be written as the difference of two dsosconvex polynomials.
Corollary: Same holds for sdsos-convex, sos-convex and convex.
13
DC Decomposition of Nonconvex Polynomials with Algebraic Techniques
Convex relaxations for DC decompositions (5/5)
Proof idea: Let 𝐾 be a full dimensional cone in a vector space 𝐸.
Then any 𝑣 ∈ 𝐸 can be written as 𝑣 = 𝑘1 − 𝑘2 , 𝑘1 , 𝑘2 ∈ 𝐾.
=:
𝑘′
Proof sketch:
K
E
∃ 𝛼 < 1 such that 1 − 𝛼 𝑣 + 𝛼𝑘 ∈ 𝐾
𝒌
𝒗
𝒌′
1
𝛼
′
⇔𝑣=
𝑘 −
𝑘
1−𝛼
1−𝛼
𝑘1 ∈ 𝐾
𝑘2 ∈ 𝐾
What’s left to show: dsos-convex polynomials form a full-dimensional cone.
• “Obvious” choices (i.e., 𝑝 𝑥 =
𝑑/2
2
(∑𝑖 𝑥𝑖 ) )
do not work.
14
DC Decomposition of Nonconvex Polynomials with Algebraic Techniques
Comparing different decompositions (1/2)
• Solving the problem:
min
𝐵= 𝑥
𝑥 ≤𝑅}
𝑓0 , where 𝑓0 has 𝑛 = 8 and 𝑑 = 4.
• Decompose 𝑓0 , run CCP for 4 minutes and compare objective value.
Feasibility
𝝀𝒎𝒂𝒙 𝑯𝒉 (𝒙𝟎 )
𝝀𝒎𝒂𝒙,𝑩 𝑯𝒉
Undominated
min 0
min 𝑡
s.t. 𝑓0 = 𝑔 − ℎ
𝑔, ℎ sos-convex
s.t. 𝑓0 = 𝑔 − ℎ
𝑔, ℎ sos-convex
𝑡𝐼 − 𝐻ℎ 𝑥0 ≽ 0
min 𝑡
𝑡,𝑔,ℎ 𝑡
min
min max𝑡,𝑔,ℎ
𝜆 𝟐 (𝐻ℎ 𝑥𝟐 )
𝒕𝑰 − 𝑔,ℎ
𝑯𝒉 𝒙𝑥∈B≽ 𝑚𝑎𝑥
𝑹 − ∑𝒙𝒊 𝝉(𝒙)
𝒙 ∈ 𝑩 ⇒𝑻 𝒕𝑰 − 𝑯𝒉 𝒙 ≽
𝟎
s.t.
𝑓
=
𝑔
−
ℎ,
𝒚𝑓 𝝉(𝒙)𝒚
= 𝑔 −sos
ℎ
𝑔, ℎ
sos-convex
𝑓
=
𝑔
−
ℎ
𝑔, ℎ convex
𝑔, ℎ sos-convex
1
min
𝑇𝑟 𝐻𝑔 𝑑𝜎
𝑔,ℎ 𝐴𝑛 𝑆
𝑛−1
𝑠. 𝑡. 𝑓0 = 𝑔 − ℎ
𝑔, ℎ sos-convex
g,h
g,h
15
DC Decomposition of Nonconvex Polynomials with Algebraic Techniques
Comparing different decompositions (2/2)
• Average over 30 iterations
• Solver: Mosek
• Computer: 8Gb RAM, 2.40GHz
processor
𝝀𝒎𝒂𝒙,𝑩 𝑯𝒉 Undominated
Feasibility 𝝀𝒎𝒂𝒙 𝑯𝒉 𝒙𝟎
Conclusion: Rate of convergence of CCP strongly affected by initial
decomposition.
16
DC Decomposition of Nonconvex Polynomials with Algebraic Techniques
Comparing the different relaxations (1/2)
• Impact of relaxations on solving
1
min
𝑇𝑟 𝐻𝑔 𝑑𝜎
𝐴𝑛 𝑆 𝑛−1
𝑠. 𝑡. 𝑓 = 𝑔 − ℎ, 𝑔, ℎ s/d/sos-convex
for some random 𝑓 of degree 4.
𝒏=𝟔
𝒏 = 𝟏𝟎
Type of
relaxation
Time
Opt value
dsos-convex
<1s
62090
<1s
sdsos-convex
<1s
53557
sos-convex
<1s
11602
Time Opt Value
𝒏 = 𝟏𝟒
𝒏 = 𝟏𝟔
Time
Opt value
Time
Opt value
168481
2.33s
136427
6.91s
48457
1.11s
132376
3.89s
99667
12.16s
32875
44.42s
18346
800.16s
9828
30hrs+
------------17
DC Decomposition of Nonconvex Polynomials with Algebraic Techniques
Comparing the different relaxations (2/2)
Sos-Convex + CCP
Decompose
1
min
𝑇𝑟 𝐻𝑔𝑖 𝑑𝜎
Convexify
𝐴𝑛 𝑆 𝑛−1
𝑓𝑖 = 𝑔𝑖 − ℎ𝑖 ,
𝑔𝑖 , ℎ𝑖 sos-convex
1000000
0
Sdsos-Convex + multiple decomposition CCP
Solve convex
subproblem
𝒏=𝟖
𝒏 = 𝟏𝟎
Decompose
min 𝑡
𝑡𝐼 − 𝐻ℎ𝑘 (𝑥𝑘 ) sdd
𝑖
𝑓𝑖 = 𝑔𝑖 − ℎ𝑖 ,
𝑔𝑖 , ℎ𝑖 sdsos-convex
Convexify
Solve convex
subproblem
𝒏 = 𝟏𝟐
-1000000
-2000000
-3000000
-4000000
-5000000
-6000000
-7000000
• Unconstrained problem with
objective of degree 4
• Objective value after 4mins
• Average over 30 instances
-8000000
-9000000
18
DC Decomposition of Nonconvex Polynomials with Algebraic Techniques
Main messages
• To apply CCP to polynomial optimization, a DC decomposition is
needed. Choice of decomposition impacts convergence rates.
• Dcds can be efficiently obtained using convex relaxations based on
dsos-convexity (LP), sdsos-convexity (SOCP), and sos-convexity (SDP).
• Dsos-convex and sdsos-convex relaxations scale to a larger number of
variables.
• The choice of the method depends on the size of the problem: sosconvexity and undominated dcds for small n, s/dsos-convexity and
multiple decomposition CCP for larger n.
19
Thank you for listening
Questions?
Want to learn more? http://scholar.princeton.edu/ghall
7/26/2016
INFORMS 2015
20