Transcript Engineering Optimization - Center of Mechanics, ETH Zurich
Engineering Optimization Concepts and Applications
Fred van Keulen Matthijs Langelaar CLA H21.1
Engineering Optimization – Concepts and Applications
Contents
●
Constrained Optimization: Optimality conditions recap
● Constrained Optimization: Algorithms ● Linear programming Engineering Optimization – Concepts and Applications
Inequality constrained problems
● Consider problem with only inequality constraints: min s.
x
t.
f
(
x
)
g i
(
x
) 0
x
n i
1
m x 2 g 2
f
●
At optimum
, only
active
constraints matter: min
x
s.
t.
g f j
(
x
) (
x
) 0
x
n j
1
k
m g 1
● Optimality conditions similar to equality constrained case
g 3 x 1
Engineering Optimization – Concepts and Applications
Inequality constraints
● First order optimality:
L
(
x
)
f
(
x
)
μ
T
g
(
x
)
g μ
T
f
x
0
μ
T
f
s
g
x
g
s
0
T
1 ● Consider feasible local variation around optimum:
f
x
μ
T
g
x
0
T
f
x
x
μ
T
g
x
x
0 0 (boundary optimum) Since
g
0
0 (feasible perturbation) Engineering Optimization – Concepts and Applications
Optimality condition
● Multipliers must be non-negative: ●
f
f
f
x
μ
T
g
x
x
0 0
k j
m
1
j j
g
g j j
0
x
0 0
x 2
g
2
g 2
f
f
g
1
g 1 x 1
Interpretation: negative gradient (descent direction)
f
lies in cone spanned by positive constraint gradients Engineering Optimization – Concepts and Applications
Optimality condition (2)
f
j
g j
● Feasible direction:
j T
s
0
j
1
k
g j
T
s
0
m
● Descent direction:
T
s
f
0
T
s
0
g 2 x 2
g
1 Feasible cone
f
g
2
g 1
g
2
f
g
1 ● Equivalent interpretation: no descent direction exists
x 1
within the cone of feasible directions Engineering Optimization – Concepts and Applications
Karush-Kuhn-Tucker conditions
● First order optimality conditions for constrained problem: min
x
s.
t.
f
(
x
)
g
(
x
)
0 h
(
x
)
0
Lagrangian:
L
f
(
x
)
μ
T
g
(
x
)
λ
T
h
(
x
)
L
x
f
x
i
g
x
i
i
h i
x
0 g
0
,
h
0
,
k g k
0
Engineering Optimization – Concepts and Applications
Sufficiency
● KKT conditions are
necessary
conditions for local constrained minima ● For sufficiency, consider the sufficiency conditions based on the active constraints:
T
2
L
x
2
x
0 on tangent subspace of
h
and active
g
.
● Special case: convex objective & convex feasible region: KKT conditions sufficient for
global optimality
Engineering Optimization – Concepts and Applications
Significance of multipliers
● Consider case where optimization problem depends on parameter
a
: min
x
s. t.
h f
x x
a
0
Lagrangian:
L
x λ
f
x
KKT:
L
x
f
x
T
h
x
0
Looking for:
df da
f
a
f
x
T d
x
da
Engineering Optimization – Concepts and Applications
T h
x
a
f
x
T
h
x
T
Significance of multipliers (3)
● Lagrange multipliers describe the
sensitivity of the objective to changes in the constraints
:
df da
f
a
h
a
● Similar equations can be derived for multiple constraints and inequalities ● Multipliers give “
price of raising the constraint
” ● Note, this makes it logical that at an optimum, multipliers of inequality constraints must be positive!
Engineering Optimization – Concepts and Applications
Contents
● Constrained Optimization: Optimality Criteria ●
Constrained Optimization: Algorithms
● Linear Programming Engineering Optimization – Concepts and Applications
Constrained optimization methods
● Approaches: – Transformation methods (penalty / barrier functions) plus unconstrained optimization algorithms – Random methods / Simplex-like methods – Feasible direction methods – Reduced gradient methods – Approximation methods (SLP, SQP) ● Note, constrained problems can also have interior optima!
Engineering Optimization – Concepts and Applications
Augmented Lagrangian method
● Recall penalty method:
f
~
f
p
max( 0 ,
g
) 2
f
~ (
p
10 ) ● Disadvantages: – High penalty factor needed for accurate results – High penalty factor causes ill-conditioning, slow convergence Engineering Optimization – Concepts and Applications
Augmented Lagrangian method
● Basic idea: – Add penalty term to Lagrangian – Use estimates and updates of multipliers
L
f
i h i
p
h i
2 ● Also possible for inequality constraints ● Multiplier update rules determine convergence
i
(
k
1 )
i
(
k
) 2 ●
Exact
convergence for moderate values of
p
Engineering Optimization – Concepts and Applications
i
(
k
) )
Contents
● Constrained Optimization: Optimality Criteria ● Constrained Optimization: Algorithms – Augmented Lagrangian –
Feasible directions methods
– Reduced gradient methods – Approximation methods – SQP ● Linear Programming Engineering Optimization – Concepts and Applications
Feasible direction methods
● Moving along the boundary – Rosen’s gradient projection method – Zoutendijk’s method of feasible directions ● Basic idea: – move along
steepest descent direction
until constraints are encountered
f
– step direction obtained by projecting steepest descent direction on
tangent plane
– repeat until KKT point is found Engineering Optimization – Concepts and Applications
1.
Gradient projection method
● Iterations follow the constraint boundary: ● For nonlinear constraints, mapping back to the constraint surface is needed, in
normal space x 1
● For simplicity, consider linear equality constrained problem:
h
= 0 min
x
s.
t.
x 3
f x 2 f
(
x
)
h
(
x
)
Ax
b
0
Engineering Optimization – Concepts and Applications
● ●
Gradient projection method (2)
f
Recall: – – Tangent space: Normal space:
h
x z
s
As
h
x
T
p 0
A
T
p s z A
* :
f
s
z
s
A
T
p
A
f
As
AA
T
p s
f
f
A
T
A
T
p
AA
T
1
A
f
Engineering Optimization – Concepts and Applications
A
f
p
AA AA
T T
p
1
A
f
Gradient projection method (3)
● Search direction in tangent space:
s
I
f
A
T
A
T
AA
1
A
1
A
f
f
f
Projection matrix ● Nonlinear case:
P
I
h
T
x
h
x
h
T
x
1
h
x
f
● Correction in normal space:
s
'
h
T
x c
Engineering Optimization – Concepts and Applications
Correction to constraint boundary
● Correction in normal subspace, e.g. using Newton iterations: First order Taylor approximation:
h
(
x
'
k
1
s
'
k
)
h
(
x
'
k
1 )
h
x s
'
k
h
(
x
'
k
1 )
h
x
h
x
T
c
k
0
0 c
k
Iterations:
x
k
1
x
'
k
1
h x
h x
x
k
s
k
x’
k+1
s
'
k
x
k+1
h
T
x
h
T
x
h
x
h x
T
1
h
(
x
'
k
1 1
h
(
x
'
k
1 )
c
k
) Engineering Optimization – Concepts and Applications
Practical aspects
● How to deal with inequality constraints?
● Use
active set strategy
: – Keep set of active inequality constraints – Treat these as equality constraints – Update the set regularly (heuristic rules) ● In gradient projection method, if
s
= 0 : – Check multipliers: could be KKT point – If any
i
< 0, this constraint is inactive and can be removed from the active set Engineering Optimization – Concepts and Applications
Slack variables
● Alternative way of dealing with inequality constraints: using
slack variables:
min
x
s.
t.
f
(
x
x
)
g
(
x
)
0 h
(
x
)
0 x
x
min s
x
,
y
.
t.
x 0
f
0
0 x
Large numb e r ● Disadvantages: all constraints considered all the time, + increased number of design variables Engineering Optimization – Concepts and Applications
2.
Zoutendijk’s feasible directions
● Basic idea: – move along
steepest descent direction
until constraints are encountered
f
– at constraint surface, solve subproblem to find descending feasible direction ● – repeat until KKT point is found Subproblem: Descending: Feasible:
g i T
f
s
T
s
0 0
i
min
s
, s.t.
f T
s
,
g i T
s
s i
1 Engineering Optimization – Concepts and Applications
Zoutendijk’s method
● Subproblem linear: efficiently solved ● Determine active set before solving subproblem!
● When = 0 : KKT point found ● Method needs feasible starting point. Engineering Optimization – Concepts and Applications
Contents
● Constrained Optimization: Optimality Criteria ● Constrained Optimization: Algorithms – Augmented Lagrangian – Feasible directions methods –
Reduced gradient methods
– Approximation methods – SQP ● Linear Programming Engineering Optimization – Concepts and Applications
Reduced gradient methods
● Basic idea: – Choose set of
n - m
decision variables
d
– Use reduced gradient in unconstr. gradient-based method ● Recall:
d
f
d
f
s
h
1
s
h
d
x
2
x
3 reduced gradient ● State variables
s
can be determined from:
s
h
1
s
h
d
d
Engineering Optimization – Concepts and Applications
x
1 (iteratively for nonlinear constraints)
Reduced gradient method
● Nonlinear constraints: Newton iterations to return to constraint surface (determine
s
):
h
s
d
s
d
s
h s
1 until convergence ● Variants using 2 nd order information also exist ● Drawback: selection of decision variables (but some procedures exist) Engineering Optimization – Concepts and Applications
Contents
● Constrained Optimization: Optimality Criteria ● Constrained Optimization: Algorithms – Augmented Lagrangian – Feasible directions methods – Reduced gradient methods –
Approximation methods
– SQP ● Linear Programming Engineering Optimization – Concepts and Applications
Approximation methods
● SLP: Sequential Linear Programming ● Solving series of linear approximate problems ● Efficient methods for linear constrained problems available min
x
s.
t.
f
(
x
)
g
(
x
)
0 h
(
x
)
0
min
x
s.
t.
f
(
x
k
)
f
x
k
x
x
k
g
(
x
k
)
g
x
k
x
x
k
0 h
(
x
k
)
h
x
k
x
x
k
0
Engineering Optimization – Concepts and Applications
SLP 1-D illustration
● SLP iterations approach convex feasible domain from outside: min s.
x
t.
f
2
x g
(
x
) 3 (
x
1 )(
x
4 )
0
f g dg
6
x
15
dx g
(
x
) 1 12 15
x
0
g
(
x
) 2 1 .
92 10 .
2
x
0 Engineering Optimization – Concepts and Applications
x
= 0.8
x
= 0.988
SLP points of attention
● Solves LP problem in every cycle: efficient only when analysis cost is relatively high ● Tendency to diverge – Solution: trust region (move limits)
x 2 x 1
Engineering Optimization – Concepts and Applications
SLP points of attention (2)
● Infeasible starting point can result in unsolvable LP problem – Solution:
relaxing
constraints in first cycles min
x
s.
t.
f
(
x
k
)
f
x
k
x
x
k
g
(
x
k
)
g
x
k
x
x
k
0
min
x
, s. t.
g
f
(
x
k
) (
x
k
)
f
x
k
g
x
k
k k
k
1
0
k
sufficiently large to force solution into feasible region Engineering Optimization – Concepts and Applications
SLP points of attention (3)
● Cycling can occur when optimum lies on curved constraint – Solution: move limit reduction strategy
f x 2 x 1
Engineering Optimization – Concepts and Applications
Method of Moving Asymptotes
● First order method, by Svanberg (1987) ● Builds
convex
approximate problem, approximating responses using:
y i n
1
U i P i
x i
x i Q i
L i
● Approximate problem solved efficiently ● Popular method in
topology optimization x i Q i
L i
Engineering Optimization – Concepts and Applications
U i P i
x i
Sequential Approximate Optimization
● Zeroth order method: ● 1.
Determine initial trust region 2.
3.
Generate sampling points (design of experiments) Build response surface (e.g. Least Squares, Kriging, …) 4.
Optimize approximate problem 5.
Check convergence, update trust region, repeate from 2 Many variants!
● See also Lecture 4 Engineering Optimization – Concepts and Applications
Sequential Approximate Optimization
● Good approach for expensive models ● RS dampens noise Design domain ● Versatile Optimum Response surface Engineering Optimization – Concepts and Applications Sub-optimal point Trust region
Contents
● Constrained Optimization: Optimality Criteria ● Constrained Optimization: Algorithms – Augmented Lagrangian – Feasible directions methods – Reduced gradient methods – Approximation methods –
SQP
● Linear Programming Engineering Optimization – Concepts and Applications
SQP
● SQP: Sequential Quadratic Programming ● Newton method to solve the KKT conditions min
x
s.
t.
f
h
( (
x
)
x
)
0
KKT points:
L
x
f
x
λ
T
h
x
0
T
Newton: 2
L
x
2 2
L
Engineering Optimization – Concepts and Applications 2
L
2
λ
L
2
x λ
L
L
x λ
SQP (2)
● Newton: 2
x
2
f
2
L
x
2 2
L
λ
T
2
h
x
2
h
x
W A
2 2
λ x 0
L L
2
h
T
x λ
x λ
L T
x
L
λ
T
f
x
T
h
h
T
x λ
A 0
T
x
h
T
A λ
Engineering Optimization – Concepts and Applications
SQP (3)
W A
W
k
A
k
A
T
0
x
h
T
A λ
A
k T
0
λ s
k k
1
f
h
k k
λ
k
1
x
k
1
λ
k
x
k
λ
k
s
k
f k
A
k
s
k
W
k
s
k
h
k
0 A
k T
λ
k
1
0
Note: KKT conditions of: min
s
k
s.t.
q
(
s
k
)
A
k
s
k
f k
h
k
s
k
0
1 2
s
k T
W
k
s
k
Quadratic subproblem for finding search direction
s
k
Engineering Optimization – Concepts and Applications
Quadratic subproblem
● Quadratic subproblem with linear constraints can be solved efficiently: – General case: min
x
s.t.
1 2
x
T
Qx Ax
b
0 c
T
x
– KKT condition:
Q A
– Solution:
x λ
Q A
Engineering Optimization – Concepts and Applications
A
T
0 x λ
A
T
0
1
b c
b c
Basic SQP algorithm
1.
Choose initial point
x
0 and initial multiplier estimates 0 2.
Set up matrices for QP subproblem 3.
Solve QP subproblem
s
k
,
k
+1 4.
Set
x
k
+1 = x
k
+ s
k
5.
Check convergence criteria Finished Engineering Optimization – Concepts and Applications
SQP refinements
● positive definite 2
x
L
2 ● – Line search along
s
k improves robustness 2
L
x
quasi-Newton approaches (DFP, BFGS) can be used (also ensure positive definiteness) ● For dealing with inequality constraints, various active set strategies exist Engineering Optimization – Concepts and Applications
Comparison
AugLag Zoutendijk GRG Method SQP
Feasible starting point?
Nonlinear constraints?
Equality constraints?
Uses active set?
Iterates feasible?
Derivatives needed?
No Yes Yes Yes No Yes Yes Yes Hard Yes Yes Yes Yes Yes Yes No No Yes No Yes Yes Yes No Yes SQP generally seen as best general-purpose method for constrained problems Engineering Optimization – Concepts and Applications
Contents
● Constrained Optimization: Optimality Criteria ● Constrained Optimization: Algorithms ●
Linear programming
Engineering Optimization – Concepts and Applications
Linear programming problem
● Linear objective and constraint functions: min
x
c
1
x
1
c
2
x
2
c n x n
c
T
x
s.t.
h i
a
i T
x
b i
0
g j
a
j T
x
b j
0
i j
1
m
1
m
1 1
m
2 min
x
s.t.
c
T
x h g
A
1
x A
2
x
b
1
b
2
0 0 A
1
a a
1
T
a
2
m
1
T T
,
b
1
b b
1 2
b m
1 Engineering Optimization – Concepts and Applications
Feasible domain
● Linear constraints divide design space into two convex half-spaces:
x 2 X
1
g
1
a
1
T
x
b
1 0
X x 1
● Feasible domain = intersection of convex half-spaces:
X
X
1
X
2
X m
● Result:
X
= convex polyhedron Engineering Optimization – Concepts and Applications
Global optimality
● ● Convex objective function on convex feasible domain: KKT point = unique global min
x
s.t.
c
T
x h g
A
1
x A
2
x
b
1
b
2
0 0
optimum KKT conditions:
L
c
T
x
λ
T
A
1
x
b
1
μ
T
A
2
x
b
2
A c
T
1
x
λ
T
b
1
μ λ
T
A 0
, 2
x A
1
μ
b 0
, 2
0
μ
T
A
A
0 2 2
x
b
2
0
T
0
Engineering Optimization – Concepts and Applications