Transcript Coordinate Descent Methods - Georgia Institute of Technology

Some Zero-th Order / Direct Search Algorithms
Optimization in Engineering Design
Georgia Institute of Technology
Systems Realization Laboratory
11
Coordinate Descent Methods (see section 7.2)
•
Basically, each coordinate axis is searched and a descent is only made
along a unit vector.
•
The Cyclic Coordinate Descent method minimizes a function
ƒ(x1, x2, ..., xn)
cyclically with respect to the coordinate variables. That is, first x1 is
searched, then x2, etc.
•
Various variations are possible.
–
•
•
Can you think of some?
Plus: Generally attractive because of their easy implementation.
Minus: Generally their convergence properties are poorer than
steepest descent.
Optimization in Engineering Design
Georgia Institute of Technology
Systems Realization Laboratory
11
Hook and Jeeves Pattern Search (see section 7.3)
1. Define:
–
–
–
–
Starting point x0
Increment Di for all variables (i = 1, .., n)
Step reduction factor a
Termination parameter e
2. Perform Exploratory search
3. If exploratory move successful, Go to 5. If not, continue
4. Check for termination: Is ||Di||< e ?
•
•
Yes: Stop. Current point is x*
No: Reduce increments Di= (Di / a) for i = 1, .., n. Go to 2
5. Perform pattern move: xpk+1 = xk + (xk – xk-1)
6. Perform exploratory move with xp as the base point. Let result
be xk+1.
7. Is f(xk+1) < f(xk)?
•
•
Yes: Set xk-1 = xk and xk = xk+1. Go to 5.
No: go to 4.
Optimization in Engineering Design
Georgia Institute of Technology
Systems Realization Laboratory
11
Another Pattern Search Algorithm
(1)
Start at a base point,x base .
(2)
P erform a cyclic search about
x base t o find a direction of improvement . Selecting
each axis in turn, step in the direction of improvement assuming monotonic
behavior. T he finalpoint derived after exploring and st epping parallel t o each
axis in t urn is denoted as t he t emporary point , x te m p . T his represents t he
“ explorat ory move.” If t his procedure is successful,
go to st ep (3). Otherwise,
reduce t he st ep size and repeat (2).

)
(3)
P erform a “pat tern m ove” xt aocc by set t ingx a cc  x base  a x te m p  x base .
(4)
T est yxa cc ) vs y x te m p :

)
if yxa cc ) is bet ter, setx base  x te m p , x te m p  x a cc and ret urn t o (3), else
if yxa cc ) is worse, set x base  x te m p and return to (2).
(5)
Repeat (2) t hrough (4) unt il cyclic search yields no improvem ent wit h minimum
step size.
Optimization in Engineering Design
Georgia Institute of Technology
Systems Realization Laboratory
11
Pattern Search Graphically
x3acc
p
x2 acc
p
x 1acc
p
x1temp
= x 2base
e
x 1base
Optimization in Engineering Design
e
e
= x3 temp
= x 4base
e
x4temp
x4acc
p
e
= x 2temp
= x3base
KEY
e : explorat ory move
p : pattern mov e
improv ement
no improv ement
Georgia Institute of Technology
Systems Realization Laboratory
11
Pattern Search
Advantages:
• Simple
• Robust (relatively)
• No gradients necessary.
Disadvantages:
• Can get stuck and special "tricks" are needed to get the search
going again.
• May take a lot of calculations
•
Nonlinear multi-objective multiplex implementations using
pattern search algorithms have been made.
Optimization in Engineering Design
Georgia Institute of Technology
Systems Realization Laboratory
11
Exploratory Search
18
18
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
13
l
l
l
l
l
l
l
l
l
l
12
l
l
l
l
l
l
l
l
l
l
BEAM (meters)
16
15
14
11
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
10
116118 120122124126128130132134
LBP (meters)
(a) Points evaluated
Optimization in Engineering Design
17
16
BEAM (meters)
l
17
1
15
14
13
12
11
5
876
2
43
1
10
116118 120122124126128130132134
LBP (meters)
(b) Contours of function generated
Georgia Institute of Technology
Systems Realization Laboratory
11
Exploratory Search Algorithms
Two options currently exist to identify the vector of design
variables, x.
1) Randomly select n points in the bounded space. A uniform
distribution is assumed for the variables between the defined bounds.
(An enhancement could be to allow for alternative distributions.)
Monte Carlo methods are based upon this.
2) Use a systematic search algorithm such as from Aird and Rice
(1977). While they claim that their systematic algorithm provides a
better way of searching a region (and provides better starting points)
than a random algorithm, their method uses a small number of
variable values repeatedly on each axis.
This can be a problem when considering a large number of variables
and attempting to “visualize” the effect of changes. Considering a
problem in 15 variables, 2 to the power 15 or 32,768 candidate
designs would need to be evaluated to ensure just two sample values
on each axis.
Optimization in Engineering Design
Georgia Institute of Technology
Systems Realization Laboratory
11