Transcript One dimensional optimization

Optimality Conditions for
Unconstrained optimization
• One dimensional optimization
– Necessary and sufficient conditions
• Multidimensional optimization
– Classification of stationary points
– Necssary and sufficient conditions for local
optima.
• Convexity and global optimality
One dimensional optimization
• We are accustomed to think that if f(x) has
a minimum then f’(x)=0 but….
5
|x-5|
4.5
0.2(x-5)2
4
3.5
3
2.5
2
1.5
1
0.5
0
0
1
2
3
4
5
6
7
8
9
10
1D Optimization jargon
• A point with zero
derivative is a
stationary point.
• x=5, Can be
25
20
15
10
5
0
-5
a minimum
2
y   x  5
A maximum
y  10 x  x 2
3
An inflection point y  0.2  x  5 
-10
-15
-20
-25
0
1
2
3
4
5
6
7
8
9
10
Optimality criteria for smooth 1D
functions at point x*
• f’(x*)=0 is the condition for stationarity and
a necessary condition for a minimum or a
maximum.
• f“(x*)>0 is sufficient for a minimum
• f“(x*)<0 is sufficient for a maximum
• With f”(x*)=0 needs information from
higher derivatives.
• Example?
Problems 1D
• Classify the stationary points of the following functions
from the optimality conditions, then check by plotting
them
1.
2.
3.
4.
2x3+3x2
3x4+4x3-12x2
x5
f=x4+4x3+6x2+4x
• Answer true or false:
– A function can have a negative value at its maximum point.
– If a constant is added to a function, the location of its minimum
point can change.
– If the curvature of a function is negative at a stationary point,
then the point is a maximum.
Taylor series expansion in n
dimensions
• Expanding f ( x1 , xn ) about a candidate
minimum x*
1 n
* f
f (x)  f (x*)    xi  xi  (x*)  
xi
2 j 1
i 1
n
2

f
*
*
x

x
x

x
 i i  j j  x x (x*) 

i 1
i
j
n
1 T
 f (x*)  x f (x*)  x H (x*)x 
2
T
if
f
 0 choose  xi  xi*  of opposite sign and other  x j  x*j   0
xi
So must have f  0
• This is the condition for stationarity
Conditions for minimum
1 T
f (x)  f (x*)  x f (x*)  x H (x*)x 
2
T
• Sufficient condition for a minimum is that
xT H (x*)x  0 for all x
• That is, the matrix of second derivatives
(Hessian) is positive definite
• Simplest way to check positive definiteness is
eigenvalues: All eigenvalues need to be positive
• Necessary conditions matrix is positive-semi
definite, all eigenvalues non-negative
Types of stationary points
•
•
•
•
•
Positive definite: Minimum
Positive semi-definite: possibly minimum
Indefinite: Saddle point
Negative semi-definite: possibly maximum
Negative definite: maximum
Example
f  x  x1 x2  x
2
1
2
2
Stationary point:
2 x1  x2 
f  

 x1  2 x2 
x1  x2  0
12
10
Hessian matrix
8
 2 f
2 f 
 x 2


x

x
2 1
1
1
2

H


2
2
  f

1
2
 f



2 
x2 
 x1x2
Eigenvalues: 1,2  1,3  minimum
Change to f  x  3 x1 x2  x
2
1
2
2
Eigenvalues: 1,2  1,5  saddle point
6
4
2
0
2
1
2
1
0
0
-1
-1
-2
x2
-2
x1
20
15
10
5
0
-5
2
1
2
1
0
0
-1
x2
-1
-2
-2
x1
Problems n-dimensional
• Find the stationary points of the following
functions and classify them:
1. x12  4 x1 x2  2 x1 x3  7 x22  6 x2 x3  5 x32
2. x  2 x2 x3  x  4 x
2
1
2
2
3.40 x1  x12 x2  x22 / x1
2
3
Global optimization
• The function x+sin(2x)
12
10
8
6
4
2
0
0
1
2
3
4
5
6
7
8
9
10
Convex function
• A straight line connecting two points will
not dip below the function graph.
f  x1  (1   )x 2    f (x1 )  (1   ) f  x 2 
• Convex function will have a single
minimum.
10
5
0
Sufficient condition: Positive
semi-definite Hessian
everywhere.
-5
-10
-15
-20
-25
0
1
2
3
4
5
6
7
8
9
10
Problems convexity
• Check for convexity the following
functions. If the function is not convex
everywhere, check its domain of
convexity.
1.
2.
3 x12  2 x1 x2  2 x22  8
5 x1  x12 x22  4 x22 / x1
3. x13  12 x1 x22  2 x22  5 x12
Reciprocal approximation
• Reciprocal approximation (linear in one over the
variables) is desirable in many cases because it
captures decreasing returns behavior.
n
 f 
f
(
x
)

f
(
x
)

(
x

x
)
• Linear approximation L


0
i
0i 

x
i 1
 i x
• Reciprocal approximation
0
 f 
f L (y )  f (y 0 )   ( yi  y0i ) 
yi  1/ xi

i 1
 yi y 0
m
n
x0i
f R (x)  f (x 0 )   ( xi  x0i )
xi
i 1
 f 


 xi x0
Conservative-convex
approximation
• At times we benefit from conservative
approximations
n
( x  x ) 2  f 
fL  fR  
i 1
i
0i
xi


 xi x0
 f 
fC (x)  f (x 0 )   Fi ( xi  x0i ) 


x
i 1
 i  x0
n
{
1
if x0i (f / xi )  0
Fi 
x0i / xi
otherwise
• All second derivatives of fC are non-negative
• Called convex linearization (CONLIN), Claude
Fleury
Problems approximations
1. Construct the linear, reciprocal, and
convex approximation at (1,1) to the
3
1
f

1


function
x x x
2. Plot and compare the function and the
two approximations.
3. Check on their properties of convexity
and conservativeness.
1
2
1