Transcript Numerical Comparison of Optimization Strategies

Numerical Comparison of Optimization Strategies
for Solving a Nonzero Residual Nonlinear Least Squares Problem
Brenda Bueno
Advisors: Drs. L. Velázquez and M. Argáez
Department of Mathematical Sciences
University of Texas at El Paso
Sponsored by the Minority Access to Research Careers Program and
the US Department of the Army DAAD19-01-1-0741
Problem Data and Model
The positions of 40 atoms corresponding to the
selected beta sheets of the protein of interest
Introduction
An important research activity in the area of global
optimization is to determine an effective strategy for
solving least squares problems that commonly arises
in science and engineering. Our objectives are:

To present a numerical comparison of optimization
strategies applied to a nonzero residual problem.
[xi,yi,zi] = [10.056 5.984 12.751
13.300 6.344 10.771
16.629 5.519 12.424
atom 1
atom 2
atom 3
………………………………
19.034 2.351 8.007
22.709 1.484 7.941]
Numerical Results
Our Goal
Information given:
………..
atom 39
atom 40]
Initial point: wo= [1; 1; 1; 5; 8; 20; -10; -10; -12]
To find the global minimum w* of

40
f ( w) 
1
2
ri 2 ( w)  12 R( w) T R( w),
such that f ( w*)  0.
f ( w*) 


ri ( w*) 2 ri ( w*)  0
Given an initial vector wo , a maximum number
of iterations kmax  1500, and tolerance
f ( w )  f ( w)
*
of
n
Hyperboloid Least Squares Problem
ri(w)  
xˆi
yˆ i
zˆi
 2  2
2
a
b
c
2
2
2
2.
Compute the Hessian, H, according to a chosen
methodology
i) Newton:
H  JT J 
ii)Gauss-Newton:
di
H  J J  I , where   f
T
iv)Finite-Difference:
Observed Data
H  2 f
where the unknown parameters are given by
ri ( w)  M ( w; xi y i , z i )  d i
Residual = Calculated Data - Observed Data
Nonzero Residual Problem
We are interested in least squares problems
where the function at the global minimum w*
will never be zero, i.e.
1.452
58.1
31
498.4
21
5.77e-11
1.31e-6
6.54e10
Levenberg
Marquadt
4.677
1500
1.77
2.54e-7
*Lowest
function
value
obtained
This technique
does not require
second order
information
iii)Levenberg-Marquadt:
Calculated Data
R(w)  r1 (w), r2 (w), , rm (w)

 2 ri  ri   2 f
H  JTJ
M ( w; xi yi , zi )
f ( w*)  0
m
i 1
w
T
-1.1277
-2.6494
2.5350
2.3806e4
9.9378e9
-4.8725e10
17.859
-2.6041e10
-4.4005e10
R( wk ), f k
Compute
 1
Newton’s
Method using
forward
difference
approximation
of the second
derivative
Gauss-Newton
Method
1.
The nonlinear residual function is calculated by:
i 1
  10 5
Do the following:
Types of minima

5.67e-6
ri ( w*)ri ( w*)  J ( w*)T R( w*)  0
Algorithm
ri2 (w)  12 R( w)T R( w), m  n
510.2
m
f   n  , find w*   n such that
1
2
29
Sufficient Condition
2
Given a real valued function f  C ,
Minimize f ( w) 
78.1
Approximated
Solution
w*
m
Global Optimization Problem
m
Newton’s
Method
f (w*)
f (w*)
Necessary Condition
i 1
Global Minima for
Least Squares Problems
CPU
Iterations for
time in convergence
seconds
i 1
 2 f ( w*)  J ( w*)T J ( w*) 
f (w)
Optimization
Strategy
i 1
 To introduce preliminary numerical results of a
proposed novel algorithm that seems to work best.
for any w  
Future Work
w  ( ,  , , a, b, c, t1 , t 2 , t 3 )  
9
3.
4.
 xˆ i 
 xi  t1 
ˆ 
   
 yi   A   yi   t 2 
 zˆi 
 z i  t 3 
where A is the 3x3 rotation matrix:
- cossinsin + coscos coscossin + cossin sinsin 
 - cossincos - sincos coscoscos - sinsin cossin 


 sinsin

- sincos
cos
Hs k  f k
Solve
Update
wk  wk  sk
5. Check convergence
If min
6. Else,
 f
k  k 1
k
2. As the work of Velazquez, Tapia and Zhang, there
is no damped step added to the algorithm in search of a
global minimum
Optimization
Strategy
CPU
Iterations
time in
for
second converge
s
nce
f (w*)
f (w*)
wk  wk   s k ,   1
0.7985
1.8808
0.4824
8.9669
6.5582
33.8493
-20.0065
-10.4050
-2.0352
w* (refined solution)
Levenberg Marquadt 14.1
&
Newton’s Method
6
1.766
5.93e-7
.78790
1.8871
.48306
8.9842
6.6050
42.350
-20.102
-10.346
-1.5673
Levenberg Marquadt 29.1
&
Newton’s Method
using forward
difference
6
1.766e
8.82e-7
same solution as
above
and go to step 1
1. The selection of perturbation parameter set to
 f
.59099
.67011
1.1831
2.6981e4
9.4475e6
4.1283e4
-8.1302
-9.4475e6
3.6276
Refinement Stage
Initial point: corresponding w* from Levenberg-Marquadt Method
, f k    , break, end
Main Modification
-.23003
3.1190
.048160
4.2320e4
3.8198e10
-1.8009e11
-15.746
-9.9654e10
-1.6247e11
Accepted/
Rejected
Solution by
Chemists
Rejected due
to large value
of certain
parameters
Rejected due
to large value
of certain
parameters
Rejected due
to large value
of certain
parameters
Accepted
1) To add a multistart technique
2) To improve the rate of convergence of the
algorithm
3) To include constraints on the variables
4) To test the algorithm on more problems
Acknowledgment
We thank E. Tolonen, S.
Kulshreshtha, and B. Stec from
the Macromolecular
Crystallography Lab, Chemistry
Department, UTEP, for providing
the problem formulation, data
and revising the approximated
solutions.
Contact Information
Brenda Bueno, Undergraduate Student
University of Texas at El Paso
Department of Mathematical Sciences
500 W. University Avenue
El Paso, Texas 79968-0514
USA
Email: [email protected]
Phone: (915) 747-6858
Fax: (915) 747-6502
Office: Bell Hall, Room 215