Transcript Document

Paper review of ENGG*6140 Optimization Techniques
Paper Review
-- Interior-Point Methods
for LP
Yanrong Hu
Ce Yu
Mar. 11, 2003
Outline
General introduction
 The original algorithm description
 A variant of Karmarkar’s algorithm
- a detailed simple example
 Another example
 More issues

General introduction
Interior-point methods for linear programming:

The most dramatic new development in operations
research (linear programming) during the 1980s.

Opened in 1984 by a young mathematician at AT&T
Bell Labs, N. Karmarkar.

Polynomial-time algorithm and has great potential for
solving huge linear programming problems beyond the
reach of the simplex method.

Karmarkar’s method stimulated development in both
interior-point and simplex methods.
General introduction
Difference between Interior point methods and the simplex method
The big difference between Interior point methods and the simplex
method lies in the nature of trial solutions.
Simplex method
Trial
solutions
CPF (Corner Point
Feasible) solutions
worst case:# iterations
can increase
complexity
exponentially in the
number of variables n: 2 n
Interior-point
method
Interior points
(points inside the
boundary of the
feasible region)
Polynomial time
Karmarkar’s original Algorithm
Karmarkar assumes that the LP is given
in Canonical form of the problem:
Assumptions:
1 1
1
1. X  ( , ,..., )
n n
n
2. min z  0
Min z  CX
s.t.
AX  0
1X  1
X0
sat isfiesAX  0
To apply the algorithm to LP problem in standard
form, a transformation is needed.
Min
Z = CX
s.t. AX b
X0
Karmarkar’s original Algorithm
An example of transformation
from standard form
to canonical form
Min z = y1+y2
Min z=5x1 + 5x2
(z=cy)
s.t. y1+2y2  2 (AYb)
y1, y2  0
s.t. 3x1+8x2+3x3-2x4=0 (AX=0)
x1+ x2+ x3+ x4=1 (1X =1)
xj  0, j=1,2,3,4
Karmarkar’s original Algorithm
The principal idea:
The algorithm creates a sequence of
points x( 0) , x(1) , x( 2) ,..., x( k ) having
decreasing values of the objective
th
(k )
function. In the k step, the point x
is
brought into the center of the simplex by a
projective transformation.
Karmarkar’s original Algorithm
The steps of Karmarkar’s algorithm are:
Step 0.
start with the solution point
Step k.
and compute the step
length parameters
Define:
1 1
1
,
,..., )
n n
n
1
(n  1)
r
and  
3n
n(n  1)
X0  (
Dk  diagxk 1...xkn  and compute:
T
T 1
T
c

I

P
(
PP
)
P
(
CD
)
p
k
 ADk 
P

cp
1 1
1 T
1


Ynew  ( , ,..., )  r
n n
n
cp
X is brought to the

center by:
Dk1 X
Y 
1Dk1 X
DkYnew
X k 1 
1DkYnew

A Variant
The Affine Variant of Karmarkar’s Algorithm:
Concept 1: Shoot through the interior of the feasible region toward an
optimal solution.
Concept 2: Move in a direction that improves the objective function value
at the fastest feasible rate.
Concept 3: Transform the feasible region to place the current trail solution
near its center, thereby enabling a large improvement when
concept 2 is implemented.
An Example
The Problem:
Max Z= x1+ 2x2
s.t.
x1+ x28
x1,x2 0
Optimal solution
is (x1,x2)=(0,8)
with Z=16
An Example
Concept 1:
Shoot through
the interior of the
feasible region
toward an
optimal solution.
 The algorithm begins with an initial solution that lies in
the interior of the feasible region.
 Arbitrarily choose (x1,x2)=(2,2) to be the initial solution.
Max Z= x1+ 2x2
s.t.
Concept 2:
Move in a
direction that
improves the
objective
function value at
the fastest
feasible rate.
x1+ x28
x1,x2 0
 The direction is perpendiculars to (and toward) the
objective function line. (3,4)=(2,2)+(1,2), where the
vector (1,2) is the gradient of the Objective Function.
The gradient is the coefficient of the objective function.
An Example
The algorithm actually operates on the augmented form.
The augmented form can be
Max Z = cT X
written in general as:
s.t.
AX = b
X0
Max Z = x1+ 2x2
s.t.
letting x3 be the slack
x1+ x28
Max Z = x1+ 2x2
s.t.
x1,x2  0
x1,x2,x3  0
so
1 
c   2 ,
 
0
 x1 
x   x2 ,
 
 x3 
Initial solution:
x1+ x2+ x3 = 8
A  [1,1,1],
0
b  [8], 0  0
 
0
(2,2)
(2,2,4)
Gradient of obj. fn.: (1,2)
(1,2,0)
An Example
Show The augmented form graphically
- Feasible region : the triangle
- Optimum (0,8,0), Initial solution: (2,2,4)
- Gradient of Obj. fn. : cT =[ 1, 2, 0 ]
Using Projected Gradient to
implement Concept 1 & 2:

Adding the gradient to the
initial leads to (3,4,4)= (2, 2,
4)+ (1,2,0) (infeasible)

To remain feasible, the
algorithm project the point
(3,4,4) down onto the
feasible triangle.

To do so, the projected
gradient – gradient projected
onto the feasible region – is
used.

Then the next trial solution
move in the direction of
projected gradient
An Interior-point Algorithm
Using Projected Gradient to implement Concept 1 & 2

a formula is available for computing the projected gradient:
projection matrix : P = I-AT(AAT)-1A
projected gradient: cp = Pc

now we are ready to move from (2,2,4)
in the direction of cp to a new point:

 2
2
0 
x  2  4c p  2  4 1 
4
4
 1
 determines how far we move.
large   too close to the boundary
small   more iterations
we have chosen  =0.5, so the new
trial solution move to:
x = ( 2, 3, 3)
 2
 3
 1
P  
 3
 1

 3
1
3
2
3
1

3

1
3
1
 
3
2 
3 


0 
c p  1 
 
  1
An Interior-point Algorithm
Concept 3:
Centering scheme for implementing Concept 3
Why -- The centering scheme keeps turning the direction of
Transform the
the projected gradient to point more nearly toward an
feasible region
optimal solution as the algorithm converges toward this
to place the
solution.
current trail
solution
How -- Simply changing the scale (units) for each of the
near its center,
variable so that the trail solution becomes equidistant from
thereby
the constraint boundaries in the new coordinate system.
enabling a large
Define D=diag{x},
improvement
when concept 2
1
is implemented.
~
xD x
bring x to the center
coordinate
~
x  (1,1,1)
in the new
An Interior-point Algorithm
The rescaled variable are :
Initial trial solution
(x1,x2,x3) =(2,2,4)
1
2

~
x  D 1 x   0

0


0
1
2
0
 x1 

0
 
 x1   2 

x
0   x2    2 
 
x   2 
1  3   x 
 3
4

4
~
In this new coordinate system,
the problem becomes:
~
~
( x1 , x2 , x3 )  (1,1,1)
~
Max Z  2 x1  4 x 2
~
s.t.
~
~
2 x1  2 x 2  4 x 3  8
~
~
~
~
x1 x 2 x 3  0
Summary and Illustration of the Algorithm
Iteration 1. Given the initial
solution (x1,x2,x3) = (2,2,4)
Step1. Let D be the corresponding
diagonal matrix
2 0 0
D  0 2 0 


0 0 4
Step1. given the current
trial solution (x1,x2,…,xn),
set
X is brought to the center (1,1,1) by:
1
2

~
x  D 1 x   0

0

0
1
2
0

0
x 
 1 
0  x2 
 
 x 
1  3 
4 
Summary of the
General Procedure
 x1 
2 
 
 x2  
2
x 
 3
4
1
1
 
1
 x1 0 0
0 x 0
2

D   0 0 x3

... ... ...
 0 0 0
... 0 
... 0 

... 0 

... ...
... xn 
Summary and Illustration of the Algorithm
Iteration 1. (cont.)
Summary of the General
Procedure (cont.)
step2. In these new coordinates,
A and c have become
2 0 0
~
A  AD  [1,1,1]0 2 0  [2,2,4]


0 0 4
2 0 0
 2
~
c  Dc  0 2 0[1,2,0]  4


 
0 0 4
0 
To compute projected
gradient:
Step2. calculate
~
~
A  AD and c  Dc
Summary and Illustration of the Algorithm
Iteration 1. (cont.)
Summary of the General Procedure
(cont.)
st ep3.t heproject edmat rixis
 5
 6
 1
~T ~ ~T
~
-1
P  I - A ( A A ) A  
 6
 1
 3
the projected gradientis
1 
~
c p  P c  3 
 
  2
1
6
5
6
1

3

1
 
3
1
 
3
1 
3 
Compute projected gradient:
st ep3. Calculat e t he
project edmat rix
~T
~ ~T
~
P  I - A (A A ) A
-1
the projected gradient
~
cp  Pc
Summary and Illustration of the Algorithm
Iteration 1. (cont.)
step4. define v as the absolute value of
the negative component of cp having
the largest value, so that v=|-2|=2.
In this coordinate, the algorithm moves
from the current trial solution to the next
one.
5
4
1
1
1   
~

0.5    7 




x  1  cp  1 
3 
 v
  2   4
1
1
 2  
1
 
2
step4. Determine how far to
move by identify v. Then make
move by calculating
1

x  1 
cp
 
v

1

~
Summary and Illustration of the Algorithm
Iteration 1. (cont.)
Step 5. In the original coordinate, the solution is
 x1 
2
~
 x   D x  0
 2



0
 x3 

0
2
0
5
5
 
2
0  4 
 
7
7


0
  
 4 
2
4


2 
 1
 
 
2
 
9
Step 5. Back to the
original coordinate
by calculating
~
xDx
(0,8) optimal
8
Summary of the
General Procedure
(cont.)
7
6
Z=16=x1+2x2
this completes the
iteration 1. The new
solution will be used to
start the next iteration.
x2
5
4
(2.5,3.5)
x1+2x2=8
3
(2,2)
2
1
0
0
1
2
3
4
5
x1
6
7
8
9
Summary and Illustration of the Algorithm
Iteration 2. Given the current trial solution
St ep1. Let Dbe t hecorresponding diagonal mat rix
~
such t hatx  Dx
5
2

D  0

0

0
7
2
0

0

0

2

 13
 18
 7
P  
 18
2

 9
(x1, x2, x3)  (
 11
  12 


133

cp  
 60 
 41


 15
2 
9 
14 

 and
45
37 
45 

7

18
41
90
14

45
5 7
, ,2)
2 2
X is brought to the center (1,1,1,) in the new coordinate, and move to
(0.83, 1.4, 0.5), corresponds (2.08, 4.92,1.0) in the original coordinate
9
(0,8) optimal
8
7
6
Z=16=x1+2x2
5
x2
(2.08, 4.92)
4
(2.5,3.5)
x1+2x2=8
3
(2,2)
2
1
0
0
1
2
3
4
5
x1
6
7
8
9
Summary and Illustration of the Algorithm
Iteration 3. Given the current trial solution x=(2.08, 4.92, 1.0)
0
2.08 0
  1.63
D 0
4.92 0  c p   1.05 




  1.79
0 1.0
 0
X is brought to the center (1,1,1,) in the new coordinate, and move to
( 0.54,1.30,0.50), corresponds (1. 13, 6.37, 0.5) in the original coordinate
9
(0,8) optimal
8
7
(1.13,6.37)
6
Z=16=x1+2x2
5
x2
(2.08, 4.92)
4
(2.5,3.5)
x1+2x2=8
3
(2,2)
2
1
0
0
1
2
3
4
5
x1
6
7
8
9
An Example
Effect of rescaling of each iteration: Sliding the optimal solution toward
(1,1,1) while the other BF solutions tend to slide away.
A
B
C
D
Summary and Illustration of the Algorithm
More iterations. Starting from the current trial solution x, following the
steps, x is moving toward the optimum (0,8). When the trial solution is
virtually unchanged from the proceeding one, then the algorithm has
virtually converged to an optimal solution. So stop.
9
(0,8) optimal
8
7
(1.13,6.37)
6
Z=16=x1+2x2
5
x2
(2.08, 4.92)
4
(2.5,3.5)
x1+2x2=8
3
(2,2)
2
1
0
0
1
2
3
4
5
x1
6
7
8
9
Another Example
The problem
MAX
Z = 5x1 + 4x2
ST
6x1 + 4x2<=24
x1 + 2x2<=6
-x1 + x2<=1
x2<=2
x1, x2>=0
2.5
2.5
2
2
(3,1.5) optimal
(3,1.5) optimal
1.5
1.5
x2
1
1
Z=21=5x1+4x2
Z=21=5x1+4x2
0.5
0.5
0
00
0
0.5
0.5
1
1
1.5
1.5
2
2.5
2 x1 2.5
x1
Augmented form
MAX
Z = 5x1 + 4x2
ST
6x1 + 4x2 + x3=24
x1 + 2x2 + x4=6
-x1 + x2 + x5=1
x2 + x6=2
xj>=0, j=1,2,3,4,5,6
6
1
A 
 1

0
4
2
1
1
1
0
0
0
0
1
0
0
3
3
0
0
0
0
3.5
3.5
4
4
4.5
4.5
0
24
6
0
 b 
0
1

 
1
2
c  5 4 0 0 0 0
Another Example
Starting from an initial solution x=(1,1,14,3,1,1)
D  diag{x}
The trial solution at
each iteration
~
A  AD
~
c  Dc
2.5
~T
~ ~T
~
P  I - A (A A ) A
-1
the projected gradient
~
cp  Pc
1
~

x  1  c p
  v
1
~
x Dx
2
(1.67,1.41)
1.5
(2.23,1.51)
x2
(3,1.5) optimal
(3.05,1.39)
1
Z=21=5x1+4x2
(1,1)
0.5
0
0
0.5
1
1.5
2
2.5
x1
3
3.5
4
4.5
More issues
 Interior-point methods is designed for dealing with
big problems. Although the claim that it’s much faster
than the simplex method is controversy, many tests on
huge LP problems show its outperformance.
 After Karmarkar’s paper, many related methods have
been developed to extend the applicability of
Karmarkar’s algorithm, e.g.
• Infeasible interior points method -- remove the
assumption that there always exits a nonempty
interior.
• Methods applying to LP problems in standard form.
More issues
• Methods dealing with finding initial solution, and
estimating the optimal solution.
•Methods working with primal-dual problems.
• Studies about moving step-long/short steps.
•Studies about efficient implementation and
complexity of various methods.
 Karmarkar’s paper not only started the development
of interior point methods, but also encouraged rapid
improvement of simplex methods.
Reference
[1] N. Karmarkar, 1984, A New Polynomial - Time Algorithm for Linear Programming,
Combinatorica 4 (4), 1984, p. 373-395.
[2] M.J. Todd, (1994), Theory and Practice for Interior-point method, ORSA Jounal on
Computing 6 (1), 1994, p. 28-31.
[3] I. Lustig, R. Marsten, D. Shanno, (1993), Interior-point Methods for Linear
Programming: Computational State of the Art, ORSA Journal on Computing, 6 (1),
1994, p. 1-14.
[4] Hillier,Lieberman,Introduction to Operations Research (7th edition) 320-334
[5] Taha, Operations Research: An Introduction (6th edition) 336-345
[6] E.R. Barnes, 1986, A Variation on Karmarkar’s Algorithm for Sloving Linear
Programming problems, Mathematical Programming 36, 1986, p. 174-182.
[7] R.J. Vanderbei, M.S. Meketon and B.A. Freeman, A Modification of karmarkar’s
Linear Programming Algorithm, Algorithmica: An International Journal in Computer
Science 1 (4), 1986 p. 395 – 407.
[8] D. Gay (1985) A Variant of Karmarkar’s Linear Programming Algorithm for Problems
in Standard form, Mathematical Programming 37 (1987) 81-90
more…
Paper Review of ENGG*6140 Optimization Techniques
Yanrong Hu
Ce Yu
Mar. 11 2003