• Linear programming (linear optimization) is the area of optimization problems with linear objective functions and constraints Example: minimize: subject to:
f
(x) = 6x 1 2x 1 + 8x 3 x 1 – 5x 2 + 5x 2 + x 4 + 2x 3 ≥ 20 – 2x 3 + 3x 4 = + 7x
-
5 4
• None of the variables are multiplied by another variable, raised to a power, or used in a nonlinear function • Because the objective function and constraints are linear, they are convex. Thus, if an optimal solution to an LP problem is found, it is the global optimum
• LP Standard form: minimize:
f
=
cx
subject to:
Ax
=
b
x
i
≥ 0;
i
= 1, …, n where
c
is called the cost vector (1 by n),
x
is the vector of variables (n by 1),
A
is the coefficient matrix (m by n), and
b
is a m by 1 vector of given constants.
• For a maximization problem, we can transform using:
max(f(x))
min(-f(x))
• For inequality constraints, use “slack” variables: 2x 1 + 3x 2 ≤ 5 2x 1 + 3x 2 + s 1 = 5 where s 1 ≥ 0
When we transform the equation 2x 1 + 3x 2 ≤ 5 to 2x 1 + 3x 2 + s 1 = 5 If the left-hand side (LHS) ( 2x 1 + 3x 2 ) is less than the right-hand side (RHS) (5) , then s 1 will take a positive value to make the equality true. The nearer the value of the LHS is to the RHS, the smaller the value of s 1 s 1 is. If the LHS is equal to the RHS, = 0 . s 1 cannot be negative because the LHS cannot be greater than the RHS.
Example: Write in Standard Form: maximize: subject to:
f
= x 2x 1 1 + x 2 + 3x 2 ≤ 6 x 1 + 7x 2 ≥ 4 x 1 + x 2 = 3 x 1 Define slack variables ≥ 0, x 2 x 3 ≥ 0 ≥ 0 & x 4 ≥ 0
The problem now can be written: minimize:
g
= –x 1 – x 2 subject to: 2x 1 + 3x 2 + x 3 x 1 + 7x 2 – x 4 = 6 = 4 x 1 x 1 ≥ 0, x 2 + x 2 ≥ 0, x 3 ≥ 0, x 4 ≥ 0 = 3
A
2 1 1 3 7 1 1 0 0 0 0 1
b
6 4 3
c
1 1 0 0
• The next few slides review several concepts from linear algebra that are the basis of the methods used to solve linear optimization problems
• Vectors – A k-vector is a row or column array of k numbers. It has a dimension of k.
• Linear Independence (LI) – A collection of vectors
a
1 ,
a
2 , …,
a
k
, each of dimension n , is called linearly independent if the equation 0
j k
1
j
a
j
means that
j
0 for
j
=1, 2, …,
k
• In other words, a set of vectors is linearly independent if one vector cannot be written as a combination of any of the other vectors.
• The maximum number of LI vectors in a n dimensional space is n .
For example, in a 2-dimension space: The vectors
x
1 4 5 &
x
2 20 25 are not linearly independent because
x
2 = 5
x
1 .
x
1 0 2 &
x
2 3 1 are LI because there is no constant you can multiply one by to get the other.
• A set of vectors
a
1 ,
a
2 , …,
a
k
in a n dimensional space is said to span the space if any other vector in the space can be written as a linear combination of the vectors • In other words, for any vector
b
, there must exist scalars 1 , 2 , …,
k
such that
b
j k
1
j
a
j
• • • A set of vectors is said to be a basis for a n -dimensional space if: 1. The vectors span the space 2. If any of the vectors are removed, the set will no longer span the space A basis of a n -dimensional space must have exactly n vectors There may be many different bases for a given space
• An example of a basis is the coordinate axis of a graph. For a 2-D graph, you cannot remove one of the axes and still form any line with just the remaining axis. • Or, you cannot have three axes in a 2-D plot because you can always represent the third using the other two.
• Linear Algebra can be used to solve a system of equations
Example:
2x 1 + 4x 2 = 8 & 3x 1 – 2x 2 = 11 This can be written as an augmented matrix: [
A
,
b
] 2 3 4 2 8 11
• Row operations may be performed on the matrix without changing the result • Valid row operations include the following: – Multiplying a row by a constant – Interchanging two rows – Adding one row to another
• In the previous example, we want to change the A matrix to be upper triangular 2 3 4 2 8 11 multiply top row by ½ 1 3 2 2 4 11 add -3 times the top row to the bottom row 1 0 2 8 4 1
1 0 2 8 4 1 multiply bottom row by -1/8 1 0 2 1 4 1 8 1
x
1 0
x
1 2
x
2 1
x
2 4 1 8 • From the upper triangular augmented matrix, we can easily see that use this to get x 1 x 2 = 1 / 8 and x 1 = 4 – 2 . 1 / 8 = 15 / 4
• The inverse of a matrix can be found by using row operations
Example: A
2 1 1 1 2 1 1 1 2 Form the augmented matrix (
A
,
I
) Transform to (
I
,
A
-1 ) using row operations 1 0 0 0 1 0 0 0 1 5 12 3 12 1 12 2 1 1 3 12 3 12 3 12 1 2 1 1 12 3 12 5 12 1 1 2 1 0 0 0 1 0 0 0 1
• We have seen that the constraints can be written in the form
Ax
b
.
• We should have more variables than equations so that we have some degrees of freedom to optimize.
– If the number of equations are more than or equal to the number of variables, the values of the variables are already specified.
• Given a system of equations in the form
Ax
b
– Assume m (number of equations) < n of variables) underspecified system (number • We can split the system into (n-m) independent variables and (m) dependent variables. The values of the dependent variables will depend on the values we choose for the independent variables.
• We call the dependent variables the basic variables because their A-matrix coefficients will form a basis. The independent variables will be called the nonbasic variables. • By changing the variables in the basis, we can change bases. It will be shown that this allows examining different possible optimum points.
Separate the
A
matrix in the following way:
a
1
x
1 ...
a
m x m
...
a
n x n
b
Or,
i m
1
a
i x i
j n
m
1
a
j x j
b
Define matrices
B B
a
1
a
2 ...
&
N a
m
as the following:
N
a
m
1
a
m
2 ...
a
n
where
B
is a m m ) matrix, &
a
j
by m matrix,
N
is the j th is a m by ( column of the
A
n matrix •
B
is called the “basic matrix” called the “nonbasic matrix” and
N
is
• The
B
matrix contains the columns of the
A
-matrix that correspond to the x -variables that are in the basis. Order must be maintained.
– So, if x 4 is the second variable of the basis, must be the second column of the B-matrix
a
4 • The
N
matrix is just the columns of the
A
matrix that are left over.
Similarly, define
x
B
[
x
1
x
2 ...
x m
]
T
&
x
N
[
x m
1
x m
2 ...
x n
]
T
We will see later how to determine which variables to put into the basis. This is an important step to examine all possible optimal solutions.
Now, we have
Bx
B
Nx
N
b
Multiply both sides by
B
-1 :
x
B
B
1
Nx
N
B
1
b
So,
x
(
x
B
x
N
) (
B
1
b
B
1
Nx
N
x
N
)
• We can choose any values for ( n-m ) variables (the ones in
x
N
) and then solve for the remaining • If we choose
x
N
= m
0
variables in
x
B
, then
x
B
B
1
b
This is called a “basic solution” to the system Basic Solution:
x
(
x
B
x
N
) (
B
1
b 0
)
Now we have a solution to
Ax
=
b
. But that was just one of two sets of constraints for the optimization problem. The other was:
x i
≥ 0,
i
= 1, …, n (non-negativity) • A basic feasible solution (BFS) is a basic solution where every
x
is non-negative A BFS satisfies all of the constraints of the optimization problem
• A point is called an extreme point (EP) if it cannot be represented as a strict ( 0 < < 1) convex combination of two other feasible points.
• Remember: a convex combination of two points is a line between them.
• So, an EP cannot be on a line of two other feasible points.
• Given a feasible region, an extreme point cannot lie on a line between two other feasible points (it must be on a corner)
Feasible Region Not Extreme Points Extreme Point
• In a n -dimensional space, an extreme point is located at the intersection of n constraints
c
Starting Point • We have a maximization problem, so we want to go as far in the direction of the
c
(objective function) vector as we can • Can we determine anything about the location of the optimum point?
c
• If we start on a line, we can move along that line in the direction of the objective function until we get to a corner • In fact, for any
c
vector, the optimum point will always be on a corner
• In a n-dimensional space, a BFS is formed by the intersection of n equations.
• In 2-D:
Basic Feasible Solution Constraint 1 Constraint 2
• But, we just saw that an extreme point is also a corner point. So, a BFS corresponds to an EP.
• We just saw that a basic feasible solution corresponds to an extreme point.
• This is very important because for LP problems, the optimum point is always at an extreme point.
• Thus, if we can solve for all of the BFS’s (EP’s), we can compare them to find the optimum.
Unfortunately, this takes too much time.
• The simplex method is the most common method for solving LP problems.
• It works by finding a BFS; determining whether it is optimal; and if it isn’t, it moves to a “better” BFS until the optimal is reached.
• This way, we don’t have to calculate every solution.
Recall:
x
B
B
1
Nx
N
B
1
b
Sum over all non basic variables
x
B
j
N
B
1
a
j x j
B
1
b Objective Function:
f
cx
c
B
x
B
c
N
x
N
substitute
x
B
B
1
b
j
N
B
1
a
j x j
into above equation:
f
c
B
(
B
1
b
j
N
B
1
a
j x j
)
j
N c j x j
Multiply through and collect x
j
terms:
f
c
B
B
1
b
j
N
(
c j
c
B
B
1
a
j
)
x j
f
c
B
B
1
b
j
N
(
c j
z j
)
x j
where
z j
c
B
B
1
a
j
So, minimize
f
c
B
B
1
b
j
N
(
c j
z j
)
x j
Subject to:
x
B
j
N
B
1
a
j x j
B
1
b
If (c
j
– z
j
) ≥ 0 for all
j
N
, then the current BFS is optimal for a minimization problem.
Because, if it were < 0 for some
j
, that nonbasic variable, x
j
, could enter the basis and reduce the objective function.
• A nonbasic variable may enter the basis and replace one of the basic variables • Since
x
N
=
0
, and we have non-negativity constraints, the entering variable must increase in value.
• The entering variable’s value will increase, reducing the objective function, until a constraint is reached.
• The equation to determine which variable enters is:
c j
c
T B
B
1
a
j
c j
z j
. Calculate for all nonbasic indices
j
• For a minimization problem, choose the index
j
– If c
j
- z
j
for which c
j
- z
j
is the most negative ≥ 0 for all
j
, the solution is optimal • For a maximization problem, choose the index
j
– If c
j
- z
j
for which c
j
- z
j
is the most positive ≤ 0 for all
j
, the solution is optimal
• As the value of the entering variable increases, the value of at least one basic variable will usually decrease – If not, the problem is called “unbounded” and the value of the minimum objective function is • The variable whose value reaches zero first will be the variable that leaves the basis
x 1
x 2 , x 3
x 4
x 4 x 3 x 2 x 1 As soon as x 2 reaches zero, we must stop because of the non-negativity constraints. But, now x 2 = 0 , so it is a nonbasic variable and x 1 > 0 , so it is a basic variable. So, basis & x
1
enters the basis.
x 2 leaves the
• Let
j
be the index of the variable that is entering the basis and
i
* be the index of the variable that is leaving the basis
i
* argmin ( (
B B
1
b
)
i
1
a j
)
i
(
B
1
a j
)
i
0 Meaning, for every index and has (
B
1
a
j
)
i
0
i
that is in the basis ( (
B B
1
b
1
a
) , calculate . The
j i
)
i
index of the value that is the minimum is the index of the leaving variable.
The previous expression is obtained from the equation:
x
B
B
1
b
B
1
a
j x j
0
which applies when a constraint is reached
x 4 x 3 x 2 x 1 • x 2 , x 3 , & x 4 start out at (
B
-1
b
) i ; (i=2, 3, 4) slopes of (–
B
-1
a
j ) i ; ( i=2, 3, 4) where j=1 and have because 1 is the index of the entering variable ( x 1 ) • Thus, the distance we can go before a basic variable reaches zero is But, if (
B
-1
a
1 ) i < 0 (
B
1
b
)
i
(
B
1
a
1 )
i
for
B
-1
a
1 > 0 (like x 3 ), it won’t ever reach . zero.
x 4 x 3 x 2 x 1 • We can also see how if none of the variables decreased, we could keep increasing x 1 and improving the objective function without ever reaching a constraint –This gives an unbounded solution
Minimize f = -x 1 Subject to: – x 2 x 2x x 1 1 1 + x 2 – x 2 ≤ 5 ≤ 4 ≤ 3 ; x 1 , x 2 ≥ 0 Given: The starting basis is x 1 , x 2 , & x 3 .
Insert slack variables x 3 , x 4 , & x 5 .
Minimize f = -x 1 – x 2 Subject to: x 2x 1 1 + x 2 – x 2 + x 3 x 1 = 5 + x 4 = 4 x 5 = 3 x 1 , x 2 , x 3 , x 4 , x 5 ≥ 0 1
A
2 1 1 1 0 1 0 0 0 1 0 0 0 1 5
b
4 3
c
1 1 0 0 0
st
B
a
1
a
2
a
3 2 1 1 1 1 0 1 0 0
B
1 1 2 1 1 1 0 1 0 0 1 0 0 1 0 1 1 1 2 3
x
B
0
B
1
b
0 1 0 1 1 1 2 3 5 4 3 3 2 0
f
c
B
x
B
1 1 0 3 2 0 5
Now, check optimality x 4 :
c
4
c
B
B
1
a
4 0 1 1 0 0 0 1 0 1 1 1 2 3 0 1 0 0 1 1
< 0
x 5 :
c
5
c
B
B
1
a
5 0 1 1 0 0 0 1 0 1 1 1 2 3 0 0 1 0 ( 3 ) 3
> 0
So, x 4 enters the basis since its optimality indicator is < 0 .
(
B
1
b
) 3 2 0 (
B
1
b
) 1 3 (
B
1
b
) 2 2 & (
B
1
b
) 3 0
j
4
B
1
a
j
B
1
a
4 0 0 1 0 1 1 1 2 3 0 1 0 0 1 1
(
B
1
b
) 1 3 (
B
1
b
) 2 2 (
B
1
b
) 3 0
i
* (
B
1
a
4 ) 1 argmin 0 ( (
B B
1
b
)
i
1
a j
)
i
(
B
1
a
4 ) 2 0 (
B
1
a j
)
i
1 (
B
1
a
4 ) 3 0 argmin 0
N
/
A
1
N
/
A
0 1 0 3 So, x 3 is the leaving variable
2 nd Iteration: 1
B
2 1 1 1 0 0 1 0
x
B
0
B
1
b
1 1 0 0 1 1 1 3 5 4 3 3 2 0
a 4 has been substituted for a 3
0
B
1 1 1 0 0 1 1 1 3
f
c
B
x
B
1 1 0 3 2 0 5
Optimality Indicators: x 3 :
c
3
c
B
B
1
a
3 0 1 1 0 0 1 1 0 0 1 1 1 3 1 0 0 0 ( 1 ) 1 0 x 5 :
c
5
c
B
B
1
a
5 0 1 1 0 0 1 1 0 0 1 1 1 3 0 0 1 0 0 0 0
All of the optimality indicators are ≥ 0, so this is the optimal solution.
So,
f
* 5
x
*
B x
1
x
2
x
4 3 2 0
x
*
N
x x
5 3 0 0
1. With the chosen basis, get
B
and solve
x
B
j
. =
B
-1
b
2. Calculate and c
j f
– z
j
=
c
B
x
B .
for all nonbasic variables, – For a min. problem, if all c
j
– z
j
’s are ≥ 0, the current solution is optimal. If not, choose the index with the most negative c
j
– z
j
.
– For a max. problem, if all c
j
– z
j
’s are ≤ 0, the current solution is optimal. If not, choose the index with the most positive c
j
– z
j
.
3. Using the equation
i
* argmin choose the leaving variable.
( (
B
1
b
)
i B
1
a j
)
i
(
B
1
a j
)
i
– If all (
B
-1
a
j
) i ’s are ≤ 0, then the solution is unbounded 0 4. Let x
j
enter the basis and x
i*
leave the basis. Obtain the new B matrix and start again with step 1.
• • In the example, we were given a starting basis. How to come up with one on our own?
Case #1: max (or min) problem with 1.
Ax
≤
b
(all ≤ inequalities) and 2. all entries of the
b
vector ≥ 0.
Insert slack variables into constraint equations and use the resulting identity matrix as the starting basis
Let
s
= vector of slack variables The problem will become max min
f
cx
0s
Subject to Where
I
=
Ax x
0
,
Is s
b 0
The Identity Matrix 1 0 0 0 0 1 ...
...
0 0 0 0 ...
...
1 0 0 0 0 1
Choose the slack variables to be the starting basis The starting basis matrix (
B
) is the coefficients of the slack variables. This happens to be the identity matrix.
We can see that the starting basis is feasible (
x
B ≥
0
):
B
I x
B
B
1
b
I
1
b
Ib
b
0
Minimize -x 1 – 3x 2 Subject to 2x 1 + 3x 2 ≤ 6 -x 1 + x 2 ≤ 1 Insert slack variables: x 1 , x 2 ≥ 0 x 1 , x 2 , x 3 , x 4 ≥ 0
A
2 1 3 1 1 0 0 1 2x 1 + 3x 2 + x 3 = 6 -x 1 + x 2 + x 4 = 1
b
6 1
c
1 3 0
Identity Matrix
0 0
Use the slacks as the starting basis:
B
a
3
a
4 1 0 0 1 &
B
1 1 0 0 1 1 1 0 0 1
x
B
B
1
b
1 0 0 1 6 1 6 1
f
c
B
x
B
0 0 6 1 0
Optimality Indicators: j=1:
c
1
z
1
c
1
c B B
1
a
1 1 0 0 1 0 0 1 2 1 1 0 1 j=2:
c
2
z
2
c
2
c B B
1
a
2 3 0 0 1 0 0 1 3 1 3 0 3 c 2 - z 2 is the most negative, so x 2 basis enters the
(
B
1
b
) 6 1 (
B
1
b
) 3 6 & (
B
1
b
) 4 1 x 2 is entering the basis
j
2
B
1
a
j
B
1
a
2 1 0 0 1 3 1 3 1
(
B
1
b
) 3 6 (
B
1
b
) 4 1 (
B
1
a
2 ) 3 3 (
B
1
a
2 ) 4 1
i
* argmin ( (
B B
1
b
)
i
1
a j
)
i
(
B
1
a j
)
i
0 argmin 6 3 1 1 argmin So, x 4 is the leaving variable.
4
B
2 nd Iteration:
a
3
a
2
x 2
1 0
replaced x 4
3 1
B
1 1 0 3 1 1 1 0 1 3
x
B
B
1
b
1 0 1 3 6 1 3 1
f
c
B
x
B
0 3 3 1 3
Optimality Indicators: j=1:
c
1
z
1
c
1
c B B
1
a
1 1 0 j=4:
c
4
z
4
c
4
c B B
1
a
4 0 0 3 1 0 3 1 0 1 3 2 1 1 3 4 1 3 0 1 0 ( 3 ) 3 So, x 1 enters the basis
Leaving Variable: (
B
1
b
) 3 1 (
B
1
b
) 3 3 & (
B
1
b
) 2 1
j
1
B
1
a
j
1 0 1 3 2 1 5 1 0
i
* argmin ( (
B B
1
b
)
i
1
a j
)
i
(
B
1
a j
)
i
0 argmin 3 5
N
/
A
3 So, x 3 leaves the basis and x 1 replaces it.
3 rd Iteration:
B
a
1
a
2 2 1
x
B
3 1
B
1
b
B
1 2 1 0 0 .
.
2 2 3 1 1 0 0 .
2 .
2 0 .
0 .
6 4 6 1 0 1 .
6 .
6 0 .
0 .
6 4
f
c
B
x
B
1 3 0 1 .
.
6 6 5 .
4
Optimality Indicators: j=3:
c
3
z
3
c
3
c B B
1
a
3 0 1 3 0 0 .
2 .
2 0 0 .
6 .
4 1 0 0 ( 0 .
8 ) 0 .
8 j=4:
c
4
z
4
c
4
c B B
1
a
4 0 1 3 0 0 .
2 .
2 0 0 .
6 .
4 0 1 0 ( 0 .
6 ) 0 .
6 Both c j -z j ’s are ≥ 0, so the current solution is optimal
x
* 0 .
6 1 .
6 0 0
f
* 5 .
4
This graph shows the path taken.
The dashed lines are perpendicular to the cost vector,
c
.
x 2 0 1 .
.
6 6 0 1 0 0 Increasing
c
x 1
• Since we were minimizing, we went in the opposite direction as the cost vector x 2 0 1 .
.
6 6 0 1 0 0 Increasing
c
x 1
• Case #2: max (or min) problem with: 1.
Ax
≥
b
(at least some ≥ constraints) & 2. All entries of the
b
vector ≥ 0 Add slacks to make the problem become
Ax
–
Is
=
b
x, s ≥ 0 .
We cannot do the same trick as before because now we would have a negative identity matrix as our
B
matrix.
• 2-Phase Method: Introduce “artificial variables” (
y
) where needed to get an identity matrix. If all constraints were ≥, the problem will become:
Ax
–
Is
+
Iy
=
b x
,
s
,
y
≥ 0
• Artificial variables are not real variables. • We use them only to get a starting basis, so we must get rid of them.
• To get rid of them, we solve an extra optimization problem before we start solving the regular problem.
Phase 1:
Solve the following LP starting with
B
=
I
and
x
B =
y
= Minimize
y b
: Subject to: If
y
0 Ax
–
Is
+
Iy
=
b
at the optimum, stop
x
,
s
,
y
≥
0
– the problem is infeasible. If
y
=
0
, then use the current basis and continue on to phase 2.
Phase 2:
Using the objective function from the original problem, change the
c
vector and continue solving using the current basis.
Minimize (or Maximize)
cx
Subject to:
Ax
–
Is
=
b x
,
s
≥
0
• Slack variables are real variables that may be positive in an optimum solution, meaning that their constraint is a strict (< or >) inequality at the optimum.
• Artificial variables are not real variables. They are only inserted to give us a starting basis to begin the simplex method. They must become zero to have a feasible solution to the original problem.
x 1 + 2x 2 ≥ 4 -3x 1 + 4x 2 ≥ 5 2x 1 + x 2 ≤ 6 x 1 , x 2
≥ 0 x 1 + 2x 2 – x 3 = 4 -3x 1 + 4x 2 2x 1 + x 2 – x 4 + x 5 = 5 = 6
We can see that we cannot get an identity matrix in the coefficients and positive numbers on the right-hand side. We need to add artificial variables: x 1 + 2x 2 – x 3 + y 1 = 4 -3x 1 + 4x 2x 1 + x 2 2 – x 4 + x 5 + y 2 = 5 = 6
Now we have an identity matrix, made up of the coefficient columns of y 1 , y 2 , & x 5 . We would solve the problem with the objective of minimizing y 1 + y 2 to get rid of the artificial variables, then use whatever basis we had and continue solving, using the original objective function.
x 1 3x 1 + 2x 2 – x 2 – 5x + 3x 3 3 ≥ -4 ≤ 2 -x 1 + x 2 + x 3 = -1
x 1 , x 2 , x 3 ≥ 0 x 1 3x 1 + 2x 2 – x 2 – 5x + 3x 3 3 – x4 = -4 + x5 = 2 -x 1 + x 2 + x 3 = -1
We don’t have to add an artificial variable to the 1 st constraint if we multiply by -1. When we multiply the last constraint by -1 and add an artificial variable, we have: -x 1 – 2x 2 + 5x 3 + x 4 x 1 , x 2 3x 1 x 1 – x 2 – x 2 + 3x – x 3 3 , x 3 , x 4 , x 5 , y 1 ≥ 0 + x 5 = 4 = 2 + y 1 = 1
So, after adding slacks, we must make the right-hand side numbers positive. Then we add artificial variables if we need to.
• Consider the problem: Maximize x 1 + 8x 2 Subject to: x 1 -x 1 + x 2 ≥ 1 + 6x 2 ≤ 3 x 2 ≤ 2 x 1 , x 2 ≥ 0
Insert slacks: x 1 + x 2 – x 3 -x 1 + 6x 2 x 2 + x 4 + x 5 = 1 = 3 = 2 So, we need an artificial variable in the 1 st constraint.
Insert artificial variable: x 1 -x 1 + x 2 – x 3 + 6x 2 + x 4 x 2 + x 5 + y 1 = 1 = 3 = 2
Minimize y 1 Subject to: x 1 -x 1 + x 2 – x 3 + 6x 2 x 2 + x 4 + x 5 + y 1 = 1 = 3 = 2 Our starting basis is: y 1 , x 4 , & x 5 .
B c
0
a
6
a
4 0
x
B
B
1
b
1 0 0 0 0
a
5 1 0 0 0 1 0 0 0 1 0 1 0 0 1 0 0 1 1 3 2 1 3 2 &
B
1 1 0 0
b
0 1 0 1 3 2 0 0 1 1 1 0 0
f
c
B
x
B
1 0 0 1 3 2 0 1 0 1 0 0 1
Optimality Indicators: j=1:
c
1
z
1
c
1
c B B
1
a
1 0 1 j=2:
c
2
z
2
c
2
c B B
1
a
2 0 1 j=3:
c
3
z
3
c
3
c B B
1
a
3 0 1 0 0 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1 0 0 1 0 0 1 0 0 0 1 1 0 1 0 1 1 0 0 1 1 6 1 0 0 1 0 1 0 0 0 1 1 1 1
It’s a tie between x 1 the basis (
B
1
b
) 1 3 2 (
B
1
b
) 6 & 1 x 2 – pick (
B
1
b
) 4 x 1 3 to enter & (
B
1
b
) 5 2 x 1 is entering the basis
j
1
B
1
a
j
B
1
a
1 1 0 0 0 1 0 0 0 1 1 0 1 1 0 1
(
B
1
b
) 6 1 (
B
1
b
) 4 3 (
B
1
b
) 5 2 (
B
1
a
2 ) 6 1 0 (
B
1
a
2 ) 4 1 0 (
B
1
a
2 ) 5 0 0
i
* argmin ( (
B
1
b
)
i B
1
a j
)
i
(
B
1
a j
)
i
0 argmin 1 1
N
/
A N
/
A
6 So, x 1 replaces y 1 in the basis
B
a
1
a
4
a
5 1 0 1 0 1 0 0 0 1
B
1 1 0 1 0 1 0 0 0 1 1 1 1 0 0 1 0 0 0 1
x
B
B
1
b
1 1 0 0 1 0 0 0 1 1 3 2 1 4 2
f
c
B
x
B
0 0 0 1 4 2 0
Optimality Indicators: j=2:
c
2
z
2
c
2
c B B
1
a
2 0 0 0 0 1 1 0 0 1 0 0 0 1 1 6 1 0 0 0 j=3:
c
3
z
3
c
3
c B B
1
a
3 0 0 j=6:
c
1
z
1
c
1
c B B
1
a
1 0 0 0 0 0 1 1 0 0 1 1 0 0 1 0 0 1 0 0 0 1 0 1 0 0 0 0 0 0 1 1 0 1 0 0 0
All of the optimality indicators are ≥ 0, so this is an optimum solution.
So, we keep this basis and change the objective function to the original one: Maximize –x 1 + 8x 2 Our basis is still x 1 , x 4 , & x 5 .
Back to original problem:
B
1 1 1 0 0 1 0 0 0 1
c
1 8 0 0 0 The basis remains the same
f
c
B
x
B
1 0 0 1 4 2 1
x
B
B
1
b
1 4 2
Optimality Indicators: j=2:
c
2
z
2
c
2
c B B
1
a
2 8 1 0 0 1 1 0 0 1 0 0 0 1 1 6 1 8 1 7 j=3:
c
3
z
3
c
3
c B B
1
a
3 0 1 0 0 1 1 0 0 1 0 0 0 1 0 1 0 0 1 1 Since we are maximizing now, we want the most positive. So, x 2 enters the basis.
(
B
1
b
) 1 4 2 (
B
1
b
) 1 1 (
B
1
b
) 4 4 & (
B
1
b
) 5 2 x 2 is entering the basis
j
2
B
1
a
j
B
1
a
2 1 1 0 0 1 0 0 0 1 1 6 1 1 7 1
(
B
1
b
) 5 1 (
B
1
a
2 ) 1 1
(
B
1
b
) 4 4 (
B
1
a
2 ) 4 7 (
B
1
b
) 5 2 (
B
1
a
2 ) 5 1
i
* argmin ( (
B
1
b
)
i B
1
a j
)
i
(
B
1
a j
)
i
0 argmin 1 1 4 7 2 1 4
Minimum
So x 4 leaves the basis
B
a
1
a
2
a
5 1 0 1 1 6 1 0 0 1
B
1 0 .
856 0 .
143 0 .
143
x
B
B
1
b
0 .
0 .
143 0 .
856 143
f
0 .
143 0 .
143 0 .
143
c
B
x
B
1 8 0 0 1 1 3 2 0 .
0 1 .
429 .
571 429 0 0 .
429 0 .
571 1 .
429 4 .
143 0 .
143 0 .
143 0 .
143 0 0 1
Optimality Indicators: j=3:
c
3
z
3
c
3
c B B
1
a
3 0 1 0 0 0 .
857 0 .
143 0 .
143 0 .
143 0 .
143 0 .
143 0 0 1 0 1 0 0 .
857 j=4:
c
4
z
4
c
4
c B B
1
a
4 0 1 0 0 0 .
857 0 .
143 0 .
143 0 .
143 0 .
143 0 .
143 0 0 1 0 1 0 0 0 0
All of the optimality indicators are ≤ 0, so this is the optimum solution:
x
* 0 .
429 0 1 .
.
571 0 0 429
f
* 4 .
143
• The Karush-Kuhn-Tucker (KKT) Conditions can be used to see optimality graphically • We will use them more in nonlinear programming later, but we can use a simplified version here
• Change the constraints so that they are all ≥ constraints.
• The optimum point is the point where the gradient of the objective function lies within the cone formed by the vectors normal to the intersecting constraints.
• Reminder: – The gradient ( ) of a function
f
variables is calculated: with
n
f
f
x
1 ,
x f
2 , ...
f
x n
Example:
f f
3 (
x
1 ) 2 6
x
1 , 5
x
3 , 5
x
2
x
3 5
x
2
•
: In the example problem #2, we had the problem: Minimize
f
Subject to: = -x 1 – 3x 2 2x -x 1 1 + 3x 2 ≤ 6 + x 2 ≤ 1 x 1 , x 2 ≥ 0 The gradient of the cost function, x 1
f
c
1 , 3 – 3x 2 is:
Previously, we saw that this problem looks like: x 2 Constraint 2
( 3 / 5 , 8 / 5 ) Extreme Points (0, 1)
Constraint 1
(0, 0) (3, 0)
x 1
Change to constraints to all ≥: g g g g 1 2 3 4 : : : : -2x 1 x 1 – 3x 2 – x 2 ≥ -1 ≥ -1 x 1 ≥ 0 x 2 ≥ 0
The gradients of the four constraints (counting the non-negativity constraints), g 1 , …, g 4
g
1 2 , are: 3
g
3 1 , 0
g
2 1 , 1
g
4 0 , 1
The graph of the problem with the normals of the constraints becomes: x 2 Constraint 2
( 3 / 5 , 8 / 5 )
The gradient corresponding to each constraint ( g
i
) is perpendicular to the constraint
i
.
g 1
g 3
g 2 (0, 1)
Constraint 1
g 2
g 4 (0, 0)
g 3
g 4 (3, 0)
g 1
x 1
c
= (-1, -3) looks like this: So, whichever cone this vector fits into corresponds to the optimal extreme point.
x 2
( 3 / 5 , 8 / 5 ) Doesn’t Fit (0, 1)
g 1
g 3
g 2
g 4 (0, 0) Doesn’t Fit
g 3
g 2 It Fits!
Doesn’t Fit
g 1
g 4 (3, 0)
x 1
• So, we get the same optimal point as when we used the simplex method • This can also be done for problems with three variables in a 3-D space • With four or more variables, visualization is not possible and it is necessary to use the mathematical definition
Given a LP minimization problem: Modify the constraints so that we have:
g i
(
x
) 0 ,
i
1
m
Where
g i
(
x
) is the linear constraint equation i . The b
i
that was on the right side of the inequality sign is moved to the left side and included in
g i
.
If there exists a solution for
x
* the conditions below, then
x
* & the
i
’s for is the global optimum
f
c
1
g
1 (
x
* )
m
g m
(
x
* ) Equation 1
i
g i
(
x
* ) 0 ,
g i
(
x
* ) 0 ,
i
0 ,
i
1
m i
1
m i
1
m
Equation 2 Equation 3 Equation 4
• Equation 1 mathematically states that the objective function vector must lie inside the cone formed by the vectors normal to the active constraints at the optimal point
• Equation 2 forces
i
to be zero for all of the inactive constraints – called the “complementary slackness” condition – If the constraint is active, g
i
(
x
* ) = 0 , so
i
be positive and g
i
may will be part of the cone in Equation 1 . – If the constraint is inactive, g
i
(
x
* ) must be zero. g
i
0 , so
i
will not be included in the cone in Equation 1 because it will be multiplied by zero.
• Equation 3 ensures that
x
* is feasible • Equation 4 ensures that the direction of the cone is correct.
– If the
i
’s were negative, the cone would be in the opposite direction. So, this equation prevents that from happening.
• The KKT Conditions are not very useful in solving for optimal points, but they can be used to check for optimality and they help us visualize optimality • We will use them frequently when we deal with nonlinear optimization problems in the next section
• There are many software programs available that will solve LP problems numerically • Microsoft Excel is one program that solves LP problems – To see the default Excel examples for optimization problems, search for and open the file “solvsamp.xls” (it should be included in a standard installation of Microsoft Office)
Let’s solve the first example problem in this chapter with Excel The problem was: Minimize f = -x 1 Subject to: – x 2 x 2x x 1 1 1 + x 2 – x 2 ≤ 5 ≤ 4 ≤ 3 ; x 1 , x 2 ≥ 0
Here is the Excel spreadsheet with the necessary data:
x1
0
x2
0 limit
Objective Function:
value =-A2-B2
Constraint 1: Constraint 2: Constraint 3:
=A2+B2 =2*A2-B2 =A2 5 4 3 In the spreadsheet, A2 is the cell reference for x 1 & B2 is the reference for x 2
x1
0
Constraint 1: Constraint 2: Constraint 3: x2
0
Objective Function:
value =-A2-B2 =A2+B2 =2*A2-B2 =A2 limit 5 4 3 You can see that under the “value” heading for the constraints & objective function, we simply use the given functions to calculate the value of the function
x1
0
x2
0
Objective Function:
value =-A2-B2 limit
Constraint 1: Constraint 2: Constraint 3:
=A2+B2 =2*A2-B2 =A2 5 4 3 On the right side of the constraints, in the “limit” column, we write the value of the “ b
i
” for that constraint Obviously, the objective function doesn’t have a limit
So, the spreadsheet looks like this:
• Now, we need to use the Excel solver feature • Look under the Tools menu for “solver” – If it is not there, go to “Add-Ins” under Tools and select the Solver Add-In
The Solver toolbox should look something like this:
• This is a minimization problem, so select “Min” and set the target cell as the objective function value • The variables are x 1 & x 2 , so in the “By Changing Cells” box, select A2 & B2
• Now add the constraints: – For the “Cell Reference,” use the value of the constraint function and for the “Constraint,” use the number in the Limit column – The constraints are all ≤, so make sure that “<=“ is showing between the Cell Reference and Constraint boxes
• Now, the Solver window should look like this:
• Finally, click the Options button • All of the variables are specified as being positive, so check the “Assume Non Negative” box
• Since this is an LP problem, check the “Assume Linear Model” box • Finally, the default tolerance of 5% is usually much too large. Unless the problem is very difficult, a tolerance of 1% or even 0.1% is usually fine
• Click “Solve” and the Solver Results box should appear • Under “Reports,” select the Answer Report and click OK • A new worksheet that contains the Answer Report is added to the file
• The spreadsheet with the optimum values should look like this:
• The values for x 1 & x 2 are the same as when we solved the problem using the simplex method • Also, if you look under the Answer Report, you can see that all of the slack variables are equal to zero, which is also what we obtained with the simplex method
Let’s solve one more LP problem with Excel: Maximize Subject to: 5x 1 – 2x 2 + x 3 2x 2x x 1 1 1 , x + 4x + x 2 2 ≥ 0 2 + x 3 ≤ 6 + 3x 3 ≥ 2 x 3 unrestricted in sign
Entering the equations into the spreadsheet should give:
x1
0
x2
0
Objective Function: Constraint 1: Constraint 2: x3
0
Value:
=5*A2-2*B2+C2 =2*A2+4*B2+C2 =2*A2+B2+3*C2
Limit:
6 2
Unlike last time, all the variables are not specified as being positive, so we cannot use the “Assume Non-negative” option for all of the variables. So we have to manually specify x 1 & x 2 non-negative by adding two more constraints to be
Now, the formulas in the spreadsheet should look like this:
x1
0
x2
0
Objective Function: Constraint 1: Constraint 2: Constraint 3: Constraint 4: x3
0
Value:
=5*A2-2*B2+C2 =2*A2+4*B2+C2 =2*A2+B2+3*C2 =A2 =B2
Limit:
6 2 0 0
Now, open the solver toolbox and specify: • The Target Cell, • The range of variable cells, • Maximization problem • The constraints – The first is ≤ and the rest are ≥ constraints.
Click the Options button and check the “Assume Linear Model” box.
Remember, since x 3 do not is unrestricted in sign, check the “Assume Non-negative” box You can reduce the tolerance if you would like
The Solver window should look like this:
After solving, the spreadsheet should look like this:
• Notice that because x 3 was unrestricted in sign, it was able to have a negative value and this improved the solution • To see how much of a difference this made in the solution, re-solve the problem with the “Assume Non-negative” option selected
• From these examples, you can see that Excel can be an efficient tool to use for solving LP optimization problems • The method for solving the problems that was outlined here is obviously just one way and the user should feel free to experiment to find their own style
• Linear Programming and Network Flows; Bazaraa, Mokhtar; John Jarvis; & Hanif Sherali. • Optimization of Chemical Processes 2 nd Ed.; Edgar, Thomas; David Himmelblau; & Leon Lasdon.
• Pollution Prevention Through Process Integration; El-Halwagi, Mahmoud