Transcript Linear Programming (Optimization)

Chap 2. The Geometry of LP
 In the text, polyhedron is defined as 𝑃 = {𝑥 ∈ 𝑅𝑛 : 𝐴𝑥 ≥ 𝑏}. So some of our
earlier results should be taken with modifications.
 Thm 2.1:
(a) The intersections of convex sets is convex.
(b) Every polyhedron is a convex set.
(c) Convex combination of a finite number of elements of a convex set also
belongs to that set.
(recall that S closed for convex combination of 2 points.
 S closed for convex combination of a finite number of points)
(d) Convex hull of a finite number of vectors (polytope) is convex.
Linear Programming 2012
1
(a) Let 𝑥, 𝑦 ∈ 𝑖∈𝐼 𝑆𝑖 , 𝑆𝑖 convex  𝑥, 𝑦 ∈ 𝑆𝑖 , ∀ 𝑖
𝜆𝑥 + 1 − 𝜆 𝑦 ∈ 𝑆𝑖 , ∀ 𝑖 since 𝑆𝑖 convex
𝜆𝑥 + 1 − 𝜆 𝑦 ∈ 𝑖∈𝐼 𝑆𝑖 
𝑖∈𝐼 𝑆𝑖 is convex.
Halfspace {𝑥: 𝑎′ 𝑥 ≥ 𝑏} is convex.
Polyhedron 𝑃 is intersection of halfspaces  From (a), P is convex.
( or we may directly show 𝐴(𝜆𝑥 + 1 − 𝜆 𝑦) ≥ 𝑏.)
(c) Use induction. True for 𝑘 = 2 by definition.
Suppose statement holds for 𝑘 elements. Suppose 𝜆𝑘+1 ≠ 1.
Pf)


(b)
Then
𝑘+1
𝑖
𝑖=1 𝜆𝑖 𝑥
= 𝜆𝑘+1 𝑥 𝑘+1 + (1 − 𝜆𝑘+1 )(
𝜆𝑖
𝑘
𝑖=1 1−𝜆
𝑘+1
𝑥𝑖)
𝜆𝑖 /(1 − 𝜆𝑘+1 ) ≥ 0 and sum up to 1, hence
𝑘
𝑖=1(𝜆𝑖 /(1 −
(d) Let 𝑆 be the convex hull of
 𝑦=
𝑘+1
𝑖
𝑖=1 𝜆𝑖 𝑥 ∈ 𝑆
vectors 𝑥 1 , … , 𝑥 𝑘
𝜆𝑘+1 ))𝑥 𝑖 ∈ 𝑆 
𝑘
𝑖
𝑖=1 𝜉𝑖 𝑥
𝜆𝑦 + 1 − 𝜆 𝑧 = 𝜆
, 𝑧=
𝑘
𝑖=1 𝜉𝑖
𝑘
𝑖
𝑖=1 𝜃𝑖 𝑥
𝑥𝑖 + 1 −
and 𝑦, 𝑧 ∈ 𝑆
for some 𝜉𝑖 , 𝜃𝑖 .
𝜆
𝑘
𝑖
𝑖=1 𝜃𝑖 𝑥
=
𝑘
𝑖=1(𝜆𝜉𝑖
+ 1 − 𝜆 𝜃𝑖 )𝑥 𝑖
𝜆𝜉𝑖 + (1 − 𝜆)𝜃𝑖 ≥ 0 and sum up to 1  convex combination of 𝑥 𝑖 ′𝑠
 𝜆𝑦 + 1 − 𝜆 𝑧 ∈ 𝑆.
Linear Programming 2012

2
Extreme points, vertices, and b.f.s’s
 Def: (a) Extreme point ( as we defined earlier)
(b) 𝑥 ∈ 𝑃 is a vertex if ∃ 𝑐 ∈ 𝑅𝑛 such that 𝑐 ′ 𝑥 < 𝑐 ′ 𝑦, ∀ 𝑦 ∈ 𝑃 and 𝑦 ≠ 𝑥.
( 𝑥 is a unique optimal solution of min 𝑐 ′ 𝑥, 𝑥 ∈ 𝑃 )
(c) Consider polyhedron 𝑃 and 𝑥 ∗ ∈ 𝑅𝑛 . Then 𝑥 ∗ is a basic solution if all
equality constraints are active at 𝑥 ∗ and ∃ 𝑛 linearly independent active
constraints among the constraints active at 𝑥 ∗ .
( basic feasible solution if 𝑥 ∗ is basic solution and 𝑥 ∗ ∈ 𝑃 )
 Note: Earlier, we defined the extreme point same as in the text.
Vertex as 0-dimensional face ( dim(𝑃) + rank 𝐴= , 𝑏 = = 𝑛 ) which is the same
as the basic feasible solution defined in the text.
We defined basic solution (and b.f.s) only for the standard LP. (𝑥𝐵 = 𝐵−1 𝑏,
𝑥𝑁 = 0)
Definition (b) is new. It gives an equivalent characterization of extreme point.
(b) can be extended to characterize a face 𝐹 of 𝑃.
Linear Programming 2012
3
𝑥3
𝑥2
A
P
C
E
𝑥1
D
B
 Fig. 2.6: 𝑃 = { 𝑥1 , 𝑥2 , 𝑥3 : 𝑥1 + 𝑥2 + 𝑥3 = 1, 𝑥1 , 𝑥2 , 𝑥3 ≥ 0}
Three constraints active at A, B, C, D. Only two constraints active at E.
Note that D is not a basic solution since it does not satisfy the equality
constraint. However, if 𝑃 is given as 𝑃 = { 𝑥1 , 𝑥2 , 𝑥3 : 𝑥1 + 𝑥2 + 𝑥3 ≥
1, 𝑥1 + 𝑥2 + 𝑥3 ≤ 1, 𝑥1 , 𝑥2 , 𝑥3 ≥ 0}, D is a basic solution by the definition in
the text, i.e. whether a solution is basic depends on the representation of 𝑃.
Linear Programming 2012
4
A
E
D
P
F
B
C
 Fig. 2.7: A, B, C, D, E, F are all basic solutions. C, D, E, F are basic
feasible solutions.
Linear Programming 2012
5
 Comparison of definitions in the notes and the text
Notes
Text
Extreme
point
Geometric definition
Geometric definition
Vertex
0-dimensional face
Existence of 𝑐 vector which makes 𝑥 ∗
as the unique optimal solution for the
LP
Basic
solution,
b.f.s.
Defined for standard
form. Set 𝑛 − 𝑚
variables at 0 and solve
the remaining system.
b.f.s. if nonnegative.
Defined for general polyhedron.
Satisfy equality constraints and 𝑛
linearly independent constraints are
active. ( 0-dimensional face if feasible)
Linear Programming 2012
6
 Thm 2.3: 𝑥 ∗ ∈ 𝑃, then 𝑥 ∗ vertex, extreme point, and b.f.s. are equivalent
statements.
Pf) We follow the definitions given in the text. We already showed in the
notes that extreme point and 0-dimensional face ( 𝐴𝐼 𝑥 ∗ = 𝑏𝐼 , 𝐴𝐼 : rank 𝑛,
b.f.s. in the text) are equivalent.
To show all are equivalent, take the following steps:
𝑥 ∗ vertex (1)  𝑥 ∗ extreme point (2)  𝑥 ∗ b.f.s. (3)  𝑥 ∗ vertex
(1) 𝑥 ∗ vertex  𝑥 ∗ extreme point
Suppose 𝑥 ∗ is vertex, i.e. ∃ 𝑐 ∈ 𝑅𝑛 such that 𝑥 ∗ is unique minimum of
min 𝑐 ′ 𝑥, 𝑥 ∈ 𝑃.
If 𝑦, 𝑧 ∈ 𝑃, 𝑦, 𝑧 ≠ 𝑥 ∗ , then 𝑐 ′ 𝑥 ∗ < 𝑐 ′ 𝑦 and 𝑐 ′ 𝑥 ∗ < 𝑐 ′ 𝑧.
Hence 𝑐 ′ 𝑥 ∗ < 𝑐 ′ 𝜆𝑦 + 1 − 𝜆 𝑧 , 0 ≤ 𝜆 ≤ 1 𝜆𝑦 + 1 − 𝜆 𝑧 ≠ 𝑥 ∗ .
Hence 𝑥 ∗ cannot be expressed as convex combination of two other points in
𝑃.
 𝑥 ∗ extreme point.
Linear Programming 2012
7
(continued)
(2) 𝑥 ∗ extreme point  𝑥 ∗ b.f.s.
Suppose 𝑥 ∗ is not a b.f.s.. Let 𝐼 = {𝑖: 𝑎𝑖′ 𝑥 ∗ = 𝑏𝑖 }.
Since 𝑥 ∗ is not a b.f.s., the number of linearly independent vectors 𝑎𝑖 in 𝐼 < 𝑛.
Hence ∃ nonzero 𝑑 ∈ 𝑅𝑛 such that 𝑎𝑖′ 𝑑 = 0, ∀ 𝑖 ∈ 𝐼.
Consider 𝑦 = 𝑥 ∗ + 𝜀𝑑, 𝑧 = 𝑥 ∗ − 𝜀𝑑. But, 𝑦, 𝑧 ∈ 𝑃 for sufficiently small
positive 𝜀, and 𝑥 ∗ = (𝑦 + 𝑧)/2, which implies 𝑥 ∗ is not an extreme point.
(3) 𝑥 ∗ b.f.s.  𝑥 ∗ vertex
Let 𝑥 ∗ be a b.f.s. and let 𝐼 = {𝑖: 𝑎𝑖′ 𝑥 ∗ = 𝑏𝑖 }.
Let 𝑐 = 𝑖∈𝐼 𝑎𝑖 . Then 𝑐 ′ 𝑥 ∗ = 𝑖∈𝐼 𝑎𝑖 ′𝑥 ∗ = 𝑖∈𝐼 𝑏𝑖 .
∀ 𝑥 ∈ 𝑃, we have 𝑐 ′ 𝑥 = 𝑖∈𝐼 𝑎𝑖 ′𝑥 ≥ 𝑖∈𝐼 𝑏𝑖 = 𝑐 ′ 𝑥 ∗ , hence 𝑥 ∗ optimal.
For uniqueness, equality holds  𝑎𝑖 ′𝑥 = 𝑏𝑖 , 𝑖 ∈ 𝐼.
Since 𝑥 ∗ is b.f.s., it is the unique solution of 𝑎𝑖 ′𝑥 = 𝑏𝑖 , 𝑖 ∈ 𝐼.
Hence 𝑥 ∗ is a vertex.

Linear Programming 2012
8
 Note: Whether 𝑥 ∗ is a basic solution depends on the representation of 𝑃.
However, 𝑥 ∗ is b.f.s. if and only if 𝑥 ∗ extreme point and 𝑥 ∗ being extreme point
is independent of the representation of 𝑃. Hence the property of being a b.f.s.
is also independent of the representation of 𝑃.
 Cor 2.1: For polyhedron 𝑃 ≠ ∅, there can be finite number of basic or basic
feasible solutions.
 Def: Two distinct basic solutions are said to be adjacent if we can find 𝑛 − 1
linearly independent constraints that are active at both of them. ( In Fig 2.7, D
and E are adjacent to B; A and C are adjacent to D.)
If two adjacent basic solutions are also feasible, then the line segment that joins
them is called an edge of the feasible set (one dimensional face).
Linear Programming 2012
9
2.3 Polyhedra in standard form
 Thm 2.4: 𝑃 = {𝑥: 𝐴𝑥 = 𝑏, 𝑥 ≥ 0}, 𝐴: 𝑚 × 𝑛, full row rank.
Then 𝑥 is a basic solution  𝑥 satisfies 𝐴𝑥 = 𝑏 and ∃ indices
𝐵 1 , … , 𝐵(𝑚) such that 𝐴𝐵(1) , … , 𝐴𝐵(𝑚) are linearly independent and 𝑥𝑖 =
0, 𝑖 ≠ 𝐵 1 , … , 𝐵(𝑚).
Pf) see text.
( To find a basic solution, choose 𝑚 linearly independent columns
𝐴𝐵(1) , … , 𝐴𝐵(𝑚) . Set 𝑥𝑖 = 0 for all 𝑖 ≠ 𝐵 1 , … , 𝐵(𝑚), then solve 𝐴𝑥 = 𝑏
for 𝑥𝐵(1) , … , 𝑥𝐵(𝑚) . )
 Def: basic variable, nonbasic variable, basis, basic columns, basis matrix 𝐵.
(see text)
( 𝐵𝑥𝐵 = 𝑏  𝑥𝐵 = 𝐵−1 𝑏 )
Linear Programming 2012
10
 Def: For standard form problems, we say that two bases are adjacent if they
share all but one basic column.
 Note: A basis uniquely determines a basic solution.
Hence if have two different basic solutions  have different basis.
But two different bases may correspond to the same basic solution. (e.g. when
𝑏=0)
Similarly, two adjacent basic solutions  two adjacent bases
Two adjacent bases with different basic solutions  two adjacent basic
solutions.
However, two adjacent bases only not necessarily imply two adjacent basic
solutions. The two solutions may be the same solution.
Linear Programming 2012
11
 Check that full row rank assumption on 𝐴 results in no loss of generality.
 Thm 2.5: 𝑃 = {𝑥: 𝐴𝑥 = 𝑏, 𝑥 ≥ 0} ≠ ∅, 𝐴: 𝑚 × 𝑛, rank is 𝑘 < 𝑚.
𝑄 = {𝑥: 𝐴𝐼 𝑥 = 𝑏𝐼 , 𝑥 ≥ 0}, 𝐼 = {𝑖1 , … , 𝑖𝑘 } with linearly independent rows.
Then 𝑃 = 𝑄.
Pf) Suppose first 𝑘 rows of 𝐴 are linearly independent.
𝑃 ⊆ 𝑄 is clear. Show 𝑄 ⊆ 𝑃.
Every row 𝑎𝑖 ′ of 𝐴 can be expressed as 𝑎𝑖 ′ =
Hence, for 𝑥 ∈ 𝑃, 𝑏𝑖 = 𝑎𝑖 ′𝑥 =
𝑘
𝑗=1 𝜆𝑖𝑗 𝑎𝑗 ′𝑥
=
𝑘
𝑗=1 𝜆𝑖𝑗 𝑎𝑗 ′.
𝑘
𝑗=1 𝜆𝑖𝑗 𝑏𝑗 ,
𝑖 = 1, … , 𝑚
i.e. 𝑏𝑖 is also linear combination of 𝑏𝑗 , 𝑗 ∈ 𝐼.
Suppose 𝑦 ∈ 𝑄, then ∀ 𝑖 = 1, … , 𝑚,
𝑎𝑖 ′𝑦 =
𝑘
𝑗=1 𝜆𝑖𝑗 𝑎𝑗 ′𝑦
=
𝑘
𝑗=1 𝜆𝑖𝑗 𝑏𝑗
Hence, 𝑦 ∈ 𝑃  𝑄 ⊆ 𝑃
Linear Programming 2012
= 𝑏𝑖

12
2.4 Degeneracy
 Def 2.10: A basic solution 𝑥 ∈ 𝑅𝑛 is said to be degenerate if more than 𝑛 of
the constraints are active at 𝑥.
 Def 2.11: 𝑃 = {𝑥 ∈ 𝑅𝑛 : 𝐴𝑥 = 𝑏, 𝑥 ≥ 0}, 𝐴: 𝑚 × 𝑛, full row rank.
Then 𝑥 is a degenerate basic solution if more than 𝑛 − 𝑚 of the components
of 𝑥 are 0 ( i.e. some basic variables have 0 value)
 For standard LP, if we have more than 𝑛 − 𝑚 variables at 0 for a basic
feasible solution 𝑥 ∗ , it means that more than 𝑛 − 𝑚 of the nonnegativity
constraints are active at 𝑥 ∗ in addition to the 𝑚 constraints in 𝐴𝑥 = 𝑏.
The solution can be identified by defining 𝑛 − 𝑚 nonbasic variables (value
= 0). Hence, depending on the choice of nonbasic variables, we have
different bases, but the solution is the same.
Linear Programming 2012
13
A
D
C
P
B
E
 Fig 2.9: A and C are degenerate basic feasible solutions. B and E are
nondegenerate. D is a degenerate basic solution.
Linear Programming 2012
14
𝑥3 = 0
𝑥4 = 0
A
B
𝑥5 = 0
𝑃
𝑥2 = 0
𝑥1 = 0
𝑥6 = 0
 Fig 2.11: (𝑛 − 𝑚)-dimensional illustration of degeneracy. Here, 𝑛 = 6,
𝑚 = 4. A is nondegenerate and basic variables are 𝑥1 , 𝑥2 , 𝑥3 , 𝑥6 . B is
degenerate. We can choose 𝑥1 , 𝑥6 as the nonbasic variables. Other
possibilities are to choose 𝑥1 , 𝑥5 , or to choose 𝑥5 , 𝑥6 .
Linear Programming 2012
15
 Degeneracy is not purely geometric property, it may depend on
representation of the polyhedrom
ex) 𝑃 = {𝑥: 𝐴𝑥 = 𝑏, 𝑥 ≥ 0}, 𝐴: 𝑚 × 𝑛
𝑃′ = {𝑥: 𝐴𝑥 ≥ 𝑏, −𝐴𝑥 ≥ −𝑏, 𝑥 ≥ 0}
We know that 𝑃 = 𝑃′, but representation is different.
Suppose 𝑥 ∗ is a nondegenerate basic feasible solution of 𝑃.
Then exactly 𝑛 − 𝑚 of the variables 𝑥𝑖∗ are equal to 0.
For 𝑃′, at the basic feasible solution 𝑥 ∗ , we have 𝑛 − 𝑚 variables set to 0
and additional 2𝑚 constraints are satisfied with equality. Hence, we have
𝑛 + 𝑚 active constraints and 𝑥 ∗ is degenerate.
Linear Programming 2012
16
2.5 Existence of extreme points
 Def 2.12: Polyhedron 𝑃 ⊆ 𝑅𝑛 contains a line if ∃ a vector 𝑥 ∈ 𝑃 and a
nonzero 𝑑 ∈ 𝑅𝑛 such that 𝑥 + 𝜆𝑑 ∈ 𝑃 for all 𝜆 ∈ 𝑅.
Note that if 𝑑 is a line in 𝑃, then 𝐴(𝑥 + 𝜆𝑑) ≤ 𝑏 for all 𝜆 ∈ 𝑅
 𝐴𝑑 = 0
Hence 𝑑 is a vector in the lineality space 𝑆. (in 𝑃 = 𝑆 + 𝐾 + 𝑄)
 Thm 2.6: 𝑃 = {𝑥 ∈ 𝑅𝑛 : 𝑎𝑖 ′𝑥 ≥ 𝑏𝑖 , 𝑖 = 1, … , 𝑚} ≠ ∅, then the following are
equivalent.
(a) 𝑃 has at least one extreme point.
(b) 𝑃 does not contain a line.
(c) ∃ 𝑛 vectors out of 𝑎1 , … , 𝑎𝑚 , which are linearly independent.
Pf) see proof in the text.
Linear Programming 2012
17
 Note that the conditions given in Thm 2.6 means that the lineality space 𝑆 =
{0}.
 Cor 2.2: Every nonempty bounded polyhedron (polytope) and every
nonempty polyhedron in standard form has at least one basic feasible
solution (extreme point).
Linear Programming 2012
18
2.6 Optimality of extreme points
 Thm 2.7: Consider the LP of minimizing 𝑐 ′ 𝑥 over a polyhedron 𝑃. Suppose
𝑃 has at least one extreme point and there exists an optimal solution.
Then there exists an optimal solution which is an extreme point of 𝑃.
Pf) see text.
 Thm 2.8: Consider the LP of minimizing 𝑐 ′ 𝑥 over a polyhedron 𝑃. Suppose
𝑃 has at least one extreme point.
Then, either the optimal cost is −∞, or there exists an extreme point which is
optimal.
Linear Programming 2012
19
(continued)
Idea of proof in the text)
Consider any 𝑥 ∈ 𝑃. Let 𝐼 = {𝑖: 𝑎𝑖 ′𝑥 = 𝑏𝑖 }
Then we move to 𝑦 = 𝑥 + 𝜆𝑑, where 𝑎𝑖 ′𝑑 = 0, 𝑖 ∈ 𝐼 and 𝑐 ′ 𝑑 ≤ 0.
Then either the optimal cost is −∞ ( if the half line 𝑑 is in 𝑃 and 𝑐 ′ 𝑑 < 0 )
or we meet a new inequality which becomes active ( cost does not increase).
By repeating the process, we eventually arrive at an extreme point which
has value not inferior to 𝑥.
Therefore, for any 𝑥 in 𝑃, there exists an extreme point 𝑦 such that 𝑐 ′ 𝑦 ≤
𝑐 ′ 𝑥. Then we choose the extreme point which gives the smallest objective
value with respect to 𝑐.
Linear Programming 2012
20
 ( alternative proof of Thm 2.8)
𝑃 = 𝑆 + 𝐾 + 𝑄. Pointedness of 𝑃 implies 𝑆 = {0}.
Hence 𝑥 ∈ 𝑃  𝑥 = 𝑖 𝑞𝑖 𝑑 𝑖 + 𝑗 𝑟𝑗 𝑢 𝑗 , where 𝑑 𝑖 ′𝑠 are extreme rays of 𝐾
and 𝑢 𝑗 ′𝑠 are extreme points of 𝑃 and 𝑞𝑖 ≥ 0, ∀ 𝑖, 𝑟𝑗 ≥ 0, ∀ 𝑗, 𝑗 𝑟𝑗 = 1.
Suppose ∃ 𝑖 such that 𝑐 ′ 𝑑 𝑖 < 0, then LP is unbounded.
( For 𝑥 ∈ 𝑃, 𝑥 + 𝜆𝑑 𝑖 ∈ 𝑃 for 𝜆 ≥ 0. Then 𝑐′(𝑥 + 𝜆𝑑 𝑖 )  −∞ as 𝜆 → ∞)
Otherwise, 𝑐′𝑑 𝑖 ≥ 0 for all 𝑖, take 𝑢∗ such that 𝑐 ′ 𝑢∗ = 𝑚𝑖𝑛𝑗 𝑐′𝑢 𝑗 .
Then ∀ 𝑥 ∈ 𝑃,
𝑐′𝑥 =
𝑖
𝑖 𝑞𝑖 (𝑐′𝑑 ) +
𝑗
𝑗 𝑟𝑗 (𝑐′𝑢 ) ≥
𝑗
′𝑢
𝑗 𝑟𝑗 (𝑐′𝑢 ) ≥ 𝑐
∗
𝑗 𝑟𝑗
= 𝑐′𝑢∗ .
Hence LP is either unbounded or ∃ an extreme point of 𝑃 which is an optimal
solution.
Proof here shows that the existence of an extreme ray 𝑑 𝑖 of the pointed
recession cone 𝐴𝑥 ≥ 0 ( if have min problem and polyhedron is 𝐴𝑥 ≥ 𝑏)
such that 𝑐 ′ 𝑑 𝑖 < 0 is the necessary and sufficient condition for unboundedness
of the LP.
( If 𝑃 has at least one extreme point, then LP is unbounded
 ∃ an extreme ray 𝑑 𝑖 in recession cone 𝐾 such that 𝑐 ′ 𝑑 𝑖 < 0)
Linear Programming 2012

21