Transcript Greedy Algorithms

Greedy Algorithms
Greed is good.
(Some of the time)
Outline

Elements of greedy algorithm



Greedy choice property
Optimal substructures
Minimum spanning tree


Kruskal’s algorithm
Prim’s algorithm
Huffman code
 Activity selection

Jaruloj Chongstitvatana
Chapter 3: Greedy Algorithms
2
Introduction


Concepts

Choosing the best possible choice at each step.

This decision leads to the best over all solution.
Greedy algorithms do not always yield
optimal solutions.
Jaruloj Chongstitvatana
Chapter 3: Greedy Algorithms
3
Elements of
Greedy Algorithms
Greedy-choice property

A globally optimal solution is derived from a
locally optimal (greedy) choice.

When choices are considered, the choice that
looks best in the current problem is chosen,
without considering results from
subproblems.
Jaruloj Chongstitvatana
Chapter 3: Greedy Algorithms
5
Optimal substructures

A problem has optimal substructure if an
optimal solution to the problem is composed
of optimal solutions to subproblems.

This property is important for both greedy
algorithms and dynamic programming.
Jaruloj Chongstitvatana
Chapter 3: Greedy Algorithms
6
Greedy Algorithm v.s. Dynamic Programming
Greedy algorithm
 The best choice is made at
each step and after that the
subproblem is solved.
 The choice made by a
greedy algorithm may
depend on choices so far, but
it cannot depend on any
future choices or on the
solutions to subproblems.
 A greedy strategy usually
progresses in a top-down
fashion, making one greedy
choice after another,
reducing each given problem
instance to a smaller one.
Jaruloj Chongstitvatana
Dynamic programming
 A choice is made at each
step.

The choice made at each
step usually depends on the
solutions to subproblems.

Dynamic-programming
problems are often solved in
a bottom-up manner.
Chapter 3: Greedy Algorithms
7
Steps in Design Greedy Algorithms

Determine the optimal substructure of the problem.

Develop a recursive solution.

Prove that at any stage of the recursion, one of the
optimal choices is the greedy choice. Thus, it is
always safe to make the greedy choice.

Show that all but one of the subproblems induced by
having made the greedy choice are empty.

Develop a recursive algorithm that implements the
greedy strategy.

Convert the recursive algorithm to an iterative
algorithm.
Jaruloj Chongstitvatana
Chapter 3: Greedy Algorithms
8
Shortcuts Design



Form the optimization problem so that, after a
choice is made and there is only one subproblem left
to be solved.
Prove that there is always an optimal solution to the
original problem that makes the greedy choice, so
that the greedy choice is always safe.
Demonstrate that, having made the greedy choice,
what remains is a subproblem with the property that
if we combine an optimal solution to the subproblem
with the greedy choice we have made, we arrive at
an optimal solution to the original problem.
Jaruloj Chongstitvatana
Chapter 3: Greedy Algorithms
9
Minimum Spanning Tree
Definitions

Let G = (V, E) be an undirected graph.
T is a minimum spanning tree of G if T ⊆ E is an
acyclic subset that connects all of the vertices and whose
total weight w(T ) =  w(u, v) is minimized.
(u,v)∈T

Let A be a subset of some minimum spanning tree.
An edge (u, v) is a safe edge for A if A  {(u, v)} is also
a subset of a minimum spanning tree.
Jaruloj Chongstitvatana
Chapter 3: Greedy Algorithms
11
Definitions

Let G = (V, E) be an undirected graph.
A cut (S, V − S) of G is a partition of V.
An edge (u, v)  E crosses the cut (S, V − S) if one of its
endpoints is in S and the other is in V − S.
A cut respects a set A of edges if no edge in A crosses the
cut.
An edge is a light edge crossing a cut if its weight is the
minimum of any edge crossing the cut.
Jaruloj Chongstitvatana
Chapter 3: Greedy Algorithms
12
Theorem
S
If (u, v) is a light edge crossing (S,
V − S), then edge (u, v) is safe for
A.
Jaruloj Chongstitvatana
A
u
x
light edge
Let G = (V, E) be a connected,
undirected graph with a real-valued
weight function w defined on E,
A be a subset of E that is included in
some minimum spanning tree for G,
(S, V − S) be any cut of G that
respects A.
Chapter 3: Greedy Algorithms
v
y
V-S
13
Proof
Let T be a minimum spanning tree
that includes A.
If (u,v) is not in T, then T contains an
edge (x,y) crossing S and V-S.
Since both (u,v) and (x,y) cross S and
V-S, there is a cycle for edges in T
 (u,v).
x
T
v
Another spanning tree T’ can be
created from T.
Jaruloj Chongstitvatana
A
u
light edge
If (u,v) is in T, then (u,v) is safe for A.
S
Chapter 3: Greedy Algorithms
y
V-S
14
Proof
S
S
A
u
u
v
y
x
light edge
light edge
x
T
A
T’
v
V-S
y
V-S
Remove
Add
edge
(x,y)
-> create
->
cutaT.
the
cycle
cycle
T’edge
is(u,v)
created
from
Jaruloj Chongstitvatana
Chapter 3: Greedy Algorithms
15
Proof
Thus, T’ is a spanning tree that includes A.
Next, we need to show that T’ is a minimum spanning tree.
From the construction of T’ , T’ = T-{(x,y)}  {(u,v)}.
Thus, w(T’ ) = w(T) - w(x,y) + w(u,v).
Since (u,v) is a light edge cross S and V-S, w(u,v)  w(x,y).
Thus, w(T’ )  w(T).
Since T is a minimum spanning tree, w(T’ ) = w(T).
Then, T’ is also a minimum spanning tree .
As a result, (u, v) is safe for A.
Jaruloj Chongstitvatana
Chapter 3: Greedy Algorithms
16
Corollary
Let G = (V, E) be a connected, undirected graph with a realvalued weight function w defined on E,
A be a subset of E that is included in some minimum
spanning tree for G, and
C = (VC, EC) be a connected component (tree) in the
forest GA = (V, A).
If (u, v) is a light edge connecting C to another tree in GA,
then (u, v) is safe for A.
Proof
(VC, V−VC) respects A, and (u, v) is a light edge for this cut.
Therefore, (u, v) is safe for A.
Jaruloj Chongstitvatana
Chapter 3: Greedy Algorithms
17
Kruskal’s Algorithm

Concept



Build a forest of minimum spanning trees.
Repeatedly connect the trees to create a subset of a
minimum spanning tree, until all nodes are covered.
In connecting two trees, choose the edge of the least
weight.

Let C1 and C2 denote the two trees that are
connected by (u, v). Since (u, v) must be a light
edge connecting C1 to some other tree, we need to
prove that (u, v) is a safe edge for C1.

Kruskal’s algorithm is a greedy algorithm, because
at each step it adds to the forest an edge of least
possible weight.
Jaruloj Chongstitvatana
Chapter 3: Greedy Algorithms
18
Example of Kruskal’s Algorithm
12
2
8
5
9
3
7
8
1
Jaruloj Chongstitvatana
1
9
11
5
3
10
Chapter 3: Greedy Algorithms
19
Kruskal’s Algorithm
MST-KRUSKAL(G,w)
A=
for each vertex v in V [G]
do
MAKE-SET(v)
sort the edges of E in nondecreasing order by weight w
for each edge (u, v) in E, taken in nondecreasing order
by weight
do if FIND-SET(u)  FIND-SET(v)
then A = A  {(u, v)}
UNION(u, v)
return A
Jaruloj Chongstitvatana
Chapter 3: Greedy Algorithms
20
Kruskal’s Algorithm: Complexity
MST-KRUSKAL(G,w)
A=
for each vertex v in V [G] O(v)
do
MAKE-SET(v)
O(e lg e)
sort the edges of E in nondecreasing order by weight w
for each edge (u, v) in E, taken in nondecreasing order
by weight
do if FIND-SET(u)  FIND-SET(v)
then A = A  {(u, v)}
UNION(u, v)
O((v+e) lg v)
return A
2
e  v ; O(e lg v)
Jaruloj Chongstitvatana
Chapter 3: Greedy Algorithms
21
Prim’s Algorithm

Prim’s algorithm has the property that the edges in the
set A always form a single tree.
The tree starts from an arbitrary root vertex r.
 Grow the tree until it spans all the vertices in V.


At each step, a light edge is added to the tree A that connects A
to an isolated vertex of GA = (V, A).
Jaruloj Chongstitvatana
Chapter 3: Greedy Algorithms
22
Example of Prim’s Algorithm
12
2
8
5
9
3
7
8
1
Jaruloj Chongstitvatana
1
9
11
5
3
10
Chapter 3: Greedy Algorithms
23
Prim’s Algorithm
PRIM(G,w, r)
for each u in V[G]
do key[u] = ∞
π[u] = NIL
key[r] = 0
Q = V[G]
while Q  
do
u = EXTRACT-MIN(Q)
for each v in Adj[u]
do if v in Q and w(u, v) < key[v]
then π[v] = u
key[v] = w(u, v)
Jaruloj Chongstitvatana
Chapter 3: Greedy Algorithms
24
Execution of Prim’s Algorithm
12
0
2
Jaruloj Chongstitvatana
1
9
11
5
3
1
7
8
3
1
3
7
8
5
1
9
2
5
8
12
9
3
10
Chapter 3: Greedy Algorithms
10
5
25
Prim’s Algorithm:Complexity
PRIM(G,w, r)
for each u in V[G] Also build min-heap
do key[u] = ∞
O(v)
π[u] = NIL
key[r] = 0
O(e
Q = V[G]
while Q  
do
u = EXTRACT-MIN(Q) O(lg v) O(v
for each v in Adj[u]
do if v in Q and w(u, v) < key[v]
then π[v] = u
key[v] = w(u, v) O(lg v)
Jaruloj Chongstitvatana
Chapter 3: Greedy Algorithms
lg v)
lg v)
26
Huffman Code
Problem

Find an optimal prefix code representing a
set of characters in a file, where each
character has a frequency of occurrences.

Prefix code




Codes in which no codeword is also a prefix of
some other codeword.
{0, 110, 101, 111, 1000, 1001} is a prefix code.
{0, 110, 101, 111, 1010, 0111} is not prefix code
Optimality

The code yields a file with the minimum number
of bits.
Jaruloj Chongstitvatana
Chapter 3: Greedy Algorithms
28
Creating Huffman Code: Example
100
1
64
0
1
0
30
34
1
0
1
15
0
0
ก: 36
0
Jaruloj Chongstitvatana
1
ข: 17
ค: 17
ง: 15
จ: 10 ฉ: 5
100
101
110
1110
Chapter 3: Greedy Algorithms
1111
29
Optimal Code

An optimal code for a file is always
represented by a full binary tree, in which
every nonleaf node has two children.

If C is the alphabet from which the
characters are drawn and all character
frequencies are positive, then the tree for an
optimal prefix code has exactly |C| leaves,
one for each letter of the alphabet, and
exactly |C|−1 internal nodes.
Jaruloj Chongstitvatana
Chapter 3: Greedy Algorithms
30
Full Binary Trees for Prefix Code
Tree for 1 letter
Tree for 2 letters
1
A
1
Tree for 4 letters
2
B
Tree for 3 letters
C
B
3
3
4
A
1
2
A
1
A
1
C
B
4
3
B
2
Jaruloj Chongstitvatana
2
A
1
2
3
Chapter 3: Greedy Algorithms
31
Full Binary Trees for Prefix Code
Tree for 5 letters
Tree for 4 letters
B
B
C
3
4
C
A
1
2
Jaruloj Chongstitvatana
5
A
1
2
D
B
3
1
4
C
4
3
D
5
A
2
B
C
4
A
3
Chapter 3: Greedy Algorithms
1
2
32
Creating Huffman Code: Example
กขคงจฉ: 100
1
ขคงจฉ: 64
0
0
1
งจฉ: 30
ขค: 34
1
0
1
จฉ: 15
0
0
ก: 36
0
Jaruloj Chongstitvatana
1
ข: 17
ค: 17
ง: 15
จ: 10 ฉ: 5
100
101
110
1110
Chapter 3: Greedy Algorithms
1111
33
Theorem
A full binary tree for an optimal prefix code for C letters
has exactly C leaves, and exactly C−1 internal nodes.
Proof by induction.
Basis:
C=1. If there is one letter, the binary tree requires only 1
leaf node, and 0 internal node.
Induction hypotheses:
For C< n, the full binary tree for an optimal prefix code
for C letters has exactly C leaves, and exactly C−1
internal nodes.

Jaruloj Chongstitvatana
Chapter 3: Greedy Algorithms
34
Theorem

Induction Step:
Let T be a full binary tree for an optimal prefix code for
C+1 letters.
To create a full binary tree or optimal prefix code, we can
take a full binary tree for an optimal prefix code for C
letters, and add another leaf node L by either


adding a new root node R and put L and the old root of T as its
children, or
replacing a leaf node N of T by an internal node and put L and
N as its children.
In either case, the number of leaf nodes of T is C+1 and
the number of internal nodes is C.
Jaruloj Chongstitvatana
Chapter 3: Greedy Algorithms
35
Creating Huffman Code: Algorithms
HUFFMAN(C) Use min-heap for Q
n = |C|
O(n) to build min-heap
Q=C
for i = 1 to n − 1
do allocate a new node z
z.left = x = EXTRACT-MIN(Q)
z.right = y = EXTRACT-MIN(Q)
z.f =x.f + y.f
INSERT(Q, z)
►Return the root of the tree.
return EXTRACT-MIN(Q)
Jaruloj Chongstitvatana
O(n lg n)
O(lg n)
Chapter 3: Greedy Algorithms
36
Greedy-choice property of Huffman Code
Let C be an alphabet in which each character c  C has
frequency f [c], and
x and y be two characters in C having the lowest
frequencies.
 Then, there exists an optimal prefix code for C in which
the codewords for x and y have the same length and
differ only in the last bit.
Jaruloj Chongstitvatana
Chapter 3: Greedy Algorithms
37
Proof

The idea of the proof is to :



take the tree T representing an arbitrary optimal prefix code
modify it to make a tree representing another optimal prefix
code such that the characters x and y appear as sibling leaves of
maximum depth in the new tree.
If we can do this, then their codewords will have the
same length and differ only in the last bit.
Jaruloj Chongstitvatana
Chapter 3: Greedy Algorithms
38
Proof
Let a and b be two characters that are sibling leaves of
maximum depth in T .
Assume that f [a] ≤ f [b] and f [x] ≤ f [y].
Since f [x] and f [y] are the two lowest leaf frequencies, in
order, and f [a] and f [b] are two arbitrary frequencies, in
order, we have f [x] ≤ f [a] and f [y] ≤ f [b].
Jaruloj Chongstitvatana
Chapter 3: Greedy Algorithms
39
Proof
Exchange the positions of a and x in T to produce T’ .
B(T) − B(T’ ) =  f(c) dT (c) −  f(c)dT’ (c)
cC
cC
= f [x] dT (x) + f [a] dT (a) − f [x] dT’ (x) − f [a] dT’ (a)
= f [x] dT (x) + f [a] dT (a) − f [x] dT (a) − f [a] dT (x)
= ( f [a] − f [x])( dT (a) − dT (x)) ≥ 0
T’
T
x
y
y
a
Jaruloj Chongstitvatana
a
b
x
Chapter 3: Greedy Algorithms
b
40
Proof
Then, exchange the positions of b
and y in T’ to produce T’’.
T’
Similarly, it does not increase the
cost, and so B(T’ ) − B(T’’ )  0.
y
Therefore, B(T’’ ) ≤ B(T).
Since T is optimal, B(T) ≤ B(T’’ ).
Then, B(T’’ ) = B(T).
T’’
Thus, T’’ is an optimal tree in which
x and y appear as sibling leaves of
b
maximum depth.
a
x
a
x
Jaruloj Chongstitvatana
Chapter 3: Greedy Algorithms
b
y
41
Optimal-substructure Property
Let C be a given alphabet with frequency f [c] defined for
each character c  C,
x and y be two characters in C with minimum frequency,
C’ be the alphabet C with characters x, y removed and
(new) character z added, so that C’ = C − {x, y}  {z};
define f for C’ as for C, except that f [z] = f [x] + f [y], and
T be any tree representing an optimal prefix code for the
alphabet C ’.
 Then the tree T , obtained from T ’ by replacing the leaf
node for z with an internal node having x and y as
children, represents an optimal prefix code for the
alphabet C.
Jaruloj Chongstitvatana
Chapter 3: Greedy Algorithms
42
Proof
Show that the cost B(T) can be expressed in terms of the
cost B(T’) by considering the component costs.
For each c  C − {x, y}, we have


dT (c) = dT ’ (c).
f [c] dT (c) = f [c] dT ’ (c).
Since dT (x) = dT (y) = dT ’(z) + 1, we have
f [x]dT (x) + f [y]dT (y) = ( f [x] + f [y])( dT’ (z) + 1)
= f [z] dT ’ (z) + ( f [x] + f [y])
Thus, B(T) = B(T ’) + f [x] + f [y].
That is, B(T ’) = B(T) − f [x] − f [y] .
Jaruloj Chongstitvatana
Chapter 3: Greedy Algorithms
43
Proof
We now prove by contradiction.
Suppose T does not represent an optimal prefix code for C.
Then there exists a tree T ’’ such that B(T ’’) < B(T).
Without loss of generality, T ’’ has x and y as siblings.
Jaruloj Chongstitvatana
Chapter 3: Greedy Algorithms
44
Proof
Let T ’’’ be the tree T ’’ with the common parent of x and y
replaced by a leaf z with frequency f [z] = f [x] + f [y].
Then, B(T ’’’) = B(T ’’) − f [x] − f [y]
< B(T) − f [x] − f [y]
=
B(T ’).
We reach a contradiction to the assumption that T ’
represents an optimal prefix code for C’.
Thus, T must represent an optimal prefix code for the
alphabet C.
Jaruloj Chongstitvatana
Chapter 3: Greedy Algorithms
45
Interval Scheduling
Problem definition
Let S be {a1, a2, . . . , an} of n proposed activities that
wish to use the same resource.
 Only one activity can use the resource at a time.
 Each activity ai has a start time si and a finish time fi,
where 0 ≤ si < fi < ∞.
 If selected, activity ai takes place during the half-open
time interval [si , fi ).
 Activities ai and aj are compatible if the intervals [si , fi )
and [s j , f j ) do not overlap .
 The activity-selection problem is to select a largest
subset of mutually compatible activities.

Jaruloj Chongstitvatana
Chapter 3: Greedy Algorithms
47
Example
S
A1
A2
A3
Jaruloj Chongstitvatana
Chapter 3: Greedy Algorithms
48
Subproblems

Si j = {ak  S : fi ≤ sk < fk ≤ s j }

the subset of activities in S that can start after activity ai
finishes and finish before activity aj starts.

Add activities a0 and an+1 such that f0 = 0 and sn+1 =∞.

Then S = S0,n+1, and 0 ≤ i, j ≤ n + 1.
Jaruloj Chongstitvatana
Chapter 3: Greedy Algorithms
49
Optimal Solution
Let Ai j be an optimal solution to Si j .
 Let c[i, j ] be the number of activities in a maximum-size
subset of mutually compatible activities in Si j.


c[i, j ] = 0 for i ≥ j (i.e. Si j = )
ak

c[i, j ] = c[i, k] + c[k, j ] + 1
if ak be an activity in Ai j.
Then,
 c[i, j ] =  0
max {c[i, k] + c[k, j ] + 1}
i<k<j

if Si j = 
if Si j  
akSi j
Jaruloj Chongstitvatana
Chapter 3: Greedy Algorithms
50
Theorem
Consider any nonempty subproblem Si j , and let am be
the activity in Si j with the earliest finish time:
fm = min { fk : ak ∈ Si j } .
Then,
 Activity am is used in some maximum-size subset of
mutually compatible activities of Si j .
 The subproblem Sim is empty, so that choosing am leaves
the subproblem Smj as the only one that may be
nonempty.

Jaruloj Chongstitvatana
Chapter 3: Greedy Algorithms
51
Proof: First part
Let Ai j be a maximum-size subset of mutually
compatible activities of Si j , and Ai j is sorted in
monotonically increasing order of finish time.
 Let ak be the first activity in Ai j.
 If ak = am, am is used in some maximum-size subset of
mutually compatible activities of Si j.
 If ak  am, we construct the subset A’i j = Ai j − {ak} 
{am}.

S
Aij ak
A’ij
Jaruloj Chongstitvatana
am
Chapter 3: Greedy Algorithms
52
Proof: First part
The activities in A’i j are disjoint, since the activities in Aij
are, ak is the first activity in Ai j to finish, and fm ≤ fk .
 Noting that A’i j has the same number of activities as Ai j ,
we see that A’i j is a maximum-size subset of mutually
compatible activities of Si j that includes am.

S
Aij ak
A’ij
Jaruloj Chongstitvatana
am
Chapter 3: Greedy Algorithms
53
Proof: Second part
Let Sim be nonempty.
 Then, there is an activity ak such that fi ≤ sk <fk ≤ sm< fm.
 Then, ak is also in Si j and it has an earlier finish time than
am, which contradicts our choice of am.
 We conclude that Sim is empty.
ai
am
S
Jaruloj Chongstitvatana
ak
aj
Chapter 3: Greedy Algorithms
54
Greedy Solution
Ai j = {ak}  Akj
where ak is the activity which finishes earliest
among activities in Si j.
Jaruloj Chongstitvatana
Chapter 3: Greedy Algorithms
55
Recursive Algorithms
ACTIVITY-SELECTOR(s, f, i, n)
m=i+1
while m ≤ n and s[m] < f [i]
► Find the first activity in Si,n+1.
do
m= m+1
if m ≤ n
then return {am} 
ACTIVITY-SELECTOR(s, f,m, n)
else return 
Jaruloj Chongstitvatana
Chapter 3: Greedy Algorithms
56
Iterative Algorithm
GREEDY-ACTIVITY-SELECTOR(s, f )
n = length[s]
A = {a1}
i=1
for m = 2 to n
do if s[m ] ≥ f[i]
then A = A  {am}
i=m
return A
Jaruloj Chongstitvatana
Chapter 3: Greedy Algorithms
57