Transcript Document

Chapter 5
Fundamental Algorithm
Design Techniques
Overview



The Greedy Method (5.1)
Divide and Conquer (5.2)
Dynamic Programming (5.3)
Greedy Outline




The Greedy Method Technique (§5.1)
Fractional Knapsack Problem (§5.1.1)
Task Scheduling (§5.1.2)
Minimum Spanning Trees (§7.3) [future lecture]
A New Vending Machine?


A new Coke machine in the
lounge behind the ulab?
Being efficient computer
scientists, we want to return
the smallest possible number
of coins in change
The Greedy Method
The Greedy Method

The greedy method is a general algorithm design
paradigm, built on the following elements:



configurations: different choices, collections, or values
to find
objective function: a score assigned to configurations,
which we want to either maximize or minimize
It works best when applied to problems with the
greedy-choice property:

a globally-optimal solution can always be found by a series
of local improvements from a starting configuration.
Making Change





Problem: A dollar amount to reach and a collection of coin
amounts to use to get there.
Configuration: A dollar amount yet to return to a customer
plus the coins already returned
Objective function: Minimize number of coins returned.
Greedy solution: Always return the largest coin you can
Example 1: Coins are valued $.32, $.08, $.01


Has the greedy-choice property, since no amount over $.32
can be made with a minimum number of coins by omitting a
$.32 coin (similarly for amounts over $.08, but under $.32).
Example 2: Coins are valued $.30, $.20, $.05, $.01

Does not have greedy-choice property, since $.40 is best made
with two $.20’s, but the greedy solution will pick three coins
(which ones?)
How to rob a bank…
The Fractional Knapsack
Problem

Given: A set S of n items, with each item i having




bi - a positive benefit
wi - a positive weight
Goal: Choose items with maximum total benefit but with weight
at most W.
If we are allowed to take fractional amounts, then this is the
fractional knapsack problem.

In this case, we let xi denote the amount we take of item i

Objective: maximize
b (x / w )
iS

Constraint:
x
iS
i
W
i
i
i
Example

Given: A set S of n items, with each item i having



bi - a positive benefit
wi - a positive weight
Goal: Choose items with maximum total benefit but with weight at
most W.
“knapsack”
Items:
Weight:
Benefit:
Value:
($ per ml)
1
2
3
4
5
4 ml
8 ml
2 ml
6 ml
1 ml
$12
$32
$40
$30
$50
3
4
20
5
50
Solution:
•
•
•
•
10 ml
1
2
6
1
ml
ml
ml
ml
of
of
of
of
5
3
4
2
The Fractional Knapsack
Algorithm

Greedy choice: Keep taking item
with highest value (benefit to
weight ratio)



Since  bi ( xi / wi )   (bi / wi ) xi
iS
iS
Run time: O(n log n). Why?
Correctness: Suppose there is a
better solution




there is an item i with higher
value than a chosen item j, but
xi<wi, xj>0 and vi<vj
If we substitute some i with j, we
get a better solution
How much of i: min{wi-xi, xj}
Thus, there is no better solution
than the greedy one
Algorithm fractionalKnapsack(S, W)
Input: set S of items w/ benefit bi
and weight wi; max. weight W
Output: amount xi of each item i
to maximize benefit w/ weight
at most W
for each item i in S
xi  0
vi  bi / wi
{value}
w0
{total weight}
while w < W
remove item i w/ highest vi
xi  min{wi , W - w}
w  w + min{wi , W - w}
Task Scheduling

Given: a set T of n tasks, each having:



A start time, si
A finish time, fi (where si < fi)
Goal: Perform all the tasks using a minimum number of
“machines.”
Machine 3
Machine 2
Machine 1
1
2
3
4
5
6
7
8
9
Task Scheduling
Algorithm


Greedy choice: consider tasks by
Algorithm taskSchedule(T)
their start time and use as few
Input: set T of tasks w/ start time si
machines as possible with this
and finish time fi
order.
Output: non-conflicting schedule

Run time: O(n log n). Why?
with minimum number of machines
Correctness: Suppose there is a
m0
{no. of machines}
better schedule.
while T is not empty

We can use k-1 machines
remove task i w/ smallest si

The algorithm uses k
if there’s a machine j for i then

Let i be first task scheduled on
schedule i on machine j
machine k
else

Machine i must conflict with k-1
mm+1
other tasks
schedule i on machine m

But that means there is no nonconflicting schedule using k-1
machines
Example

Given: a set T of n tasks, each having:




A start time, si
A finish time, fi (where si < fi)
[1,4], [1,3], [2,5], [3,7], [4,7], [6,9], [7,8] (ordered by start)
Goal: Perform all tasks on min. number of machines
Machine 3
Machine 2
Machine 1
1
2
3
4
5
6
7
8
9
Divide-and-Conquer
7 29 4  2 4 7 9
72  2 7
77
22
94  4 9
99
44
Outline and Reading



Divide-and-conquer paradigm (§5.2)
Review Merge-sort (§4.1.1)
Recurrence Equations (§5.2.1)





Iterative substitution
Recursion trees
Guess-and-test
The master method
Integer Multiplication (§5.2.2)
Divide-and-Conquer

Divide-and conquer is a general
algorithm design paradigm:





Divide: divide the input data S in
two or more disjoint subsets S1, S2,
…
Recur: solve the subproblems
recursively
Conquer: combine the solutions for
S1, S2, …, into a solution for S
The base case for the recursion
are subproblems of constant size
Analysis can be done using
recurrence equations
Binary Search



I have a number X between c1 and c2…
Divide and conquer algorithm?
Analysis:
b

T (n)  
T (n / 2)  c


if n  2
if n  2
Called a Recurrence Equation
How to solve?
Solving the Binary Search Recurrence
T(n) = T(n/2) + c
T(n/2) = T(n/4) + c
T(n/4) = T(n/8) + c …
----------T(n) = T(n/2) + c
= [T(n/4) + c] + c
= T(n/22) + 2 c
= T(n/23) + 3 c
[telescope]
= T(n/2k) + k c
done when k=lg n; then T(n/2k)=T(1) is base case
= b + c lg n
Merge Sort Recurrence
Equation Analysis

Recurrence Equation?
b
if n  2

T (n)  
2T (n / 2)  bn if n  2


We can analyze by finding a closed form
solution.
Method: expand and telescope
Solving the Merge Sort recurrence
T(n) = 2T(n/2) + b n
T(n/2) = 2T(n/4) + b n/2
T(n/4) = 2T(n/8) + b n/4 …
----------T(n) = 2T(n/2) + b n
= 2[2T(n/4) + b n/2] + b n
= 22T(n/22) + 2 b n
= 23T(n/23) + 3 b n
[telescope]
= 2kT(n/2k) + k b n
done when k=lg n; then T(n/2k)=T(1) is base case
= b n + b n lg n
Min & Max



Given a set of n items, find the min and max. How
many comparisons needed?
Simple alg: n-1 comparisons to find min, n-1 for max
Recursive algorithm:
void minmax(int i, int j, int& min0, int& max0) {
if (j<=i+1) [handle base case; return]
m=(i+j+1)/2;
minmax(i,m-1,min1,max1);
minmax(m,j,min2,max2);
min0 = min(min1,min2);
max0 = max(max1,max2);
}
Recurrence: T(n)=2T(n/2) + 2; T(2)=1
Solution?
1.5 n - 2
Solving the Min&Max Recurrence
T(n) = 2T(n/2) + 2; T(2)=1
T(n/2) = 2T(n/4) + 2
T(n/4) = 2T(n/8) + 2 …
----------T(n)
= 2T(n/2) + 2
= 2[2T(n/4) + 2] + 2
= 22T(n/22) + 22 + 2
= 23T(n/23) + 23 + 22 + 2
[telescope]
k
i
= 2kT(n/2k) +  2
i 1
= 2kT(n/2k) + 2k+1 - 2
done when k=(lg n) - 1; then T(n/2k)=T(2) is base case
= n/2 + n – 2 = 1.5n - 2
Hanoi runtime?
void hanoi(n, from, to, spare {
if (n > 0) {
hanoi(n-1,from,spare,to);
cout << from << “ – “ << to << endl;
hanoi(n-1,spare,to,from);
}
}
T(n) = 2T(n-1) + c
T(0) = b
Look it up: T(n) = O(2n)
Master Method
Another way to solve recurrences: “Look it up”
 The Master Theorem solves many of them:
If T(n)=c, n<d, and T(n)=aT(n/b)+f(n), n≥d,
and for small constants e>0, k>=0, and d<1 then

logb a 
) then T(n) is Q(n

If f(n) is O(n

If f(n) is Q( nlogb a logk

log a 
If f(n) is W(n b
T(n) is Q(f(n))
logb a
)
n ) then T(n) is Q( nlogb a logk 1 n )
) and a f(n/b) ≤ df(n), for n ≤ d, then
An Important use of Recurrence
Relations by Donald Knuth
09-0804
More Divide and Conquer Examples




pow(x,n)
log(n,base)
Point-in-convex-polygon
Fractals
09-0804
09-0804

http://www.3villagecsd.k12.ny.us/wmhs/Departments/Math/OBrien/fibonacci2.html

http://www.religiousforums.com/forum/material-world/80450-fibonacci-sequence.html
Creating Fractals



Koch Snowflake
Fractal Star
Your own…
Integer Multiplication

Algorithm: Multiply two n-bit integers I and J.

Divide step: Split I and J into high-order and low-order bits
I  I h 2n / 2  I l
J  J h 2n / 2  J l

We can then define I*J by multiplying the parts and adding:
I * J  ( I h 2n / 2  I l ) * ( J h 2n / 2  J l )
 I h J h 2n  I h J l 2n / 2  I l J h 2n / 2  I l J l


So, T(n) = 4T(n/2) + n, which implies T(n) is O(n2).
But that is no better than the algorithm we learned in grade
school.
An Improved Integer
Multiplication Algorithm

Algorithm: Multiply two n-bit integers I and J.

Divide step: Split I and J into high-order and low-order bits
I  I h 2n / 2  I l
J  J h 2n / 2  J l

Observe that there is a different way to multiply parts:
I * J  I h J h 2 n  [(I h  I l )(J l  J h )  I h J h  I l J l ]2 n / 2  I l J l
 I h J h 2 n  [(I h J l  I l J l  I h J h  I l J h )  I h J h  I l J l ]2 n / 2  I l J l
 I h J h 2 n  ( I h J l  I l J h )2 n / 2  I l J l


So, T(n) = 3T(n/2) + n, which implies T(n) is O(nlog23) [by
master theorem].
Thus, T(n) is O(n1.585).
Practice Analyzing Recursive
Functions
1. int fun1(int n) {
if (n==1) return 50;
return fun1(n/2) * fun1(n/2) + 97;
}
2. int fun2(int n) {
if (n==1) return 1;
int sum=0;
for (int i=1; i<=n; i++)
sum += fun(n-1);
return sum;
}
Dynamic Programming
Outline and Reading





Fibonacci
Making Change
Matrix Chain-Product (§5.3.1)
The General Technique (§5.3.2)
0-1 Knapsack Problem (§5.3.3)
Opinions?
public class Fib1 {
public static BigInteger fibonacci(long n)
{
if (n <= 2) return new BigInteger("1");
return fibonacci(n-1).add(fibonacci(n-2));
}
public static void main( String args[] )
{
long n = Integer.parseInt(args[0]);
System.out.println("Fib(" + n + ") = " +
fibonacci(n).toString());
}
}
Bottom up…
public class Fib2 {
static BigInteger[] fibs;
public static BigInteger fibonacci(int n) {
BigInteger result;
fibs[1]=new BigInteger("1");
fibs[2]=new BigInteger("1");
for (int i=3; i<=n; i++)
fibs[i] = fibs[i-1].add(fibs[i-2]);
return fibs[n];
}
public static void main( String args[] ) {
int n = Integer.parseInt(args[0]);
fibs = new BigInteger[n+1];
System.out.println("Fib(" + n + ") = " +
fibonacci(n).toString());
}
}
Memoize…
public class Fib3 {
static BigInteger[] fibs;
public static BigInteger fibonacci(int n) {
long result;
if (fibs[n] != null)
return fibs[n];
else {
fibs[n] = fibonacci(n-1).add(fibonacci(n-2));
return fibs[n];
}
}
public static void main( String args[] ) {
int n = Integer.parseInt(args[0]);
fibs = new BigInteger [n+1];
fibs[1] = new BigInteger("1");
fibs[2] = new BigInteger("1");
System.out.println("Fib("+n+") = " + fibonacci(n));
} }
Dynamic Programming



Applies to problems that can be solved
recursively, but the subproblems overlap
(Typically optimization problems)
Method:



Write definition of value you want to compute
Express solution in terms of sub-problem
solutions
Compute bottom-up or “memoize”
Making Change


Being good Computer Scientists, we want to
have a General Solution for making change
in our vending machine.
Suppose that our coin values were 40, 30, 5,
and 1 cent? How would me make change
using the smallest number of coins?
Making Change – DP solution



Suppose we have coin values c1, c2, …, ck and we want to make
change for value V using the smallest number of coins?
What sub-problem solutions would give us solution to the whole
problem?
Define NC(V) = smallest number of coins that will add up to V.
NC (V )  1  MINin1 NC (V  ci )


Base case?
Run time?


Some kind of exponential, if you’re not careful…
How can we implement this efficiently?
Dynamic Programming: Two
methods for efficiency

Build a table bottom-up


Compute NC(1), NC(2), …, NC(V), storing subproblem results in a table as you go up
Memoize

Compute NC(V) recursively, but store new subproblem results when you first compute them
Matrix Chain-Products

Dynamic Programming is a general algorithm
design paradigm.



Rather than give the general structure, let us
first give a motivating example:
Matrix Chain-Products
Review: Matrix Multiplication.


B
j
e
C = A*B
A is d × e and B is e × f
e
e 1
C[i, j ]   A[i, k ] * B[k , j ]
k 0

f
O(def ) time
d
A
C
i
i,j
f
d
Matrix Chain-Products

Matrix Chain-Product:




Compute A=A0*A1*…*An-1
Ai is di × di+1
Problem: How to parenthesize?
Example





B is 3 × 100
C is 100 × 5
D is 5 × 5
(B*C)*D takes 1500 + 75 = 1575 ops
B*(C*D) takes 1500 + 2500 = 4000 ops
An Enumeration Approach

Matrix Chain-Product Alg.:




Try all possible ways to parenthesize
A=A0*A1*…*An-1
Calculate number of ops for each one
Pick the one that is best
Running time:




The number of parethesizations is equal to the
number of binary trees with n nodes
This is exponential!
It is called the Catalan number, and it is almost 4n.
This is a terrible algorithm!
A Greedy Approach


Idea #1: repeatedly select the product that
uses (up) the most operations.
Counter-example:






A is 10 × 5
B is 5 × 10
C is 10 × 5
D is 5 × 10
Greedy idea #1 gives (A*B)*(C*D), which takes
500+1000+500 = 2000 ops
A*((B*C)*D) takes 500+250+250 = 1000 ops
Another Greedy Approach


Idea #2: repeatedly select the product that uses the
fewest operations.
Counter-example:







A is 101 × 11
B is 11 × 9
C is 9 × 100
D is 100 × 99
Greedy idea #2 gives A*((B*C)*D)), which takes
109989+9900+108900=228789 ops
(A*B)*(C*D) takes 9999+89991+89100=189090 ops
The greedy approach is not giving us the optimal
value.
A “Recursive” Approach

Define subproblems:




Find the best parenthesization of Ai*Ai+1*…*Aj.
Let Ni,j denote the number of operations done by this subproblem.
The optimal solution for the whole problem is N0,n-1.
Subproblem optimality: The optimal solution can be defined
in terms of optimal subproblems




There has to be a final multiplication (root of the expression tree)
for the optimal solution.
Say, the final multiply is at index i: (A0*…*Ai)*(Ai+1*…*An-1).
Then the optimal solution N0,n-1 is the sum of two optimal
subproblems, N0,i and Ni+1,n-1 plus the time for the last multiply.
If the global optimum did not have these optimal subproblems, we
could define an even better “optimal” solution.
A Characterizing
Equation


The global optimal has to be defined in terms of optimal
subproblems, depending on where the final multiply is at.
Let us consider all possible places for that final multiply:


Recall that Ai is a di × di+1 dimensional matrix.
So, a characterizing equation for Ni,j is the following:
N i , j  min{N i ,k  N k 1, j  di d k 1d j 1}
i k  j

Note that subproblems are not independent--the subproblems
overlap.
A Dynamic Programming
Algorithm





Algorithm matrixChain(S):
Since subproblems
Input: sequence S of n matrices to be multiplied
overlap, we don’t use
Output: number of operations in an optimal
recursion.
paranethization of S
Instead, we construct
for i  1 to n-1 do
optimal subproblems
Ni,i  0
“bottom-up.”
for b  1 to n-1 do
Ni,i’s are easy, so start
with them
for i  0 to n-b-1 do
Then do length 2,3,…
j  i+b
subproblems, and so
Ni,j  +infinity
on.
for k  i to j-1 do
3
Running time: O(n )
Ni,j  min{Ni,j , Ni,k +Nk+1,j +di dk+1 dj+1}
A Dynamic Programming
Algorithm Visualization





The bottom-up
construction fills in the N
array by diagonals
Ni,j gets values from
pervious entries in i-th row
and j-th column
Filling in each entry in the
N table takes O(n) time.
Total run time: O(n3)
Getting actual
parenthesization can be
done by remembering “k”
for each N entry
N i , j  min{N i ,k  N k 1, j  di d k 1d j 1}
i k  j
N
0
1
…
i
n-1
0
1 2
j …
n-1
answer
The General Dynamic
Programming Technique

Applies to a problem that at first seems to require a lot
of time (possibly exponential), provided we have:



Simple subproblems: the subproblems can be defined in
terms of a few variables, such as j, k, l, m, and so on.
Subproblem optimality: the global optimum value can be
defined in terms of optimal subproblems
Subproblem overlap: the subproblems are not independent,
but instead they overlap (hence, should be constructed
bottom-up).
The 0/1 Knapsack Problem

Given: A set S of n items, with each item i having




bi - a positive benefit
wi - a positive weight
Goal: Choose items with maximum total benefit but with weight at
most W.
If we are not allowed to take fractional amounts, then this is the
0/1 knapsack problem.

In this case, we let T denote the set of items we take

Objective: maximize
b
iT

Constraint:
w  W
iT
i
i
Example

Given: A set S of n items, with each item i having



bi - a positive benefit
wi - a positive weight
Goal: Choose items with maximum total benefit but with
weight at most W.
“knapsack”
Solution:
Items:
Weight:
Benefit:
1
2
3
4
5
4 in
2 in
2 in
6 in
2 in
$20
$3
$6
$25
$80
• 5 (2 in)
• 3 (2 in)
• 1 (4 in)
9 in
A 0/1 Knapsack Algorithm,
First Attempt



Sk: Set of items numbered 1 to k.
Define B[k] = best selection from Sk.
Problem: does not have subproblem optimality:

Consider S={(3,2),(5,4),(8,5),(4,3),(10,9)} weight-benefit pairs
Best for S4:
Best for S5:
A 0/1 Knapsack Algorithm,
Second Attempt



Sk: Set of items numbered 1 to k.
Define B[k,w] = best selection from Sk with weight
exactly equal to w
Good news: this does have subproblem optimality:
B[k  1, w]
if wk  w

B[k , w]  
else
max{B[k  1, w], B[k  1, w  wk ]  bk }

I.e., best subset of Sk with weight exactly w is either
the best subset of Sk-1 w/ weight w or the best
subset of Sk-1 w/ weight w-wk plus item k.
(Weight,Value): (4,6) (3,8) (5,10) (8,13) (7,15)
B
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
S1
0
0
0
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
S2
0
0
8
8
8
8
14 14
S3
0
0
8
8
10
10 14
14 14
14 14
14 14 14
14 14
14 14
14
S4
S5
B[k  1, w]
if wk  w

B[k , w]  
else
max{B[k  1, w], B[k  1, w  wk ]  bk }
Runtime?
How to tell which objects were used for each box?
The 0/1 Knapsack
Algorithm





B[k  1, w]
if wk  w

B[k , w]  
else
max{B[k  1, w], B[k  1, w  wk ]  bk }
Algorithm 01Knapsack(S, W):
Input: set S of items w/ benefit bi
Since B[k,w] is defined in terms
and weight wi; max. weight W
of B[k-1,*], we can reuse the
Output: benefit of best subset with
same array
weight at most W
Recall def. of B[k,w]:
Running time: O(nW).
Not a polynomial-time algorithm
if W is large
This is a pseudo-polynomial time
algorithm
for w  0 to W do
B[w]  0
for k  1 to n do
for w  W downto wk do
if B[w-wk]+bk > B[w] then
B[w]  B[w-wk]+bk