Transcript Theory of Computation - National Tsing Hua University

Chapter 15-2: Dynamic
Programming II
1
Matrix Multiplication
• Let A be a matrix of dimension p x q
and B be a matrix of dimension q x r
• Then, if we multiply matrices A and B,
we obtain a resulting matrix C = AB
whose dimension is p x r
• We can obtain each entry in C using q
operations  in total, pqr operations
2
Matrix Multiplication
Example :
 a1,1

 a 2 ,1
a1, 2
a2 , 2
a1, 3 

a2 , 3 
 b1,1

 b2 ,1
b
 3 ,1
b1, 2 
  c1,1
b2 , 2   
c 2 ,1


b3, 2 
c1, 2 

c2 , 2 
How to obtain c1,2 ?
3
Matrix Multiplication
• In fact, ((A1A2)A3) = (A1(A2A3)) so
that matrix multiplication is associative
Any way to write down the parentheses
gives the same result
E.g.,
(((A1A2)A3)A4) = ((A1A2)(A3A4))
= (A1((A2A3)A4)) = ((A1(A2A3))A4)
= (A1(A2(A3A4)))
4
Matrix Multiplication
Question: Why do we bother this?
Because different computation sequence
may use different number of operations!
E.g., Let the dimensions of A1, A2, A3 be:
1x100, 100x1, 1x100 , respectively
#operations to get ((A1A2)A3) = ??
#operations to get (A1(A2A3)) = ??
5
Optimal Substructure
(allows recursion)
Lemma: Suppose that to multiply B1,B2,…,Bn,
the way with minimum #operations is to:
(i) first, obtain B1B2 … Bx
(ii) then, obtain Bx+1 … Bn
(iii) finally, multiply the matrices of
part (i) and part (ii)
Then, the matrices in part (i) and part (ii)
must be obtained with min #operations
6
Optimal Substructure
Let fi,j denote the min #operations to obtain
the product AiAi+1 … Aj
 fi,i = 0
Let rk and ck denote #rows and #cols of Ak
Then, we have:
Lemma: For any j > i,
fi,j = minx
{ fi,x + fx+1,j + ri cx cj }
7
Recursive-Matrix-Chain
Define a function Compute_F(i,j) as follows:
Compute_F(i, j) /* Finding fi,j */
1. if (i == j) return 0;
2. m = ;
3. for (x = i, i+1, …, j-1) {
g = Compute_F(i,x) + Compute_F(x+1,j) + ri cx cj ;
if (g  m) m = g;
}
4. return m ;
8
The recursion tree for the computation
of RECURSUVE-MATRIX-CHAIN(P, 1, 4)
9
Time Complexity
• Question: Time to get Compute_F(1,n)?
T (1)  1

T (n)  1  n1(T (k )  T (n  k )  1) for n  1

k 1
n 1
T ( n )  2  T (i )  n
i 1
• By substituion method, we can show that
• Running time = W(3n)
10
Counting the number of
parenthesizations
1
if n  1


P(n )   n 1
P(k ) p(n  k ) if n  2


 k 1
1  2n  2 
4n
 
  W( 3 / 2 )
n  n 1 
n
• Remark: On the other hand, #operations
for each possible way of writing
parentheses are computed at most once 
Running time = O( C(2n-2,n-1)/n )  Catalan
Number
11
Overlapping Subproblems
Here, we can see that :
To Compute_F(i,j) and Compute_F(i,j+1),
both have many COMMON subproblems:
Compute_F(i,i+1), …, Compute_F(i,j-1)
So, in our recursive algorithm, there are
many redundant computations !
Question: Can we avoid it ?
12
Bottom-Up Approach
• We notice that
fi,j depends only on fx,y with y-x  j-i
• Let us create a 2D table F to store all fi,j
values once they are computed
• Then, compute fi,j for j-i = 1,2,…,n-1
13
Bottom-Up Approach
BottomUp_F( ) /* Finding min #operations
1. for j = 1, 2, …, n, set F[j, j] = 0 ;
2. for (length = 1,2,…, n-1) {
Compute F[i,i+length] for all i;
*/
// Based on F[x,y] with |x-y| < length
}
3. return F[1,n] ;
Running Time = Q(n3)
14
Example:
A1
A2
30  35  p0  p1
35  15  p1  p2
A3 15  5
A4 5  10
A5 10  20
A6 20  25




p2  p3
p3  p4
p4  p5
p5  p6
15
The m and s table computed by
MATRIX-CHAIN-ORDER for n = 6
Optimal Solution: (( A1 ( A2 A3 ))((A4 A5 ) A6 ))
16
Example
m[2,5]=
Min {
m[2,2]+m[3,5]+p1p2p5=0+2500+351520=13000,
m[2,3]+m[4,5]+p1p3p5=2625+1000+35520=7125,
m[2,4]+m[5,5]+p1p4p5=4375+0+351020=11374
} =7125
17
Remarks
• Again, a slight change in the algorithm
allows us to get the exact sequence of
steps (or the parentheses) that achieves
the minimum number of operations
• Also, we can make minor changes to the
recursive algorithm and obtain a memoized
version (whose running time is O(n3))
18
MATRIX_CHAIN_ORDER
MATRIX_CHAIN_ORDER(p)
1 n  length[p] –1
2 for i  1 to n
3
do m[i, i]  0
4 for l  2 to n
5
do for i  1 to n – l + 1
6
do j  i + l – 1
7
m[i, j]  
8
for k  i to j – 1
9
do q  m[i, k] + m[k+1, j]+ pi-1pkpj
10
if q < m[i, j]
11
then m[i, j]  q
12
s[i, j]  k
13 return m and s
19
When should we apply DP?
• Optimal structure: an optimal
solution to the problem contains
optimal solutions to subproblems.
– Example: Matrix-multiplication problem
• Overlapping subproblems: a recursive
algorithm revisits the same
subproblem over and over again.
20
Common pattern in discovering
optimal substructure
1. Show that a solution to a problem consists of
making a choice, which leaves one or
subproblems to solve.
2. Suppose that for a given problem, you are given
the choice that leads to an optimal solution. You
do not concern how to determine this choice.
You just assume that it has been given to you.
3. Given this choice, determine which subproblems
ensure and how to best characterize the
resulting (solution) space of subproblems.
21
Common pattern in discovering
optimal substructure
4. Show that the solutions to the subproblems
used within the optimal solution must
themselves be optimal. Usually use “cut-andpaste” technique.
22
Optimal substructure
• Optimal substructure varies across problem
domains in two ways:
1. how many subproblems are used in an
optimal solution to the original problem, and
2. how many choices we have in determining
which subproblem(s) to use in an optimal
solution.
• Informally, running time depends on (# of
subproblems overall) x (# of choices).
23
Refinement
• One should be careful not to assume that
optimal substructure applies when it does not.
Consider the following two problems in which
we are given a directed graph G = (V, E) and
vertices u, v  V.
– Unweighted shortest path:
• Find a path from u to v consisting of the fewest
edges. Good for Dynamic programming.
– Unweighted longest simple path:
• Find a simple path from u to v consisting of the
most edges. Not good for Dynamic programming.
24
Shortest path
• Shortest path has optimal
substructure.
•
•
•
•
Suppose p is shortest path u -> v.
Let w be any vertex on p.
Let p1 be the portion of p going u -> w.
Then p1 is a shortest path u -> w.
25
Longest simple path
• Does longest path have optimal
substructure?
• Consider q -> r -> t = longest path q -> t.
Are its subpaths longest paths? No!
• Longest simple path q -> r is q -> s -> t -> r.
• Longest simple path r -> t is r -> q ->s -> t .
• Not only isn’t there optimal substructure,
but we can’t even assemble a legal solution
26
from solutions to subproblems.
Overlapping subproblems
• These occur when a recursive
algorithm revisits the same problem
over and over.
• Solutions
1. Bottom up
2. Memorization (memorize the natural,
but inefficient)
27
Top down approach
RECURSIVE_MATRIX_CHAIN(p, i, j)
1
2
3
4
5
6
7
8
if i = j
then return 0
m[i, j]  
for k  i to j – 1
do q  RMC(p,i,k) + RMC(p,k+1,j) + pi-1pkpj
if q < m[i, j]
then m[i, j]  q
return m[i, j]
28
The recursion tree for the computation
of RECURSUVE-MATRIX-CHAIN(P, 1, 4)
29
Memoization
• Alternative approach to dynamic programming:
“Store, don’t recompute.”
Make a table indexed by subproblem.
When solving a subproblem: Lookup in table.
If answer is there, use it.
Else, compute answer, then store it.
In bottom-up dynamic programming, we go
one step further. We determine in what
order we’d want to access the table, and fill
it in that way.
30
MEMORIZED_MATRIX_CHAIN
MEMORIZED_MATRIX_CHAIN(p)
1 n  length[p] –1
2 for i  1 to n
3
do for j  1 to n
4
do m[i, j]  
5 return LC (m,p,1,n)
31
LOOKUP_CHAIN
LOOKUP_CHAIN(m, p,i,j)
1 if m[i, j] < 
2
then return m[i, j]
3 if i = j
4 then m[i, j]  0
5 else for k  i to j – 1
6
do q  LC(m,p,i,k) +LC(m,p,k+1,j)+pi-1pkpj
7
if q < m[i, j]
8
then m[i, j]  q
9 return m[i, j]
Time Complexity: O(n3 )
32
Homework
• Problem: 15.2
• Practice at home: 15.2-4, 15.2-5,
15.3-1, 15.3-4
33