Transcript PPT

CS 332: Algorithms
Amortized Analysis Continued
Longest Common Subsequence
Dynamic Programming
David Luebke
1
7/27/2016
Administrivia
Midterm almost graded
 Homework 4 assigned


Due: Tuesday 28 (after Thanksgiving break)
David Luebke
2
7/27/2016
Review: MST Algorithms
In a connected, weighted, undirected graph,
will the edge with the lowest weight be in the
MST? Why or why not?
 Yes:



If T is MST of G, and A  T is a subtree of T, and
(u,v) is the min-weight edge connecting A to V-A,
then (u,v)  T
The lowest-weight edge must be in the tree (A=)
David Luebke
3
7/27/2016
Review: MST Algorithms
What do the disjoint sets in Kruskal’s
algorithm represent?
 A: Parts of the graph we have connected up
together so far

David Luebke
4
7/27/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2
19
{
14
17
T = ;
8
25
5
for each v  V
21
13
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T  {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
5
9
1
7/27/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2
19
{
9
14
17
T = ;
8
25
5
for each v  V
21
13
1?
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T  {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
6
7/27/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2
19
{
14
17
T = ;
8
25
5
for each v  V
21
13
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T  {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
7
9
1
7/27/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2?
19
{
14
17
T = ;
8
25
5
for each v  V
21
13
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T  {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
8
9
1
7/27/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2
19
{
14
17
T = ;
8
25
5
for each v  V
21
13
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T  {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
9
9
1
7/27/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2
19
{
9
14
17
T = ;
8
25
5?
for each v  V
21
13
1
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T  {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
10
7/27/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2
19
{
14
17
T = ;
8
25
5
for each v  V
21
13
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T  {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
11
9
1
7/27/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2
19
{
14
17
T = ;
8?
25
5
for each v  V
21
13
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T  {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
12
9
1
7/27/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2
19
{
14
17
T = ;
8
25
5
for each v  V
21
13
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T  {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
13
9
1
7/27/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2
19
{
9?
14
17
T = ;
8
25
5
for each v  V
21
13
1
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T  {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
14
7/27/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2
19
{
14
17
T = ;
8
25
5
for each v  V
21
13
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T  {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
15
9
1
7/27/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2
19
{
14
17
T = ;
8
25
5
for each v  V
21
13?
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T  {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
16
9
1
7/27/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2
19
{
14
17
T = ;
8
25
5
for each v  V
21
13
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T  {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
17
9
1
7/27/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2
19
{
14?
17
T = ;
8
25
5
for each v  V
21
13
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T  {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
18
9
1
7/27/2016
Kruskal’s Algorithm
Run the algorithm:
Kruskal()
2
19
{
14
17
T = ;
8
25
5
for each v  V
21
13
MakeSet(v);
sort E by increasing edge weight w
for each (u,v)  E (in sorted order)
if FindSet(u)  FindSet(v)
T = T  {{u,v}};
Union(FindSet(u), FindSet(v));
}
David Luebke
19
9
1
7/27/2016
Review: Shortest-Path Algorithms
How does the Bellman-Ford algorithm work?
 How can we do better for DAGs?
 Under what conditions can we use Dijkstra’s
algorithm?

David Luebke
20
7/27/2016
Review: Running Time of
Kruskal’s Algorithm

Expensive operations:





Sort edges: O(E lg E)
O(V) MakeSet()’s
O(E) FindSet()’s
O(V) Union()’s
Upshot:

Comes down to efficiency of disjoint-set
operations, particularly Union()
David Luebke
21
7/27/2016
Review: Disjoint Set Union

So how do we represent disjoint sets?

Naïve implementation: use a linked list to
represent elements, with pointers back to set:
 MakeSet():
O(1)
 FindSet(): O(1)
 Union(A,B): “Copy” elements of A into set B by
adjusting elements of A to point to B: O(A)

How long could n Union()s take? O(n2), worst case
David Luebke
22
7/27/2016
Disjoint Set Union: Analysis

Worst-case analysis: O(n2) time for n Union’s
Union(S1, S2)
Union(S2, S3)
…
Union(Sn-1, Sn)

“copy”
“copy”
1 element
2 elements
“copy”
n-1 elements
O(n2)
Improvement: always copy smaller into larger



How long would above sequence of Union’s take?
Worst case: n Union’s take O(n lg n) time
Proof uses amortized analysis
David Luebke
23
7/27/2016
Amortized Analysis of Disjoint Sets

If elements are copied from the smaller set into
the larger set, an element can be copied at most
lg n times

Worst case: Each time copied, element in smaller set
1st time
2nd time
…
(lg n)th time
David Luebke
resulting set size
2
4
n
24
7/27/2016
Amortized Analysis of Disjoint Sets
Since we have n elements each copied at most
lg n times, n Union()’s takes O(n lg n) time
 Therefore we say the amortized cost of a
Union() operation is O(lg n)
 This is the aggregate method of amortized
analysis:



n operations take time T(n)
Average cost of an operation = T(n)/n
David Luebke
25
7/27/2016
Amortized Analysis:
Accounting Method

Accounting method





Charge each operation an amortized cost
Amount not used stored in “bank”
Later operations can used stored money
Balance must not go negative
Book also discusses potential method

But we won’t worry about it here
David Luebke
26
7/27/2016
Accounting Method Example:
Dynamic Tables
Implementing a table (e.g., hash table) for
dynamic data, want to make it small as possible
 Problem: if too many items inserted, table may
be too small
 Idea: allocate more memory as needed

David Luebke
27
7/27/2016
Dynamic Tables
1. Init table size m = 1
2. Insert elements until number n > m
3. Generate new table of size 2m
4. Reinsert old elements into new table
5. (back to step 2)
 What is the worst-case cost of an insert?
 One insert can be costly, but the total?
David Luebke
28
7/27/2016
Analysis Of Dynamic Tables
Let ci = cost of ith insert
 ci = i if i-1 is exact power of 2, 1 otherwise
 Example:


Operation
Table Size
Insert(1)
1
David Luebke
Cost
1
29
1
7/27/2016
Analysis Of Dynamic Tables
Let ci = cost of ith insert
 ci = i if i-1 is exact power of 2, 1 otherwise
 Example:


Operation
Table Size
Insert(1)
Insert(2)
1
2
David Luebke
Cost
1
1 + 1
30
1
2
7/27/2016
Analysis Of Dynamic Tables
Let ci = cost of ith insert
 ci = i if i-1 is exact power of 2, 1 otherwise
 Example:


Operation
Table Size
Insert(1)
Insert(2)
Insert(3)
1
2
4
David Luebke
Cost
1
1 + 1
1 + 2
31
1
2
3
7/27/2016
Analysis Of Dynamic Tables
Let ci = cost of ith insert
 ci = i if i-1 is exact power of 2, 1 otherwise
 Example:


Operation
Table Size
Insert(1)
Insert(2)
Insert(3)
Insert(4)
1
2
4
4
David Luebke
Cost
1
1 + 1
1 + 2
1
32
1
2
3
4
7/27/2016
Analysis Of Dynamic Tables
Let ci = cost of ith insert
 ci = i if i-1 is exact power of 2, 1 otherwise
 Example:


Operation
Table Size
Insert(1)
Insert(2)
Insert(3)
Insert(4)
Insert(5)
1
2
4
4
8
David Luebke
Cost
1
1 + 1
1 + 2
1
1 + 4
33
1
2
3
4
5
7/27/2016
Analysis Of Dynamic Tables
Let ci = cost of ith insert
 ci = i if i-1 is exact power of 2, 1 otherwise
 Example:


Operation
Table Size
Insert(1)
Insert(2)
Insert(3)
Insert(4)
Insert(5)
Insert(6)
1
2
4
4
8
8
David Luebke
Cost
1
1 + 1
1 + 2
1
1 + 4
1
34
1
2
3
4
5
6
7/27/2016
Analysis Of Dynamic Tables
Let ci = cost of ith insert
 ci = i if i-1 is exact power of 2, 1 otherwise
 Example:


Operation
Table Size
Insert(1)
Insert(2)
Insert(3)
Insert(4)
Insert(5)
Insert(6)
Insert(7)
1
2
4
4
8
8
8
David Luebke
Cost
1
1 + 1
1 + 2
1
1 + 4
1
1
35
1
2
3
4
5
6
7
7/27/2016
Analysis Of Dynamic Tables
Let ci = cost of ith insert
 ci = i if i-1 is exact power of 2, 1 otherwise
 Example:


Operation
Table Size
Insert(1)
Insert(2)
Insert(3)
Insert(4)
Insert(5)
Insert(6)
Insert(7)
Insert(8)
1
2
4
4
8
8
8
8
David Luebke
Cost
1
1 + 1
1 + 2
1
1 + 4
1
1
1
36
1
2
3
4
5
6
7
8
7/27/2016
Analysis Of Dynamic Tables
Let ci = cost of ith insert
 ci = i if i-1 is exact power of 2, 1 otherwise
 Example:


Operation
Table Size
Insert(1)
Insert(2)
Insert(3)
Insert(4)
Insert(5)
Insert(6)
Insert(7)
Insert(8)
Insert(9)
1
2
4
4
8
8
8
8
16
David Luebke
Cost
1
1
1
1
1
1
1
1
1
37
+ 1
+ 2
1
2
3
4
5
6
7
1
8
2
9
+ 4
+ 8
7/27/2016
Aggregate Analysis

n Insert() operations cost
n
c
i 1
i
lg n
 n   2  n  (2n  1)  3n
j
j 0
Average cost of operation
= (total cost)/(# operations) < 3
 Asymptotically, then, a dynamic table costs the
same as a fixed-size table


Both O(1) per Insert operation
David Luebke
38
7/27/2016
Accounting Analysis

Charge each operation $3 amortized cost



Use $1 to perform immediate Insert()
Store $2
When table doubles



$1 reinserts old item, $1 reinserts another old item
Point is, we’ve already paid these costs
Upshot: constant (amortized) cost per operation
David Luebke
39
7/27/2016
Accounting Analysis

Suppose must support insert & delete, table
should contract as well as expand





Table overflows  double it (as before)
Table < 1/2 full  halve it: BAD IDEA (Why?)
Better: Table < 1/4 full  halve it
Charge $3 for Insert (as before)
Charge $2 for Delete
 Store
extra $1 in emptied slot
 Use later to pay to copy remaining items to new table
when shrinking table
David Luebke
40
7/27/2016
Dynamic Programming

Another strategy for designing algorithms is
dynamic programming



A metatechnique, not an algorithm (like divide &
conquer)
The word “programming” is historical and
predates computer programming
Use when problem breaks down into recurring
small subproblems


This lecture: a driving problem
Next lecture: the algorithm
David Luebke
41
7/27/2016
Dynamic Programming Example:
Longest Common Subsequence

Longest common subsequence (LCS) problem:



Given two sequences x[1..m] and y[1..n], find the
longest subsequence which occurs in both
Ex: x = {A B C B D A B }, y = {B D C A B A}
{B C} and {A A} are both subsequences of both
 What

is the LCS?
Brute-force algorithm: For every subsequence of x,
check if it’s a subsequence of y
 How
many subsequences of x are there?
 What will be the running time of the brute-force alg?
David Luebke
42
7/27/2016
LCS Algorithm
Brute-force algorithm: 2m subsequences of x to
check against n elements of y: O(n 2m)
 We can do better: for now, let’s only worry
about the problem of finding the length of LCS



When finished we will see how to backtrack from
this solution back to the actual LCS
Define c[i,j] to be the length of the LCS of
x[1..i] and y[1..j]

What is the length of LCS of x and y?
David Luebke
43
7/27/2016
Finding LCS Length

Theorem:
if x[i ]  y[ j ],
c[i  1, j  1]  1
c[i, j ]  
 max( c[i, j  1], c[i  1, j ]) otherwise

What is this really saying?
David Luebke
44
7/27/2016
The End
David Luebke
45
7/27/2016