Transcript Algorithms and Data Structures

Algorithms and Data
Structures
Lecture III
Simonas Šaltenis
Nykredit Center for Database Research
Aalborg University
[email protected]
September 12, 2002
1
This Lecture


Divide-and-conquer technique for
algorithm design. Example – the merge
sort.
Writing and solving recurrences
September 12, 2002
2
Divide and Conquer

Divide-and-conquer method for algorithm
design:



Divide: If the input size is too large to deal
with in a straightforward manner, divide the
problem into two or more disjoint subproblems
Conquer: Use divide and conquer recursively
to solve the subproblems
Combine: Take the solutions to the
subproblems and “merge” these solutions into
a solution for the original problem
September 12, 2002
3
Merge Sort Algorithm



Divide: If S has at least two elements (nothing
needs to be done if S has zero or one elements),
remove all the elements from S and put them
into two sequences, S1 and S2 , each containing
about half of the elements of S. (i.e. S1 contains
the first n/2elements and S2 contains the
remaining n/2elements).
Conquer: Sort sequences S1 and S2 using Merge
Sort.
Combine: Put back the elements into S by
merging the sorted sequences S1 and S2 into one
sorted sequence
September 12, 2002
4
Merge Sort: Algorithm
Merge-Sort(A, p, r)
if p < r then
q(p+r)/2
Merge-Sort(A, p, q)
Merge-Sort(A, q+1, r)
Merge(A, p, q, r)
Merge(A, p, q, r)
Take the smallest of the two topmost elements of
sequences A[p..q] and A[q+1..r] and put into the
resulting sequence. Repeat this, until both sequences
are empty. Copy the resulting sequence into A[p..r].
September 12, 2002
5
MergeSort (Example) - 1
September 12, 2002
6
MergeSort (Example) - 2
September 12, 2002
7
MergeSort (Example) - 3
September 12, 2002
8
MergeSort (Example) - 4
September 12, 2002
9
MergeSort (Example) - 5
September 12, 2002
10
MergeSort (Example) - 6
September 12, 2002
11
MergeSort (Example) - 7
September 12, 2002
12
MergeSort (Example) - 8
September 12, 2002
13
MergeSort (Example) - 9
September 12, 2002
14
MergeSort (Example) - 10
September 12, 2002
15
MergeSort (Example) - 11
September 12, 2002
16
MergeSort (Example) - 12
September 12, 2002
17
MergeSort (Example) - 13
September 12, 2002
18
MergeSort (Example) - 14
September 12, 2002
19
MergeSort (Example) - 15
September 12, 2002
20
MergeSort (Example) - 16
September 12, 2002
21
MergeSort (Example) - 17
September 12, 2002
22
MergeSort (Example) - 18
September 12, 2002
23
MergeSort (Example) - 19
September 12, 2002
24
MergeSort (Example) - 20
September 12, 2002
25
MergeSort (Example) - 21
September 12, 2002
26
MergeSort (Example) - 22
September 12, 2002
27
Merge Sort Revisited

To sort n numbers




if n=1 done!
recursively sort 2 lists of
numbers n/2 and n/2
elements
merge 2 sorted lists in Q(n)
time
Strategy



break problem into similar
(smaller) subproblems
recursively solve
subproblems
combine solutions to answer
September 12, 2002
28
Recurrences


Running times of algorithms with Recursive
calls can be described using recurrences
A recurrence is an equation or inequality that
describes a function in terms of its value on
smaller inputs
solving_trivial_problem
if n  1

T (n)  
num_pieces T (n / subproblem_size_factor)  dividing  combining if n  1

Example: Merge Sort
Q(1)
if n  1

T (n)  
2T (n / 2)  Q(n) if n  1
September 12, 2002
29
Solving Recurrences

Repeated substitution method


Substitution method




Expanding the recurrence by substitution and noticing
patterns
guessing the solutions
verifying the solution by the mathematical induction
Recursion-trees
Master method

templates for different classes of recurrences
September 12, 2002
30
Repeated Substitution Method

Let’s find the running time of merge sort (let’s
assume that n=2b, for some b).
1
if n  1

T (n)  
2T (n / 2)  n if n  1
T (n)  2T  n / 2   n
substitute
 2  2T  n / 4   n / 2   n expand
 22 T (n / 4)  2n substitute
 22 (2T (n / 8)  n / 4)  2n expand
 23 T (n / 8)  3n
observe the pattern
T (n)  2i T (n / 2i )  in
 2lg n T (n / n)  n lg n  n  n lg n
September 12, 2002
31
Repeated Substitution Method

The procedure is straightforward:








Substitute
Expand
Substitute
Expand
…
Observe a pattern and write how your expression looks
after the i-th substitution
Find out what the value of i (e.g., lgn) should be to get
the base case of the recurrence (say T(1))
Insert the value of T(1) and the expression of i into
your expression
September 12, 2002
32
Substitution method
Solve T (n)  4T (n / 2)  n
1) Guess that T (n)  O(n3 ), i.e., that T of the form cn3
2) Assume T (k )  ck 3 for k  n / 2 and
3) Prove T (n)  cn3 by induction
T (n)  4T (n / 2)  n (recurrence)
 4c(n/2)3  n (ind. hypoth.)
c 3

n  n (simplify)
2
c

 cn3   n3  n  (rearrange)
2

 cn3 if c  2 and n  1 (satisfy)
Thus T (n)  O(n3 )!
Subtlety: Must choose c big enough to handle
T (n)  Q(1) for n  n0 for some n0
September 12, 2002
33
Substitution Method

Achieving tighter bounds
Try to show T (n)  O( n 2 )
Assume T (k )  ck 2
T (n)  4T (n / 2)  n
 4c( n / 2) 2  n
 cn 2  n
 cn 2 for no choice of c  0.
September 12, 2002
34
Substitution Method (2)

The problem? We could not rewrite the equality
T (n)  cn2 + (something positive)

as:
T (n)  cn2


in order to show the inequality we wanted
Sometimes to prove inductive step, try to
strengthen your hypothesis

T(n)  (answer you want) - (something > 0)
September 12, 2002
35
Substitution Method (3)

Corrected proof: the idea is to strengthen
the inductive hypothesis by subtracting
lower-order terms!
Assume T (k )  c1k 2  c2 k for k  n
T ( n) 




September 12, 2002
4T (n / 2)  n
4(c1 (n / 2) 2  c2 (n / 2))  n
c1n 2  2c2 n  n
c1n 2  c2 n  (c2 n  n)
c1n 2  c2 n if c2  1
36
Recursion Tree


A recursion tree is a convenient way to visualize
what happens when a recurrence is iterated
Construction of a recursion tree
T (n)  T (n / 4)  T (n / 2)  n2
September 12, 2002
37
Recursion Tree (2)
September 12, 2002
38
Recursion Tree (3)
T (n)  T (n /3)  T (2n /3)  n
September 12, 2002
39
Master Method



The idea is to solve a class of recurrences that
have the form
T (n)  aT (n / b)  f (n)
a >= 1 and b > 1, and f is asymptotically
positive!
Abstractly speaking, T(n) is the runtime for an
algorithm and we know that


a subproblems of size n/b are solved recursively, each
in time T(n/b)
f(n) is the cost of dividing the problem and combining
the results. In merge-sort T (n)  2T (n / 2)  Q(n)
September 12, 2002
40
Master Method (2)
Split problem into a parts at logbn
levels. There are alogb n  nlogb a leaves
September 12, 2002
41
Master Method (3)


Number of leaves: alogb n  nlogb a
Iterating the recurrence, expanding the tree
yields T (n)  f (n)  aT (n / b)


f (n)  af (n / b)  a 2T (n / b 2 )
f (n)  af (n / b)  a 2T (n / b 2 )  ...
 a logb n1 f (n / blogb n1 )  a logb nT (1)
Thus,
T ( n) 
logb n 1

a j f (n / b j )  Q(nlogb a )
j 0


The first term is a division/recombination cost (totaled
across all levels of the tree)
log a
The second term is the cost of doing all n b subproblems
of size 1 (total of all work pushed to leaves)
September 12, 2002
42
MM Intuition

Three common cases:





Running time dominated by cost at leaves
Running time evenly distributed throughout
the tree
Running time dominated by cost at root
Consequently, to solve the recurrence, we
need only to characterize the dominant
term
In each case compare f (n) with O(nlog a )
b
September 12, 2002
43
MM Case 1

f (n)  O(nlogb a ) for some constant   0


f(n) grows polynomially (by factor
than nlogb a

n
) slower
The work at the leaf level dominates
logb a
 Summation of recursion-tree levels O( n
)
logb a
 Cost of all the leaves Q( n
)
logb a
 Thus, the overall cost Q( n
)
September 12, 2002
44
MM Case 2


f (n)  Q(n
lg n)
logb a
f
(
n
)

and n
are asymptotically the same
logb a
The work is distributed equally
throughout the tree T (n)  Q(nlogb a lg n)

(level cost)  (number of levels)
September 12, 2002
45
MM Case 3

f (n)  (nlogb a ) for some constant   0



Inverse of Case 1
logb a
f(n) grows polynomially faster than n
Also need a regularity condition
c  1 and n0  0 such that af (n / b)  cf (n) n  n0

The work at the root dominates
T (n)  Q( f (n))
September 12, 2002
46
Master Theorem Summarized

Given a recurrence of the form T (n)  aT (n / b)  f (n)
1.

f (n)  O n logb a 


 T (n)  Q nlogb a
2.

f (n)  Q nlogb a



 T (n)  Q nlogb a lg n
3.



f (n)   n logb a  and af (n / b)  cf (n), for some c  1, n  n0
 T ( n)  Q  f ( n) 

The master method cannot solve every recurrence of this
form; there is a gap between cases 1 and 2, as well as
cases 2 and 3
September 12, 2002
47
Strategy





Extract a, b, and f(n) from a given recurrence
log b a
Determine n
log b a
Compare f(n) and n
asymptotically
Determine appropriate MT case, and apply
Example merge sort
T (n)  2T (n / 2)  Q(n)
a  2, b  2; nlogb a  nlog2 2  n  Q(n)
Also f (n)  Q(n)


 Case 2: T (n)  Q nlogb a lg n  Q  n lg n 
September 12, 2002
48
Examples
T (n)  T (n / 2)  1
a  1, b  2; n log2 1  1
also f ( n)  1, f (n)  Q(1)
 Case 2: T (n)  Q(lg n)
Binary-search(A, p, r, s):
q(p+r)/2
if A[q]=s then return q
else if A[q]>s then
Binary-search(A, p, q-1, s)
else Binary-search(A, q+1, r, s)
T (n)  9T (n / 3)  n
a  9, b  3;
f (n)  n, f (n)  O(n log3 9 ) with   1
 Case 1: T (n)  Q  n 2 
September 12, 2002
49
Examples (2)
T (n)  3T (n / 4)  n lg n
a  3, b  4; n log4 3  n 0.793
f (n)  n lg n, f (n)  (n log4 3 ) with   0.2
 Case 3:
Regularity condition
af (n / b)  3(n / 4) lg(n / 4)  (3 / 4)n lg n  cf (n) for c  3 / 4
T (n)  Q(n lg n)
T (n)  2T (n / 2)  n lg n
a  2, b  2; nlog2 2  n1
f (n)  n lg n, f (n)  (n1 ) with  ?
also n lg n / n1  lg n
 neither Case 3 nor Case 2!
September 12, 2002
50
Examples (3)
T (n)  4T (n / 2)  n3
a  4, b  2; n log2 4  n 2
f ( n )  n 3 ; f ( n )  ( n 2 )
 Case 3: T (n)  Q  n3 
Checking the regularity condition
4 f (n / 2)  cf (n)
4n3 / 8  cn3
n3 / 2  cn3
c  3/ 4 1
September 12, 2002
51
Next Week

Sorting


QuickSort
HeapSort
September 12, 2002
52