Transcript Divide & Conquer Algorithms - American University in Cairo

Analysis & Design of
Algorithms
(CSCE 321)
Prof. Amr Goneid
Department of Computer Science, AUC
Part 7. Divide & Conquer
Algorithms
Prof. Amr Goneid, AUC
1
Divide & Conquer Algorithms
Prof. Amr Goneid, AUC
2
Divide & Conquer Algorithms









The General Method
Time Complexity
Common Recurrence relations
A General Divide & Conquer Recurrence
Merge Sort Algorithm
Quicksort Algorithm
An Application of Quicksort: Convex Hull
Selection by Partitioning
Closest Pair Problem
Prof. Amr Goneid, AUC
3
1. The General Method
To solve a problem using Divide and
Conquer:
• Divide it into smaller problems
• Solve the smaller problems recursively
• Combine their solutions into a solution
for the big problem
Prof. Amr Goneid, AUC
4
The General Method
 The base case for recursion is a sub-
problem of constant (and small) size.
 Analysis of D&Q algorithms is done by
constructing a recurrence relation
 We can obtain T(n) in a closed form by
solving the recurrence relation
Prof. Amr Goneid, AUC
5
The General Method
 For problem p, small(p) = true if p is simple. In this
case S(p) is the solution. Otherwise, Divide, Conquer,
then Combine.
 Divide & Conquer Algorithm
DQ(p)
{ if small(p) return S(p);
else {
1. Divide p into smaller instances p1,p2,..,pk
2. Apply DQ to each pi
3. Return Combine(DQ(p1),DQ(p2),…,DQ(pk))}
}
Prof. Amr Goneid, AUC
6
2. Time Complexity
Let g(n) = time for S(p) when p is small
f(n) = time to divide p and combine
solutions
then
for p small T(n) = g(n)
otherwise
k
T (n)   T (ni )  f (n)
i 1
Prof. Amr Goneid, AUC
7
3. Common Recurrence Relations
 A recursive algorithm that halves the input in one step
then processes the two halves:
T(n) = 2T(n/2) + 1 for n >1
with T(1) given
 A recursive algorithm that halves the input in one step
then processes one half and ignores the other:
T(n) = T(n/2) + 1
for n >1
with T(1) given
Prof. Amr Goneid, AUC
8
Common Recurrence Relations
 A recursive algorithm that examines every element
before dividing and then ignores one half:
T(n) = T(n/2) + n
for n >1 with T(1) given
 A recursive algorithm that makes a linear pass through
the input, before, during or after it is split into two
halves then processes the two halves:
T(n) = 2 T(n/2) + n for n >1
with T(1) given
Prof. Amr Goneid, AUC
9
Example: Recursive Binary Search
Algorithm
BinSearch (a , s , e , x)
{ if (s == e)
// if small (p), i.e. one element
{ if (x == as) return s ; else return -1 ;}
// return S(p)
else {
// two or more elements
mid = (s + e)/2 ;
// Divide
if (x == amid) return mid ;
else if (x < amid)
// Conquer
return BinSearch (a , s , mid-1 , x) ;
// n/2 or
else return BinSearch (a , mid+1 , e , x) ; } // n/2
}
// No Combine
Prof. Amr Goneid, AUC
10
Binary Search Demo
http://www.cosc.canterbury.ac.nz/people/
mukundan/dsal/BSearch.html
Prof. Amr Goneid, AUC
11
Worst Case of Binary Search
Let T(n) be the worst case number of comparisons
used on an array of n items. Hence
T ( n )  T ( n / 2 )  2 for n  1 with T ( 1 )  1
Now we seek a closed form solution for T(n)
Prof. Amr Goneid, AUC
12
Worst Case of Binary Search
T ( n )  T ( n / 2 )  2 for n  1 with T ( 1 )  1
Assume n  2 , m  0,1,2,.....,n / 2  2
m
m 1
Hence,m  log n , n  1  m  0, n  1  m  0
We obtain a secondaryrecurrence
T ( m )  T ( m 1 )  2
for m  0 with T ( 0 )  1
m
HenceT ( m )  1   2  1  2m
i 1
and T ( n )  2 log n  1  O(log n )
Prof. Amr Goneid, AUC
13
4. A General D&Q Recurrence
Many D & Q algorithmshavea recurrence
T ( n )  aT( n / c )  f ( n ) ,
for n  c with T ( 1 ) givenor n  c withT ( c ) given.
where a, c are constants,c  2,3,...
Let n  c m , m  0 ,1,... Hence m  logc n
n / c  c m1 and ( n  1 )  ( m  0 ) ,( n  c )  ( m  1 )
Hence T ( c m )  aT( c m1 )  f ( c m )
SecondaryRecurrence:
F ( m )  aF( m  1 )  f ( c m )
If T(1) is givenit means that F(0) is given.Similarly,
if T(c) is givenit means that F(1) is given.
Prof. Amr Goneid, AUC
14
A General Solution
The secondary recurrence is solved as a first order linear
recurrence. For T(1) i.e. F(0) given, we have as shown before:
m
F( m )  F( 0 )  a 
i 1
m1
m

f ( c )  a  f ( cm )
i
i 1
j i 1
m
 a F ( 0 )   a mi f ( c i )
m
i 1
m1
 a F ( 0 )   a i f ( c mi )
m
i 0
m1
HenceT ( n )  T ( 1 )a   a i f ( n / c i ) for T(1) given.
m
i 0
Also T ( n )  T ( c )a
m1
m2
  a i f ( n / c i ) for T(c) given.
i 0
Prof. Amr Goneid, AUC
15
Very Common: f(n) = b nx
x
n
n
f ( n )  bn x hence f ( i )  b ix
c
c
n
a i
i
x
and a f ( i )  bn ( x )
c
c
The solutionsbecome :
m1
a i
T ( n )  a T ( 1 )  bn  ( x )
for T(1) given.
c
i 0
m2
a i
m 1
x
T ( n )  a T ( c )  bn  ( x ) for T(c) given.
c
i 0
Now,Two cases arise: a  c x and a  c x
m
x
Prof. Amr Goneid, AUC
16
Case(a): f(n) = b nx, a = cx
For T(1) gi ve n:
T (n)  a mT (1)  bm nx  a mT (1)  bnx logc n
For T(c) gi ve n:
T (n)  a m 1T (c)  bnx (logc n  1)
Prof. Amr Goneid, AUC
17
Summary
C om m onD & Q Re cu rre n ce:
T (n)  aT ( n / c)  bnx
Th e sol u ti on be
s com e:
m 1
a i
T (n)  a T (1)  bn  ( x )
i 0 c
m
T ( n)  a
x
for T(1) gi ve n .
m2
a i
T (c)  bn  ( x ) for T(c) gi ve n .
i 0 c
m 1
x
S pe ci alC ase a  c x h as th esol u ti on :s
T (n)  a mT (1)  bm nx  a mT (1)  bnx logc n
T (n)  a m 1T (c)  bnx (logc n  1)
For T(1) gi ve n
For T(c) gi ve n
Prof. Amr Goneid, AUC
18
Case(a): f(n) = b nx, a = cx
For example: BinarySearch :
T( n )  T( n / 2 )  2
T (1 )  1
with
Here a  1, b  2 , c  2 , x  0 ,
hence a  c x
For T (1)given and m  log n :
T ( n )  a T ( 1 )  bm n  a T ( 1 )  bn logc n
m
x
m
x
T ( n )  2 log n  1
Prof. Amr Goneid, AUC
19
Other examples for a = cx
Recursive Power Function
The following function computes xn
Pow (x , n)
{
if (n == 0) return 1.0; else if (n == 1) return x;
else if (n mod 2) return Pow (x*x , n / 2) * x;
else return Pow (x*x , n / 2);
}
Show that the number of arithmetic operations performed by this
algorithm in the best case is T(n) = log n
Prof. Amr Goneid, AUC
20
Solution
In the best case n = 2m , so that n is always even and we
obtain the recurrence relation:
T(n) = T(n/2) + 1 , for n > 1 with T(1) = 0
Here, a = 1, b = 1, c = 2, x = 0
with the solution :
T(n) = bnx log n = log n
Prof. Amr Goneid, AUC
21
Other examples for a = cx
Recursive log Function
A recursive function to compute the largest integer less than or equal
to log2 n. (Hint: for n > = 2 , the value of this function for n is one
greater than for n/2 ).
int Log2(int n)
{
if (n == 1) return 0;
else return (1 + Log2(n/2));
}
The number of integer divisions has the recurrence relation:
T(n) = T(n/2) + 1 , for n > 1 with T(1) = b = 0
with the solution :
T(n) = log n = O(log n)
Prof. Amr Goneid, AUC
22
Other examples for a = cx
Find the Number of Binary Digits in the Binary Representation of a
Positive Decimal Integer using a D&Q recursive algorithm.
int BinDig (int n)
{
if (n == 1) return 1;
else return (1 + BinDig(n/2)); }
T(n) = number of arithmetic operations.
T(n) = T(n/2) + 2 for n > 1 with T(1) = 0
a = 1, b = 2, c = 2, x = 0
T(n) = 2 log n = O(log n)
Prof. Amr Goneid, AUC
23
Other examples for a = cx
Other Recurrences with a = cx
 T(n) = 4 T(n/2) + n2 for n > 1 with T(2) = 1
Hence, T(n) = n2(log n – ¾) = O(n2 log n)
 T(n) = 9T(n/3) + n2
for n > 1 with T(1) = 0
Hence, T(n) = n2 log3 n
 T(n) = 16 T(n/4) + n2
for n > 1 with T(1) = 0
Hence, T(n) = n2 log4 n
Prof. Amr Goneid, AUC
24
Case(b): f(n) = b nx, a  cx
Example: a Fractal Algorithm
The following function draws recursive squares (called a fractal star).
The drawing primitive is Box (x , y , n) which draws a square of side (n)
pixels centered at (x,y) :
void STAR( int x, int y, int n)
{
if (n > 1) { STAR(x-n , y+n , n/2); STAR(x+n , y+n , n/2);
STAR(x-n , y-n , n/2); STAR(x+n , y-n , n/2);
Box(x , y , n); }
}
Prof. Amr Goneid, AUC
25
Fractal Algorithm
The recurrence relation is:
T(n) = 4 T(n/2) + 4 n for n > 1 with T(1) = 0
Here, a = 4, b = 4, c = 2, x =1 so that a  cx
The general solution is:
m1
a i
T ( n )  a T ( 1 )  bn  ( x )
c
i 0
m
x
for T(1) given.
m1
Hence,T ( n )  4n 2 i  4n( 2 m  1 )  4n( n  1 )  O( n 2 )
i 0
Prof. Amr Goneid, AUC
26
Example: MaxMin
Straightforward Algorithm
// Return max and min values in locations s to e in array a[ ]
MaxMin (a, s , e , mx, mn)
{ mx = mn = as;
for i = s+1 to e
{
if (ai > mx) mx = ai ;
if (ai < mn) mn = ai ;
}
}
Invoke as MaxMin(a , 1 , n , mx , mn) ;
Let T(n) be the number of comparisons of array elements.
Hence,
T(n) = 2 ( n – 1) independent of the data in a[ ].
Prof. Amr Goneid, AUC
27
MaxMin D&Q Algorithm
MaxMin (a, s , e , mx , mn)
{
if (s  e-1)
// one or two elements
if (as > ae) {mx = as; mn = ae;} else {mx = ae; mn = as;}
else {
m = (s+e)/2;
MaxMin (a,s,m,mx,mn);
Maxmin (a,m+1,e,mx1,mn1);
mx = max(mx,mx1);
mn = min(mn,mn1);
}
}
// The functions max/min(a,b) find the larger/smaller of (a,b). Each will
cost one comparison.
Prof. Amr Goneid, AUC
28
MaxMin Analysis
The recurrence relation is:
T(n) = 2 T(n/2) + 2 for n > 2 with T(2) = 1
Here, a = 2, b = 2, c = 2, x =0 so that a  cx
The general solution is:
m 2
T ( n )  a T ( 2 )  2 2
m1
i
for T(2) given.
i 0
Hence,T ( n )  0.5n  2( 2
m1
 1 )  1.5n  2
This is only 75% of the cost of the straightforward
algorithm.
Prof. Amr Goneid, AUC
29
Other Examples with a  cx
Copy Binary Tree Algorithm
The following function receives a pointer (t) to a binary tree and
returns a pointer to an exact copy of the tree :
treeNode *CopyTree ( treeNode *t )
{ treeNode *p;
if (t)
{
p = new treeNode ;
p->left = CopyTree(t->left) ;
p->right = CopyTree(t->right);
p->info = t->info ; return p ; };
else return NULL ;
}
Prof. Amr Goneid, AUC
30
Other Examples with a  cx
Copy Binary Tree Algorithm
In the worst case, the tree is full and the number of visits is:
T(n) = 2 T(n/2) + 1
for n > 1 with T(1) = 1
Hence
T(n) = 2n – 1 = O(n)
Prof. Amr Goneid, AUC
31
Other Examples with a  cx
Pre-Order Traversal of a Binary Tree
void preorder (tree t)
{
if (t)
{ visit (t); preorder (t -> left); preorder (t -> right); }
}
In the worst case, the tree is full and the number of visits is:
T(n) = 2 T(n/2) + 1 for n > 1 with T(1) = 1
Hence
T(n) = 2n – 1 = O(n)
Prof. Amr Goneid, AUC
32
Other Examples with a  cx
Other Algorithms
 T(n) = T(n/2) + n2
for n > 1 with T(1) = 0
Hence T(n) = (4/3) (n2 – 1)
 T(n) = 2 T(n/2) + n2
for n > 1 with T(1) = 0
Hence T(n) = 2n(n – 1)
 T(n) = 8 T(n/2) + n2
for n > 1 with T(1) = 0
Hence T(n) = n2 (n – 1)
Prof. Amr Goneid, AUC
33
5. MergeSort
(a) Merging
 Definition:
Combine two or more sorted sequences of
data into a single sorted sequence.
 Formal Definition:
The input is two sorted sequences,
A={a1, ..., an} and B={b1, ..., bm}
The output is a single sequence, merge(A,B),
which is a sorted permutation of
{a1, ..., an, b1, ..., bm}.
Prof. Amr Goneid, AUC
34
Merging
merge(A, B) is
A, if B is empty,
B, if A is empty,
{a1}.merge({a2, ..., an}, B) if a1 <= b1, and
{b1}.merge(A, {b2, ..., bm}) otherwise.
The symbol "." stands for concatenation, for
example,
{a1, ..., an}.{b1, ..., bm} = {a1, ..., an, b1, ..., bm}.
Prof. Amr Goneid, AUC
35
Merge Algorithm
array B
L1
q q+1
p
L2
Merge(A,p,q,r)
{
copy Ap..r to Bp..r and set Ap..r to empty
while (neither L1 nor L2 empty)
{
compare first items of L1 & L2
remove smaller of the two from its list
add to end of A
}
concatenate remaining list to end of A
return A
}
Prof. Amr Goneid, AUC
r
36
Example
i
B
2
9
p
A
j
11
20 1
15 17 30
q
r
1
z
Prof. Amr Goneid, AUC
37
Example
i
B
2
j
9
11
1
15 17 30
q
p
A
20 1
r
2
z
Prof. Amr Goneid, AUC
38
Example
i
B
2
9
j
11
1
15 17 30
q
p
A
20 1
2
r
9
z
Prof. Amr Goneid, AUC
39
Example
i
B
2
9
11
1
20 1
15 17 30
q
p
A
j
2
9
r
11
z
Prof. Amr Goneid, AUC
40
Example
i
B
2
9
11
1
20 1
15 17 30
q
p
A
j
2
9
11
r
15
z
Prof. Amr Goneid, AUC
41
Example
i
B
2
9
11
1
20 1
15 17 30
q
p
A
j
2
9
11
r
15 17
z
Prof. Amr Goneid, AUC
42
Example
i
B
2
9
11
1
20 1
15 17 30
q
p
A
j
2
9
11
r
15 17 20
z
Prof. Amr Goneid, AUC
43
Example
i
B
2
9
11
p
A
1
2
9
j
20 1
15 17 30
q
r
11
15 17 20 30
z
Prof. Amr Goneid, AUC
44
Worst Case Analysis
 |L1| = size of L1, |L2| = size of L2
 In the worst case |L1| = |L2| = n/2
 Both lists empty at about same time, so
everything has to be compared.
 Each comparison adds one item to A so
the worst case is
T(n) = |A|-1 = |L1|+|L2|-1 = n-1 = O(n)
comparisons.
Prof. Amr Goneid, AUC
45
(b) MergeSort Methodology
 Invented by Von Neumann in 1945
 Recursive Divide-And-Conquer
Prof. Amr Goneid, AUC
46
MergeSort Methodology
 Divides the sequence into two
subsequences of equal size, sorts the
subsequences and then merges them
into one sorted sequence
 Fast, but uses an extra space
Prof. Amr Goneid, AUC
47
Methodology (continued)
 Divide:
Divide n element sequence into two subsequences
each of n/2 elements
 Conquer:
Sort the two sub-arrays recursively
 Combine:
Merge the two sorted subsequences to produce a
sorted sequence of n elements
Prof. Amr Goneid, AUC
48
Algorithm
MergeSort (A, p, r)
// Mergesort array A[ ] locations p..r
{
if (p < r)
// if there are 2 or more elements
{
q = (p+r)/2;
// Divide in the middle
// Conquer both
MergeSort (A , p , q);
MergeSort(A , q+1 , r);
Merge(A , p , q , r);
// Combine solutions
}
}
Invoke as MergeSort(a , 1 , n)
Prof. Amr Goneid, AUC
49
Merge Sort Example
8 3 2 9 7 1 5 4
7 1 5 4
8 3 2 9
3
8
9
2
2 9
3 8
5 4
7 1
2 9
8 3
1
7
4
5
4 5
1 7
1 4 5 7
2 3 8 9
1 2 3 4 5 7 8
9
Prof. Amr Goneid, AUC
50
Merge Sort Demos
http://www.cosc.canterbury.ac.nz/people/mukundan/dsal/MSort.html
http://coderaptors.com/?MergeSort
Prof. Amr Goneid, AUC
51
Worst case Analysis of MergeSort
Merge will use n-1 comparisons operations so that:
T (n)  2T (n / 2)  n  1 with T (1)  0
T (n)  2{2T (n / 4)  (n / 2)  1}  (n  1)
 2 2 T (n / 4)  (n  2)  (n  1)
Successive substitution gives for n  2 m
m 1
m 1
i 0
i 0
T (n)  2 m T (1)   (n  2i )  nm   2i
 nm  (2 m  1)  nm  n  1
Hence T (n)  n log2 n  n  1  O(n log n)
Prof. Amr Goneid, AUC
52
Performance of MergeSort
 MergeSort divides the array to two halves at




each level. So, the number of levels is O(log n)
At each level, merge will cost O(n)
Therefore, the complexity of MergeSort is
O(n log n)
In-Place Sort
No (uses extra array)
Stable Algorithm
Yes
 This technique is satisfactory for large data sets
Prof. Amr Goneid, AUC
53
6. QuickSort
(a) Methodology
 Invented by Sir Tony Hoare in 1962
 Recursive Divide-And-Conquer algorithm
 Partitions array around a Pivot , then sorts parts
independently
 Fast, in-place sorting
Prof. Amr Goneid, AUC
54
(a) Methodology
p
q
r
pivot
Left
Right
Divide:
Array a[p..r] is rearranged into a sub-array a[p..q-1], a pivot a[q] and a
second sub-array a[q+1..r] such that each element of the first is <= pivot
and each element of the second is >= pivot ( the pivot index (q) is
computed as part of this process)
Prof. Amr Goneid, AUC
55
Methodology (continued)
 Conquer:
Sort the two sub-arrays recursively
 Combine:
The sub-arrays are already sorted in place, i.e.,
a[p..r] is sorted.
Prof. Amr Goneid, AUC
56
(b) Algorithm
QuickSort (a, p , r)
// Sorts the elements a[p],..., a[r] which reside in
// the global array a[1:n] into ascending order;
// a[n+1] is considered to be defined and must be >= all the
// elements in a[1:n].
{
if (p < r ) {
// If there are more than one element
q = partition (a , p , r+1); // partition around Pivot
// q is the position of the pivot
QuickSort (a , p ,q-1); // Sort left sub-array
QuickSort (a , q+1 , r); // Sort Right sub-array
}
}
Prof. Amr Goneid, AUC
57
Partitioning
int partition(a, p, r)
{
pivot = ap;
// pivot is initially the first element
i = p; j = r;
do
{
do {i++; } while (ai < pivot);
do {j--; } while (aj > pivot);
if(i < j) swap (ai , aj);
} while (i < j);
ap = aj; aj = pivot;
return (j);
// j is the final location of the pivot
}
Prof. Amr Goneid, AUC
58
Example
5 3 1 9 8 2 4 7
2 3 1 4 5 8 9 7
1 2 3 4 5 7 8 9
1 2 3 4 5 7 8 9
1 2 3 4 5 7 8 9
1 2 3 4 5 7 8 9
Prof. Amr Goneid, AUC
59
Quick Sort Demo
http://www.cosc.canterbury.ac.nz/people/mukundan/dsal/QSort.html
http://coderaptors.com/?QuickSort
Prof. Amr Goneid, AUC
60
(c)Analysis of QuickSort
Partitioning will cost (n) comparisons. Sorting costs T(i) for left
sub-array + pivot, T(n-i) for right sub-array. Hence:
T(n) = n + T(i) + T(n-i) for n > 1 with T(1) = 0
partition cost = n
i elements
n-i elements
Prof. Amr Goneid, AUC
61
Best Case
Best Case when pivot has the middle value. In this case, the
array is partitioned into two almost equal halves.
partition cost = n
n/2 elements
n/2 elements
T(n) = 2T(n/2) + n for n > 1 with T(1) = 0
hence, T(n) = O(n log n)
Prof. Amr Goneid, AUC
62
Worst Case
Worst Case when pivot is the smallest element so that i = 1
T(n) = T(n-1) + n for n > 1
partition cost = n
with T(1) = 0
1
n-1 elements
Hence, T(n) = n(n+1)/2 -1= O(n2)
In this case, Quicksort is not really quick!
Prof. Amr Goneid, AUC
63
Summary of
Performance of QuickSort
 Partitioning will cost O(n) at each level
 Best Case: Pivot has the middle value
Quicksort divides the array to two equal halves at each
level. So, the number of levels is O(log n)
Therefore, the complexity of QuickSort is
O(n log n)
 Worst Case: Pivot is the minimum (maximum) value
The number of levels is O(n)
Therefore, the complexity of QuickSort is
O(n2)
 In-Place Sort
Yes
 Stable Algorithm
No
 This technique is satisfactory for large data sets
Prof. Amr Goneid, AUC
64
Median-of-Three Partitioning
If we take the elements in the first, last
and middle locations in the array and
then take the element middle in value
between these three, this will be the
median of three. In this case, there is
always at least one element below and
one element above. The worst case is
therefore unlikely
Prof. Amr Goneid, AUC
65
A Better Choice: Random Pivot
 A better solution that avoids the worst case of
O(n2) is to use a randomizer to select the
pivot element.
 Pick a random element from the sub-array
(p..r) as the partition element.
 Do this only if (r-p) > 5 to avoid cost of the
randomizer
Prof. Amr Goneid, AUC
66
A Better Choice: Random Pivot
void RQuickSort (a , p , r)
{
if (p < r ) {
if ((r-p) > 5) { int m = rand()%(r-p+1) + p;
swap (ap , am); }
q = partition(a , p , r+1);
RQuickSort(a , p ,q-1);
RQuickSort(a , q+1 , r);
}
}
Prof. Amr Goneid, AUC
67
Average Case Analysis
If the pivot is randomly chosen using a random
number generator, we have (n-1) equally probable
configurations:
{1,n-1},{2,n-2},..., {i, n-i},..{n-2,2},{n-1,1}
The probability of each configuration is 1/(n-1)
We obtain an average case cost given by:
1 n1
T ( n) 
{T (i)  T (n  i)  n}

n  1 i 1
Prof. Amr Goneid, AUC
68
Average Case Analysis
T o solve thisequat ion,we recall that:
n -1
n 1
i 1
i 1
 T (n  i)   T (i)
2 n 1
Hence T (n) 
T (i )  n

n  1 i 1
n 1
And (n  1)T (n)  2 T (i )  n(n  1)
i 1
n2
Also (n  2)T (n  1)  2 T (i )  (n  1)(n  2)
i 1
Subtract 2nd from1st we get
T (n) T (n  1) 2


n
n 1
n
Prof. Amr Goneid, AUC
69
Average Case Analysis
T hiscan be writ t enas :
F ( n)  F (n  1)  (2 / n)
wit h t hesolut ion:
n
1
F ( n)  F (1)  2
i2 i
Recall t hatF(n)  T (n)/nand F(1)  T (1) 0
n
1
Also from usefull equat ions,  ln n
i2 i
Hence T ( n)  2n ln n  2n log n / log e  1.386n log n
So, t heaveragecase is st ill T (n)  O (n log n)
Prof. Amr Goneid, AUC
70
See Internet Animation Sites:
For Example:
 http://www.cs.ubc.ca/spider/harrison/Java/sorting-
demo.html
 http://coderaptors.com/?All_sorting_algorithms
Prof. Amr Goneid, AUC
71
7. An Application of Quicksort:
Convex Hull
The Problem:
Given a set S of n points in 2-dimensional space, find
the smallest convex polygon containing all the points of
S.
Prof. Amr Goneid, AUC
72
Convex Hull
 The convex hull of a set of points is an elementary
problem in computational geometry.
 The hull gives a rough idea of the shape or extent
of a data set.
 It is an important step for various geometric
algorithms. For example, finding the diameter of a
set of points.
 A famous algorithm
is the
Graham Scan
Prof. Amr Goneid, AUC
73
Graham Scan
 Select lowest-y point (p)
 Compute angle i between
y
lines p-pi and the x-axis
 Sort points according to 
 Scan through the sorted set
starting at p:




3
2
1
p
select 3 successive points
if they form a left turn, move one point ahead
if not, delete the 2nd point and move back
repeat until we reach point p again
Prof. Amr Goneid, AUC
x
74
Graham Scan
 Triplet = (p,1,2) Left, move to (1)
y
3
2
1
p
x
Prof. Amr Goneid, AUC
75
Graham Scan
Triple = (1,2,3)
y
Right, Delete (2), back to (p)
3
2
1
p
x
Prof. Amr Goneid, AUC
76
Graham Scan
 Triplet = (p,1,3) left, Move to (1)
y
3
2
1
p
x
Prof. Amr Goneid, AUC
77
Graham Scan
 Triplet = (1,3,4) left, Move to (3)
y
3
2
1
p
x
Prof. Amr Goneid, AUC
78
Graham Scan
 Triplet = (3,4,5) left, move to (4)
y
3
2
1
p
x
Prof. Amr Goneid, AUC
79
Graham Scan
 Triplet = (4,5,6) Right, Delete (5), back to (3)
y
3
2
1
p
x
Prof. Amr Goneid, AUC
80
Graham Scan
 Triplet = (3,4,6) left, move to (4)
y
3
2
1
p
x
Prof. Amr Goneid, AUC
81
Graham Scan
 Triplet = (4,6,7) left, Move to (6)
y
3
2
1
p
x
Prof. Amr Goneid, AUC
82
Graham Scan
 Triplet = (6,7,p) left, Move to (7), 3rd point is (p)
y
3
2
1
p
x
Prof. Amr Goneid, AUC
83
Graham Scan
 Stop, Final Hull
y
3
2
1
p
x
Prof. Amr Goneid, AUC
84
Graham Scan
 After the scan, all points remaining will lie on the hull
 To distinguish a left turn from a right turn, we compute the
signed area of the triangle formed by the point triplet:
signed area = (1/2) det (A)
 det (A) = the determinant of the matrix:
 x1 x2 x3 


A   y1 y2 y3 
1 1 1


 If the area is (+)ve, we have a left turn, if (-)ve , we have a
right turn. If zero, the 3 points are co-linear.
Prof. Amr Goneid, AUC
85
Graham Scan Algorithm
void GScan (pointList S)
{
// p is the start point in S
//(p1 , p2 , p3) is a point triplet
// A is the signed area of the triangle
// formed by the point triplet
set p1 = p;
do {
p2 = succ(p1);
if (p2 is not the last point) then p3 = succ(p2);
else p3 = p;
Prof. Amr Goneid, AUC
86
Graham Scan Algorithm
A = Area(p1 , p2 , p3); // signed area
// if p1,p2,p3 form a left turn
// move one point ahead;
// if not, delete p2 and move back.
if (A >= 0.0) p1 = succ(p1);
// left turn
else { // right turn
delete p2;
p1 = pred(p1);
}
} while (!((p3 == p) && (A >= 0.0)));
}
Prof. Amr Goneid, AUC
87
Graham Scan Algorithm
 An implementation of the Graham scan
algorithm is given in the Book, p. 187
 It uses a doubly linked list to represent the
sorted point list.
 The scan algorithm runs in O(n) time.
 Since sorting takes O(n log n) time, the total
time of Graham scan algorithm is
O(n log n)
Prof. Amr Goneid, AUC
88
Graham Scan Demo
 http://www.cosc.canterbury.ac.nz/pe
ople/mukundan/cgeo/Gscan.html
 http://www.cs.princeton.edu/courses/archive/fall0
8/cos226/demo/ah/GrahamScan.html
Prof. Amr Goneid, AUC
89
8. Selection by Partitioning
The Selection Problem
Let A be an array of n distinct elements. Find
the kth smallest element.
Some Applications
Finding nearest neighbor
Finding the Median
Partition around median for closest pair problem
Traveling salesman problem
Prof. Amr Goneid, AUC
90
Selection by Partitioning
Algorithms: Several algorithms exist.
 Sort array (e.g. by Mergesort), kth smallest element
will be located at kth location. Complexity is
O(n logn)
 Insert all elements in a minimum Heap, then remove
k elements from Heap. Last removed is the kth
smallest. Complexity is O(n logn)
 A more efficient algorithm uses partitioning around a
pivot like Quicksort. It is a Divide and Conquer
algorithm with an average cost of O(n).
Prof. Amr Goneid, AUC
91
Selection by Partitioning
Let A be an array of n distinct elements. Find the kth smallest element from
location p through location r.
Type select (Type A[ ], p, r, k)
{
if (p == r) return A[p];
else
{
//Choose a pivot (v) from A and partition A around (v)
// Let A1, A2 be the elements of A which are respectively <,> v
q = partition (A, p, r+1); // new pivot location
// A1 = elements from p to q-1, A2 = elements from q+1 to r
if (k = =q) return A[q];
else if (k < q) return select (A, p, q-1, k);
else return select (A, q+1, r, k − q);
}
}
invoke as select (A, 1, n, k)
Prof. Amr Goneid, AUC
92
Selection by Partitioning
1
j
A1
q
j+2
n
v
cost =T(j)
when
k <= j
i.e.
j = k .. n-1
A2
cost =T(n-j-1)
when
k > j+1
i.e.
j = 0..k-2
partition
cost = n
Prof. Amr Goneid, AUC
93
Selection by Partitioning
Analysis:







The cost of partitioning is (n)
Let |A1| = j and |A2| = n-j-1
The pivot position is q = j+1 with j = 0..n-1
We call select on A1 when k <= j, i.e. in the cases of j = k...n-1
We call select on A2 when k > j+1, i.e. in the cases of j = 0 .. k-2
Each case happens with equal probability (1/n)
Hence the average cost will be:
k 2
1 n1
T (n)  { T ( j )   T (n  j  1)}  n, n  2
n j k
j 0
T (1)  0
Prof. Amr Goneid, AUC
94
Average Case Analysis
T o solve thisequation,we recall that:
k -2
n 1
j 0
j  n  k 1
 T (n  j  1)   T ( j ) increaseswith k
n 1
 T ( j ) decreases with k
j k
T hemaximumis when k  n / 2. Multiplyboth sides by n :
n 1
Hence nT (n)  2  T ( j )  n 2
j n / 2
n2
And (n  1)T (n  1)  2  T ( j )  (n  1) 2
j n / 2
Subtract 2nd from1st we get T (n)  T (n  1)  2 with T (1)  0
T hisrecurrencehas thesolutionT (n)  2(n  1)  O(n)
Prof. Amr Goneid, AUC
95
Selection using a BST
Type select (Type A[ ], int n, int k)
{
Type Amin;
binaryTree<Type, int> BST;
for (int i = 0; i<n; i++) BST.insert(A[i],0);
for (i = 1; i < k; i++) { Amin = BST.min(); BST.remove(Amin);}
return BST.min();
}
invoke as select (A, n, k)
Analysis
Insert n elements
O(n log n)
Find and delete (k-1) minima
O((k-1) log n)
Find kth min
O(log n)
Total
O((k+n) log n)
Prof. Amr Goneid, AUC
96
Other Selection Algorithms
Explore other selection algorithms at:
http://en.wikipedia.org/wiki/Selection_algorithm
Prof. Amr Goneid, AUC
97
9. Closest Pair Problem
The Closest Pair Problem in 1-D
Given a set S = {p1, p2, ..., pN} of N points on a line,
find the two points of S whose distance is smallest.
Prof. Amr Goneid, AUC
98
Closest Pair Problem
Simple solution
Dmin= 
for each pi in S
for each pj in S and (i != j)
if (Dij < Dmin) { Dmin= Dij ; K = i; L = j;}
return K , L, Dmin
Time Complexity = O(N2) ,
Space Complexity = O(N)
Can we do better?
Prof. Amr Goneid, AUC
99
Closest Pair Problem
Solution with Sorting
 Sort the points according to their distance from the
origin.
 After the points are sorted, the closest pair will lie
next to each other somewhere in sorted order. Thus
a linear-time scan through them completes the job.
 Sorting will cost O(N log N) and the scan will cost
O(N) so that the total cost is O(N log N) + O(N) =
O(N log N).
 Obviously, the use of sorting is much more efficient
than the first method.
Prof. Amr Goneid, AUC
100
Closest Pair Problem
D&Q Algorithm in 1-D
 Partition S, a set of points on a line, into two
sets S1 and S2 at some point m such that for
every point p  S1 and q  S2, p < q. Point m
could be the median
 Solve Closest Pair recursively on S1 and S2
 Let {p1, p2} be the closest pair in S1 with 1 =
|p2 - p1| and {q1, q2} the closest pair in S2
with 2 =|q2 - q1|
Prof. Amr Goneid, AUC
101
Closest Pair Problem
 Let  be the smallest distance
found so far:
= min (1, 2)
 The closest pair in S is either {p1,
p2} or {q1, q2} or some {p3, q3}
with p3  S1 and q3  S2.
Which one?
Prof. Amr Goneid, AUC
102
Closest Pair Problem
Let  be the smallest separation found so
far:
 = min(|p2 - p1|; |q2 - q1|)
Prof. Amr Goneid, AUC
103
Closest Pair Problem



How many points of S1 or S2 can lie within
 of m?
Since  is the distance between the closest
pair in either S1 or S2, there can only be at
most 1 point in each side.
Therefore, the number of distance
computations required to check for a closest
pair {p3, q3} with p3  S1 and q3  S2 is
O(1).
Prof. Amr Goneid, AUC
104
Closest Pair Problem:
D&Q Algorithm for 1-D
Closest-Pair (S)
If |S| = 1,  =  else If |S| = 2,  = |p2 - p1|;
else do the following steps:
1. Let m = median(S).
2. Divide S into S1; S2 at m.
3. 1 = Closest-Pair(S1).
4. 2 = Closest-Pair(S2).
5. 12 is minimum distance across the cut
= |p3 – q3|
6. Return  = min(1 , 2 ,,12).
Prof. Amr Goneid, AUC
105
Closest Pair Problem:
D&Q Algorithm for 1-D
Analysis:

Finding the median point (selection problem) is O(N)

Partitioning around the median is O(N)

Computing  in each recursive call is O(1)

Hence, T(N) = 2T(N/2)+O(N) so that
T(N) = O(N log N)
Source Material for 1-D and 2-D cases:
http://www.eecs.tufts.edu/~tklein/closestpair.htm
http://www.cs.mcgill.ca/~cs251/ClosestPair/
http://www.cs.ucsb.edu/~suri/cs235/
http://www.cs.ucf.edu/courses/cot5520/lectures.html
Animation
http://www.cs.mcgill.ca/~cs251/ClosestPair/ClosestPairApplet/ClosestPair
Applet.html
Prof. Amr Goneid, AUC
106