Transcript CS 615: Design & Analysis of Algorithms

CS 615:
Design & Analysis of Algorithms
Chapter 4: Sorting
Mark Allen Weiss: Chapter 7
Page 219-261
Course Content
1.
2.
3.
4.
5.
6.
7.
8.
9.
Introduction, Algorithmic Notation and Flowcharts
(Brassard & Bratley Chp: Chapter 3)
Efficiency of Algorithms (Brassard & Bratley Chp:
Chapter 2)
Basic Data Structures (Brassard & Bratley Chp:
Chapter 5)
Sorting (Weiss Chp: 7)
Searching (Brassard & Bratley Chp: Chapter 9)
Graph Algorithms (Weiss Chp: 9)
Randomized Algorithms (Weiss Chp: 10)
String Searching (Sedgewick Chap. 19)
NP Completeness (Sedgewick Chap. 40)
21 July 2015
CS 615 Design & Analysis of Algorithms
2
Lecture Content
Sorting by Selection
Sorting by Insertion
Insertion Sort
Shellsort
Heapsort
Mergesort
Quicksort
21 July 2015
CS 615 Design & Analysis of Algorithms
3
Sorting by Selection
Algorithm:
procedure select(T[1..n])
for i=1 to n-1 do
minj=i
minx=T[i]
for j=i+1 to n do
if T[j]<minx
then
minj=j
minx=T[j]
T[minj]=T[i]
T[i]=minx
Select the smallest number
within the array
Bring it to the front of the
array
Get the next smallest element
Insert it into the next
location in the array
Until all elements are sorted
When this algorithm is
programmed
Required time to sort the
array
Vary no more than %15
Whatever the initial order of
the elements to be sorted
select(T) is quadratic
21 July 2015
CS 615 Design & Analysis of Algorithms
4
Sorting by Insertion
Algorithm:
Set the first element as the minimum
For all other elements
Check the element if less than current
minimum
If so compare the item with previous
elements in array
Exchange the elements until the correct
place is found
Apply the same operation to all other
elements of the array
When this algorithm is programmed
procedure insert(T[1..n])
for i=2 to n do
x=T[i];
j=i-1
while j>0 and x<T[j]
do
T[j+1]=T[j]
j=j-1
T[j+1]=x
Variation between best and worst case
is very big
Required time depends on the number of
elements in the array
insert(T) is quadratic
21 July 2015
CS 615 Design & Analysis of Algorithms
5
Insertion Sort
One of the simplest algorithms
Consist of N-1 passes
In pass P
The element in position P is saved in Tmp
All larger elements prior to position P
Are moved one spot to the right
InsertionSort
Then Tmp is placed in correct spot
Original
34
8
64
51
32
21
Position Moved
After p=1
8
34
64
51
32
21
1
After p=2
8
34
64
51
32
21
0
After p=3
8
34
51
64
32
21
1
After p=4
8
32
34
51
64
21
3
After p=5
8
21
32
34
51
64
4
21 July 2015
CS 615 Design & Analysis of Algorithms
6
Pseudocode/Insertion Sort
void InsertionSort( ElementType A[ ], int N )
{
int j, P;
ElementType Tmp;
for( P = 1; P < N; P++ )
{
Tmp = A[ P ];
for( j = P; j > 0 && A[ j - 1 ] > Tmp; j-- )
A[ j ] = A[ j - 1 ];
A[ j ] = Tmp;
}
}
21 July 2015
CS 615 Design & Analysis of Algorithms
7
Analysis of Insertion Sort
Because of nested loops
each step will take N iterations
Order of algorithm is O(n2)
If the data is in reverse order:
Order of algorithm is O(n2)
If the data is pre-sorted
Order of algorithm is O(n)
The inner loop always fail immediately
21 July 2015
CS 615 Design & Analysis of Algorithms
8
Shell Sort
Invented by Donald Shell
Was the first algorithm to break the quadratic
time barrier
Uses a sequence, h1,h2,...,ht, called
“increment sequence”
At each sequence:
Compare h-distant element
Exchange them if the first element is greater
than the second one
ShellSort
Original
81
94
11
96
12
35
17
95
28
58
41
75
15
After 5-sort
35
17
11
28
12
41
75
15
96
58
81
94
95
After 3-sort
28
12
11
35
15
41
58
17
94
75
81
96
95
After 2-sort
11
12
15
35
28
17
58
41
81
75
94
96
95
After 1-sort
11
12
15
17
28
35
41
58
75
81
94
95
96
21 July 2015
CS 615 Design & Analysis of Algorithms
9
Pseudocode/Shell Sort
/* START: fig7_4.txt */
void Shellsort( ElementType A[ ], int N )
{
int i, j, Increment;
ElementType Tmp;
}
21 July 2015
for( Increment = N / 2; Increment > 0; Increment /= 2 )
for( i = Increment; i < N; i++ )
{
Tmp = A[ i ];
for( j = i; j >= Increment; j -= Increment )
if( Tmp < A[ j - Increment ] )
A[ j ] = A[ j - Increment ];
else
break;
A[ j ] = Tmp;
}
CS 615 Design & Analysis of Algorithms
10
Analysis of Shellsort
The worst case is O(n2)
Hibbard’s increment
The worst case is O(n3/2)
The average case is O(n5/4)
Not proven !
The performance is quite acceptable
Widely used in practice
A good choice for sorting up large input
21 July 2015
CS 615 Design & Analysis of Algorithms
11
Heapsort
Executes in order of O(NlogN)
The best of sorting techniques
10
Only Shellsort uses Sadgewick’s
increment sequence is better
(max)Heap:
4
5
The key values of children are less than the
parent
The maximum can always be found at the
root
4
(min)Heap:
21 July 2015
CS 615 Design & Analysis of Algorithms
5
7
....
The key values of children are greater than
the parent
The minimum can always be found at the
root
....
Heap order property
7
10
12
Basic Heap Operations
13
Insert
insert
Create a hole in the first
available location
If the new element X does not
violate the heap property
24
21
31
it is done
otherwise slide the hole’s parent
to the new hole
If the new element can be
inserted in the parent’s node 65
26
insert it and job is finished
Else execute the same operations
until the correct place is found
16
32
19
68
h
insert 14
13 21 162431 1968652632 14
1 2 3 4 5 6 7 8 9 10 11
21 July 2015
CS 615 Design & Analysis of Algorithms
13
Basic Heap Operations: DeleteMin
13
Create a hole in the root
Percolate down:
14
Place the smallest child
19
16
21
as the parent
19
68
Place child’s smallest child
in the hole of the child
Continue the operation
65
26
32
31
until no placement remains
13 14 16 19 21 1968652632 31
DeleteMin
21 July 2015
1 2 3 4 5 6 7 8 9 10 11
CS 615 Design & Analysis of Algorithms
14
After Deletion, The New Heap
14
19
16
14 19 162621 1968653132
26
65
21 July 2015
21
31
19
68
1 2 3 4 5 6 7 8 9 10
32
CS 615 Design & Analysis of Algorithms
15
Heap Sorting
Build a heap of N elements
N succesive insert operations
order is
Average: O(N)
Worst Case: O(NlogN)
Perform N DeleteMin operations
Each delete will take order of O(logN)
Record deleted elments in another array
or use the shrinked heap array
put the element in emptied location
The elements will be sorted in decreasing order
Use max(heap)
Start
Build
Heap
DeleteMin
End
to sort the elements in increasing order
The total running time is
O(N) + O(NlogN) = O(NlogN)
21 July 2015
CS 615 Design & Analysis of Algorithms
16
Example: HeapSort
(Max) heap
DeleteMax
97
53
26
59
41
58
53
31
97535926415831
0 1 2 3 4 5 6 7 8 9
21 July 2015
59
26
58
41
31
97
5953582641 3197
0 1 2 3 4 5 6 7 8 9 10
CS 615 Design & Analysis of Algorithms
17
Example Continues
DeleteMax
58
53
26
31
41
59
41
97
58533126415997
0 1 2 3 4 5 6 7 8 9 10
21 July 2015
53
26
31
58
59
97
5341 3126585997
0 1 2 3 4 5 6 7 8 9 10
CS 615 Design & Analysis of Algorithms
18
Example Continues
DeleteMax
41
26
53
26
31
58
59
97
53
41
58
59
97
31264153585997
41263153585997
0 1 2 3 4 5 6 7 8 9 10
21 July 2015
31
0 1 2 3 4 5 6 7 8 9 10
CS 615 Design & Analysis of Algorithms
19
Example: After HeapSort
26
31
53
41
58
59
97
2631 4153585997
0 1 2 3 4 5 6 7 8 9 10
21 July 2015
CS 615 Design & Analysis of Algorithms
20
Merge Sort
Aptr
1 132426
Runs in O(NlogN) worst-case
The fundamental operation:
2 152738
merging two sorted lists
Bptr
Takes
two already sorted input arrays:
A, B
Cptr
One output array:
C
Aptr
Merge
Three counters
Aptr,Bptr, and Cptr
Algorithm
1 132426
2 152738
Take the smaller of A[Aptr] and B[Bptr] Bptr
Put it into C[Cptr]
1
Advance the counters
When any of input lists exhausted
Cptr
Copy the next array to C
This step can be done in one pass
O(N)
21 July 2015
CS 615 Design & Analysis of Algorithms
21
Merge Sort Example
Aptr
1 132426
Aptr
2 152738
Bptr
1 2
Cptr
Aptr
1 132426
2 152738
Bptr
1 2 13
1 132426
1 2 13 1524
Cptr
21 July 2015
1 132426
2 152738
Bptr
Cptr
1 2 13 15
Cptr
Aptr
2 152738
Bptr
Aptr
1 132426
2 152738
Bptr
1 2 13 1524262738
Cptr
CS 615 Design & Analysis of Algorithms
22
Divide and Conquer
Divide the unordered list into two halves
Sort the first half
Sort the second half
Merge two halves
MergeSort
Execute the same operations recursively
until all of the array is sorted
At each step
A temporary C array is needed
At any point there could be
NlogN temporary arrays in memory
If memory is low, not a good algorithm
Allocate dynamic memory, and use input arrays
for temporary arrays
21 July 2015
CS 615 Design & Analysis of Algorithms
23
Example: Full Merge Sort
First Half
First Half
Aptr
241326 1 2 273815
Bptr
Second Half
First Half
Aptr
Aptr
241326 1
Bptr
Second Half
2 273815
Bptr
First Half
1 2 13 1526
First Half
Aptr
Cptr
Aptr
1324 1 26
Bptr
Second Half
Second Half
2 271538
Bptr
Second Half
First Half
Aptr
1 132426 2 152738
Bptr
21 July 2015
1 2 13 1524262738
Second Half
CS 615 Design & Analysis of Algorithms
24
Analysis of Mergesort
The order of total algorithm is
O(NlogN)
Is hardly used in main memory sorts
Needs extra memory to sort two lists
Slows down the algorithm
due to copying done between the lists
Used mostly for external sorting
Using disk spaces to sort very large lists
Recursion adds extra time
Recursive call is sometimes
more than 100 times slower
than the iteration
21 July 2015
CS 615 Design & Analysis of Algorithms
25
Quicksort
Like MergeSort QuickSort is also a
divide and conquer
and recursive algorithm
Average running time
O(NLogN)
QuickSort
Worst-case
O(N2)
Good algorithm in theory
But in practice not easy to code correctly
Algorithm: (For array S)
1.
2.
3.
4.
21 July 2015
If S has only 0 or 1 elements return
Pick any element v in S. This is called the pivot
Partition into S1=S-{v} and S2=S-{v} two subsets
Quicksort S1 and S2.
CS 615 Design & Analysis of Algorithms
26
QuickSort Example
-5 21 0 97 61-312449 18 52
-5 21 0 -3118
24
Pivot
Pivot
-5-31
0
21 18
97 614952
Pivot
49 52
18 21
-31 -5
-31-5 0 18 21
9761
61 97
49526197
-31-5 0 18 21 24495261 97
21 July 2015
CS 615 Design & Analysis of Algorithms
27
Selecting the Pivot
Do not select the first or the second element as
pivot
If the array sorted in reverse order
Pivot provides a poor partition
Alternatives
Median3
Choose the pivot randomly
Random number generation is expensive
Not all systems provide real random numbers
Choose the median
if the array is reverse ordered
Eliminates the bad case for Quicksort case
Hard to calculate
Some other methods are also available
21 July 2015
CS 615 Design & Analysis of Algorithms
28
Analysis of Quicksort
Pivot selection takes a constant time
Worst-Case
O(N2)
Best Case
O(NlogN)
Average Case
O(NlogN)
21 July 2015
CS 615 Design & Analysis of Algorithms
29
End of Chapter 4
Sorting
21 July 2015
CS 615 Design & Analysis of Algorithms
30