Transcript Elementary Sorting Algorithms

Analysis & Design of
Algorithms
(CSCE 321)
Prof. Amr Goneid
Department of Computer Science, AUC
Part 4. Brute Force Algorithms
Prof. Amr Goneid, AUC
1
Brute Force Algorithms
Prof. Amr Goneid, AUC
2
Brute Force Algorithms


Brute Force Algorithms
Elementary Sorting Algorithms
 Selection Sort
 Bubble Sort
 Insertion Sort




Brute Force Sequential Search
Brute Force String Matching
Closest Pair Problem
Exhaustive Search
 Traveling Salesman Problem
 0/1 Knapsack Problem
 Assignment problem
Prof. Amr Goneid, AUC
3
1. Brute Force Algorithms

Brute Force is a straightforward approach to
solving a problem, usually directly based on
the problem’s statement and definitions of
the concepts involved
 In many cases, Brute Force does not
provide you a very efficient solution
 Brute Force may be enough for moderate
size problems with current computers….
Prof. Amr Goneid, AUC
4
2. Elementary Sorting Algorithms
 Sorting
 Selection Sort
 Bubble Sort
 Insertion Sort
Prof. Amr Goneid, AUC
5
Sorting




The Sorting Problem:
Input: A sequence of keys {a1 , a2 , …, an}
output: A permutation (re-ordering) of the input,
{a’1 , a’2 , …, a’n} such that a’1 ≤ a’2 ≤ …≤ a’n
Sorting is the most fundamental algorithmic
problem in computer science
Sorting can efficiently solve many problems
Different sorting algorithms have been
developed, each of which rests on a particular
idea.
Prof. Amr Goneid, AUC
6
Some Applications of Sorting



Efficient Searching
Ordering a set so as to permit efficient binary
search.
Uniqueness Testing
Test if the elements of a given collection of
items are all distinct.
Deleting Duplicates
Remove all but one copy of any repeated
elements in a collection.
Prof. Amr Goneid, AUC
7
Some Applications of Sorting




Closest Pair
Given a set of n numbers, find the pair of numbers that
have the smallest difference between them
Median/Selection
Find the kth largest item in set S. Can be used to find the
median of a collection.
Frequency Counting
Find the most frequently occurring element in S, i.e., the
mode.
Prioritizing Events
Sorting the items according to the deadline date.
Prof. Amr Goneid, AUC
8
Types of Sorts

In-Place Sort:
•
An in-place sort of an array occurs within the
array and uses no external storage or other
arrays
•
In-place sorts are more efficient in space
utilization

Stable Sorts:
•
A sort is stable if it preserves the ordering of
elements with the same key.
•
i.e. If elements e1 and e2 have the same key,
and e1 appears earlier than e2 before sorting,
then e1 is located before e2 after sorting.
Prof. Amr Goneid, AUC
9
Sorting

What will count towards T(n):

Comparisons of array elements
 Swaps or shifts of array elements

We compare sorting methods on
the bases of:
1. The time complexity of the algorithm
2. If the algorithm is In-Place and Stable
or not
Prof. Amr Goneid, AUC
10
Sorting

We examine here 3 elementary sorting
algorithms:
 Selection Sort
 Bubble Sort
 Insertion Sort
These have a complexity of O(n2)
Prof. Amr Goneid, AUC
11
(a) Selection Sort
 The general idea of the selection sort is that for
each slot, find the element that belongs there
 Assume elements to be in locations 0..n-1
 Let (i) be the start of a sub-array of at least 2
elements, i.e. i = 0 .. n-2
 for each i = 0 .. n-2
Find smallest element in sub-array a[i..n-1]
Swap that element with that at the start of the
sub-array (i.e. with a[i]).
Prof. Amr Goneid, AUC
12
How it works
4
3
1
6
2
5
1
3
4
6
2
5
1
2
4
6
3
5
1
2
3
6
4
5
1
2
3
4
6
5
1
2
3
4
5
6
Prof. Amr Goneid, AUC
13
Demos
http://www.cosc.canterbury.ac.nz/people/mukundan/dsal/SSort.html
http://coderaptors.com/?SelectionSort
Prof. Amr Goneid, AUC
14
Selection Sort Algorithm
First, let us compute the cost of finding the location of
the minimum element in a subarray a[s..e ]
minimum (a,s,e)
e
{
m=s;

for j = s+1 to e
j  s 1
if (a[j]  a[m]) m = j ;
return m;
}
1comp
Hence, T(s,e) = e – s
For an array of size (n) it will cost T(n) = n-1 comparisons
Prof. Amr Goneid, AUC
15
Selection Sort Algorithm
SelectSort (a[0..n-1 ])
n 2

{
i 0
for i = 0 to n-2
{
m = minimum (a , i , n-1) ;
swap (a[i] , a[m]);
}
1swap
}
Prof. Amr Goneid, AUC
(n 1  i)comp
16
Analysis of Selection Sort
n2
Tswap (n)  1swap  (n  1)
i 0
n2
Tcomp (n)   n  1  i comp
i 0
n(n  1)

2
T (n)  0.5n  0.5n  1  (n )
2
2
Exact algorithm
Prof. Amr Goneid, AUC
17
Performance of Selection Sort
 The complexity of the selection sort is
 (n2),
independent of the initial data ordering.
 In-Place Sort
Yes
 Stable Algorithm
No
 This technique is satisfactory for small jobs, not
efficient enough for large amounts of data.
Prof. Amr Goneid, AUC
18
(b) Bubble Sort
 The general idea is to compare adjacent elements
and swap if necessary
 Assume elements to be in locations 0..n-1
 Let (i) be the index of the last element in a subarray of at least 2 elements, i.e. i = n-1 .. 1
Prof. Amr Goneid, AUC
19
Bubble Sort Algorithm
 for each i = n-1…1
Compare adjacent elements aj and aj+1 , j = 0..i-1 and
swap them if aj > aj+1
This will bubble the largest element in the sub-array
a[0..i] to location (i).
Prof. Amr Goneid, AUC
20
How it works
4
3
1
2
3
4
1
2
3
1
4
2
3
1
2
4
1
3
2
4
1
2
3
4
Prof. Amr Goneid, AUC
21
Demos
http://www.cosc.canterbury.ac.nz/people/mukundan/dsal/BSort.html
http://coderaptors.com/?BubbleSort
Prof. Amr Goneid, AUC
22
Bubble Sort Algorithm
1
BubbleSort (a[ 0..n-1])

i  n 1
{
for i = n-1 down to 1
i 1

for j = 0 to i-1
j0
if (a[j] > a[j+1] ) swap (a[j] , a[j+1]);
}
1comp
1swap
Prof. Amr Goneid, AUC
23
Analysis of Bubble Sort
Worst case when the array is inversely sorted:
n(n  1)
Tcomp (n)   1comp    i 
2
i  n 1 j  0
i 1
1
i 1
n 1
n(n  1)
Sim ilarlyTswap 
2
Hence T (n)  n 2  n  O(n 2 )
Prof. Amr Goneid, AUC
24
Bubble Sort (a variant)
void bubbleSort (itemType a[ ], int n)
{ int i , j; bool swapped;
for (i = n-1; i >= 1; i--)
{ swapped = false;
for (j = 0; j < i; j++ )
{ if (a[j] > a[j+1] )
{ swap(a[j] , a[j+1]); swapped = true; }
}
if (!swapped) return;
}
}
Prof. Amr Goneid, AUC
25
Performance of Bubble Sort
 The worst case complexity of the bubble sort is




when the array is inversely sorted: O(n2).
The variant has a best case when the array is
already sorted in ascending order: Ω(n). Outer
loop works for only i = n-1 and there will be no
swaps.
In-Place Sort
Yes
Stable Algorithm
Yes
For small jobs, not efficient enough for large
amounts of data.
Prof. Amr Goneid, AUC
26
(c) Insertion Sort
 The general idea of the insertion sort is that
for each element, find the slot where it
belongs.
 Performs successive scans through the data.
When an element is out of sequence (less
than its predecessor), it is pulled out and
then inserted where it should belong.
 Array elements have to be shifted to the right
to make space for the insertion.
Prof. Amr Goneid, AUC
27
How it works
2
7
1
5
8
6
3
4
1
2
7
5
8
6
3
4
1
2
5
7
8
6
3
4
1
2
5
6
7
8
3
4
1
2
3
5
6
7
8
4
1
2
3
4
5
6
7
8
Prof. Amr Goneid, AUC
28
Demos
http://www.cosc.canterbury.ac.nz/people/mukundan/dsal/ISort.html
http://coderaptors.com/?InsertionSort
Prof. Amr Goneid, AUC
29
Insertion Sort Algorithm
InsertSort (a[0..n-1 ])
n 1
{ for i =1 to n-1

i 1
{
j = pointer to element ai;
1comp
v = copy of ai;
while( j > 0 && aj-1 > v)
{ move data right: aj ← aj-1
move pointer left: j-- }
Insert v at last (j) location: aj ← v;
}
}
Prof. Amr Goneid, AUC
Best1 tim e
Worst i tim es
1shift
30
Analysis of Insertion Sort
Worst case when the array is inversely sorted, while loop
iterates (i) times:
n 1
n 1
i 1
i 1
Tcomp ( n)   i 1comp    i 
1 2
( n  n)
2
1 2
( n  n)
2
HenceT (n)  ( n 2  n)  O( n 2 )
Sim ilarly Tshift (n) 
Best case when the array is already sorted in ascending
order, while loop works zero times
n 1
T (n)  1comp  (n  1)  (n)
i 1
Prof. Amr Goneid, AUC
31
Performance of Insertion Sort
 The worst case complexity of the insertion sort




is when the array is inversely sorted: O(n2).
Best case when the array is already sorted in
ascending order: Ω(n). While loop will not
execute and there will be no data movement.
In-Place Sort
Yes
Stable Algorithm
Yes
For modest jobs, not efficient enough for large
amounts of data.
Prof. Amr Goneid, AUC
32
3. Brute Force Sequential Search
Find whether a search key is present in an array
// Search key K in A[0..n-1]
ALGORIT HMSequentialSearch( A[0..n], K )
A[n]  K
i0
while A[i ]  K do
Worst case:
Key not found.
T(n) = O(n)
comparisons
i  i 1
if i  n returni
else return-1
Prof. Amr Goneid, AUC
33
4. Brute Force String Matching





Pattern: a string of m characters to search for
Text: a (longer) string of n characters to search in
Problem: find a substring in the text that matches
the pattern
Example: Pattern – ‘NOT’,
text – ‘NOBODY_NOTICED_HIM’
Typical Applications
‘find’ function in the text editor, e.g., MS-Word,
Google search
Prof. Amr Goneid, AUC
34
Brute Force String Matching
Brute-force algorithm
Step 1 Align pattern at beginning of text
Step 2 Moving from left to right, compare each character of
pattern to the corresponding character in text until
 all characters are found to match (successful search);
or
 a mismatch is detected
Step 3 While pattern is not found and the text is not yet
exhausted, realign pattern one position to the right and
repeat Step 2
Prof. Amr Goneid, AUC
35
Brute Force String Matching
T[0..n-1]: Text string
P[0..m-1]: Pattern string
ALGORITHM BFStringMatch( T [ 0..n  1 ], P [ 0..m  1 ])
for i  0 to n-m do
j0
while j  m and P [ j ]  T [ i  j ]
j j1
if j  m return i
return -1
Show that the worst case number of comparisons is O(mn)
Prof. Amr Goneid, AUC
36
5. Closest Pair Problem
Problem:
Find the two closest points in a set of n points (in the two
dimensional Cartesian plane).
Brute-force algorithm
 Compute the distance between
every pair of distinct points
 Return the indexes of the points for
which the distance is the smallest.
Prof. Amr Goneid, AUC
37
Closest Pair Problem
n -1
n
n -1
n -1
i 1
i 1
T ( n )  cost for sqrt    1   (n - i)   i  n(n - 1)/2  O(n 2 )
i 1 j  i  1
Prof. Amr Goneid, AUC
38
Closest Pair Problem

The Hamming distance between two strings of equal length
is defined as the number of positions at which the
corresponding symbols are different. It is named after Richard
Hamming (1915–1998), a prominent American scientist.
 Suppose that the points in the problem of closest
pairs are strings and hence the distance between
two of such strings is now measured by the
Hamming Distance. Write an algorithm to find the
closest pair of strings in an array of strings and find
the number of comparisons done by this algorithm.
Prof. Amr Goneid, AUC
39
6. Exhaustive Search
 A brute-force approach to combinatorial problem
Generate each and every element of the problem’s domain
 Then compare and select the desirable element that
satisfies the set constraints
 Involve combinatorial objects such as permutations,
combinations, and subsets of a given set
 The time efficiency is usually bad – usually the
complexity grows exponentially with the input size
Three examples
 Traveling salesman problem
 Knapsack problem
 Assignment problem

Prof. Amr Goneid, AUC
40
(a)Traveling Salesman Problem (TSP)
 Given n cities with known distances between each pair, find the
shortest tour that passes through all the cities exactly once
before returning to the starting city
 Alternatively: Find shortest Hamiltonian circuit in a weighted
connected graph (The circuit problem is named after Sir
William Rowan Hamilton)
2
a
b
 Example:
5 3
4
8
c
Prof. Amr Goneid, AUC
7
d
41
Traveling Salesman Problem (TSP)
Hamiltonian Circuit
a→b→c→d→a
a→b→d→c→a
a→c→b→d→a
a→c→d→b→a
a→d→b→c→a
a→d→c→b→a
Cost
2+3+7+5 = 17
2+4+7+8 = 21
8+3+4+5 = 20
8+7+4+2 = 21
5+4+3+8 = 20
5+7+3+2 = 17
Optimal
Optimal
Number of candidate circuits = (n-1)!
Very High complexity O(n!)
Prof. Amr Goneid, AUC
42
(b) 0/1 Knapsack Problem
Given n items:
 weights: w1 w2 … wn
 values:
v1 v2 … vn
 a knapsack of capacity W
Find most valuable subset of the items that fit into the knapsack
Example: Knapsack capacity W=16
item weight
value ($)
1
2
20
2
5
30
3
10
50
4
5
10
Prof. Amr Goneid, AUC
43
0/1 Knapsack Problem
Subset Total weight
{ }
0
{1}
2
{2}
5
{3}
10
{4}
5
{1,2}
7
{1,3}
12
{1,4}
7
{2,3}
15
{2,4}
10
{3,4}
15
{1,2,3}
17
{1,2,4}
12
{1,3,4}
17
{2,3,4}
20
{1,2,3,4}
22
Total value
$00
$20
$30
$50
$10
$50
$70
$30
$80
Optimal
$40
$60
not feasible
$60
not feasible
not feasible
not feasible
Prof. Amr Goneid, AUC
Generates 2n
possible
subsets,
T(n) = O(2n)
44
(c) Assignment Problem by Exhaustive Search
There are n people who need to be assigned to n jobs, one person
per job. The cost of assigning person i to job j is C[i,j]. Find an
assignment that minimizes the total cost.
Person 1
Person 2
Person 3
Person 4
Job 1 Job 2 Job 3 Job 4
9
2
7
8
6
4
3
7
5
8
1
8
7
6
9
4
Algorithmic Plan:
Generate all legitimate assignments, compute their costs, and select the
cheapest one.
How many assignments are there? n!
Pose the problem as one about a cost matrix:
Prof. Amr Goneid, AUC
45
Assignment Problem by Exhaustive Search
9
6
C
5
7
2
4
8
6
7
3
1
9
8
7
8
4
Assignment (col.#s)
1, 2, 3, 4
1, 2, 4, 3
1, 3, 2, 4
1, 3, 4, 2
1, 4, 2, 3
1, 4, 3, 2
T(n) = # of
assignments = O(n!)
Total Cost
9+4+1+4=18
9+4+8+9=30
9+3+8+4=24
9+3+8+6=26
9+7+8+9=33
9+7+1+6=23
etc.
Prof. Amr Goneid, AUC
46