Transcript Introduction to Algorithm Analysis - Home

Asymptotic Behavior
Algorithm : Design & Analysis
[2]
In the last class…


Goal of the Course
Mathematical Background





Probability
Summations and Series
Monotonic and Convex Functions
Average and Worst-Case Analysis
Lower Bounds and the Complexity of
Problems
Asymptotic Behavior




Asymptotic growth rate
The Sets ,  and 
Complexity Class
An Example: Searching an Ordered Array



Improved Sequential Search
Binary Search
Binary Search Is Optimal
Fundamentals of Cost

Input

Input size:
Number of items (sorting)
 Total number of bits (integer multiplying)
 Number of vertices+number of edges (graph)



Structure of input
Running time


Number of primitive operations executed
Actual cost of each operation
How to Compare Two Algorithm?

Simplifying the analysis




assumption that the total number of steps is roughly
proportional to the number of basic operations
counted (a constant coefficient)
only the leading term in the formula is considered
constant coefficient is ignored
Asymptotic growth rate

large n vs. smaller n
Relative Growth Rate
Ω(g):functions that grow at
least as fast as g
g
Θ(g):functions that grow at
the same rate as g
Ο(g):functions that grow no
faster as g
The Set “Big Oh”

Definition


A function fΟ(g) if limn→[f(n)/g(n)]=c<


Giving g:N→R+, then Ο(g) is the set of f:N→R+, such
that for some cR+ and some n0N, f(n)cg(n) for all
nn0.
Note: c may be zero. In that case, f(g), “little Oh”
Example: let f(n)=n2, g(n)=nlgn, then:


fΟ(g), since limn→[f(n)/g(n)]= limn→[n2/nlgn]=
limn→[n/lgn]= limn→[1/(1/nln2)]=
gΟ(f), since limn→[g(n)/f(n)]=0
The Sets  and 

Definition


A function f(g) if limn→[f(n)/g(n)]>0


Note: the limit may be infinity
Definition


Giving g:N→R+, then (g) is the set of f:N→R+, such
that for some cR+ and some n0N, f(n)cg(n) for all
nn0.
Giving g:N→R+, then (g) = Ο(g) (g)
A function fΟ(g) if limn→[f(n)/g(n)]=c,0<c<
How Functions Grow
Algorithm
Time function(ms)
1
2
3
4
33n
46n lg n
13n2
3.4n3
Input size(n)
2n
Solution time
10
0.00033 sec.
0.0015
sec.
0.0013 sec.
0.0034 sec.
0.001 sec.
100
0.0033 sec.
0.03 sec.
0.13 sec.
3.4 sec.
41016 yr.
1,000
0.033 sec.
0.45 sec.
13 sec.
0.94 hr.
10,000
0.33 sec.
6.1 sec.
22 min.
39 days
100,000
3.3 sec.
1.3 min.
1.5 days
108 yr.
Time allowed
Maximum solvable input size (approx.)
1 second
30,000
2,000
280
67
20
1 minute
1,800,000
82,000
2,200
260
26
Increasing Computer Speed
Number of steps
performed
on input of size n
f(n)
Maximum feasible
input size
s
Maximum feasible
input size
in t times as much time
snew
lg n
s1
s1 t
n
s2
t s2
n2
s3
t s3
n3
s4
s4+lg t
Sorting a Array of 1 Million Numbers
Using merge sort,
taking time 50nlogn:
100 seconds!
Computer B
10 Mips
Computer A
1000 Mips
Using insertion sort, taking time 2n2:
2000 seconds!
Properties of O,  and 

Transitive property:


Symmetric properties



If fO(g) and gO(h), then fO(h)
fO(g) if and only if g(f)
f(g) if and only if g(f)
Order of sum function

O(f+g)=O(max(f,g))
Complexity Class



Let S be a set of f:NR* under consideration,
define the relation ~ on S as following: f~g iff.
f(g)
then, ~ is an equivalence.
Each set (g) is an equivalence class, called
complexity class.
We usually use the simplest element as possible
as the representative, so, (n), (n2), etc.
Comparison of Often Used Orders


The log function grows more slowly than
any positive power of n
lgn  o(n) for any >0
The power of n grows more slowly than
any exponential function with base greater
than 1
nk  o(cn) for any c>1
(The commonly seen base is 2)
Order of Common Sums
n

d 1
d
n


i


i 1
b
k is the largest term in the sum
k
i
r
)
r
(


r

i a
Area=nf(n)
f(n)
n
 log(i)  (n log n)
f(rn)
i 1
n
d 1
d
log(n))
n
(


)
i
log(
i

i 1
rn
n
(0<r<1)
Area=(1-r)nf(rn)
Manipulation of Big-O
n p  O(n q ) if p  q
O( f (n))  O( g (n))  O(| f (n) |  | g (n) |)
f (n)  O( f (n))
c  O( f (n))  O( f (n)) if c is constant
O(O( f (n)))  O( f (n))
O( f (n))O( g (n))  O( f (n) g (n))  f (n)O( g (n))
ln(1  O( f (n)))  O( f (n)) if lim f (n)  0
n 
eO ( f ( n ))  1  O( f (n)) if f (n)  O(1)
(1  O( f (n)))O ( g ( n ))  1  O( f (n) g (n)) if lim f (n)  0 and f (n) g (n)  O(1)
n 
Roulette Wheel
There are N
slots , some
winning and
some losing
N 1
 n | n
3
3
.
.
.
.
Winner:
2
i
n
.
.
If there are W
winners with
bonus a, and L
losers with
penalty b, then the
average winning
will be:
aW  bL
N
A Special Case with N=1000
Let winning bonus be 5, and losing penalty be 1
5W  L 5W  (1000  W ) 6W  1000
average winnings 


1000
1000
1000
1000
1000
W   n is a winner  
n 1

 k
3
n 1

 n | n   k   n   k | n1  n  1000
3
3
k ,n
 n  ( k  1) 3 n  km 1  n  1000
k ,m,n

 1   m  k

/ k  1  k  10
 1   k 3  km  ( k  1) 3 1  k  10
k ,m
2
, ( k  1) 3
k ,m
9

 1   ( k  1) / k   k
k 1
3
  1   (3k  4)  172
9
2
k 1
Solution Generalized
Let the largest slot number be N, and K 

K 1
 N
3
, then:
W   (3k  4)   K 3  Km  N
k 1

m

(7  3K  1)(K  1)
 2 N 

  m   K ,

2
K

 
m 
N 

3 2 5
2
 K  K 4
 K   1




2
2
K 

5
N  1
    K2  K 3
2
K  2
Reasonable Approximation
 N
3
Now, we have: W=N/K+K2/2+5K/2-3, where K=
1
3
We replaceK by N  (1)
1
3
K  N (1  ( N
2
3

K 2  N (1  ( N
1
3

)
1
3
2
3
))2  N (1  ( N

1
3
2
3
))  N  ( N )
1

N 
1
3
3

N
(
1


(
N
))
 (1)  N 3  ( N 3 )
 K 
So,
1
2
3
1
1
3
2
2
1
1
3
1
1
1
W  N  ( N )  ( N 3  ( N 3 ))  ( N 3 )  (1)
2
2
1
2
3 3
3
 N  ( N 3 )  N 3
2
2
If Only N Large Enough…
2
N
1,000
10,000
100,000
1,000,000
10,000,000
100,000,000
1,000,000,000
3 3
N
2
150.0
696.2
3231.7
15000.0
69623.8
323165.2
1500000.0
W
172
746
3343
15247
70158
324322
1502497
%error
12.791
6.670
3.331
1.620
0.761
0.357
0.166
Searching an Ordered Array

Problem:



Input:
 an array E containing n entries of numeric type sorted in nondecreasing order
 a value K
Output:
 index for which K=E[index], if K is in E, or, -1, if K is not in E
Algorithm:







Int seqSearch(int[] E, int n, int K)
1. Int ans, index;
2. Ans=-1; // Assume failure
3. For (index=0; index<n; index++)
4.
If (K==E[index]) ans=index;//success!
5.
break;
6. return ans
Searching a Sequence (cont.)

For a given K, there are two possibilities


K in E (say, with probability q), then K may be any
one of E[i] (say, with equal probability, that is 1/n)
K not in E (with probability 1-q), then K may be
located in any one of gap(i) (say, with equal
probability, that is 1/(n+1))
gap(0)
E[1] E[2]
gap(i-1)
gap(i-1)
E[i-1] E[i] E[I+1]
E[n]
Improved Sequential Search

Since E is sorted, when an entry larger than K is met, no
more comparison is needed
 Worst-case complexity: n, unchanged
 Average complexity
Roughly n/2
A( n)  qA (n)  (1  q ) A ( n)
succ
fail
n 1
n 1
1
Asucc ( n) 
 (i  1) 
n
2
i 0  

n 1
n
n
 1 
 1 
A fail ( n) 

(i  1)  
n  
n 1
2 n 1
 n 1
i 0 

A( n) 
n  n
 n 
 
 q
 
2  n 1
 n 1
Note: A(n)(n)
Divide and Conquer

If we compare K to every jth entry, we can locate
the small section of E that may contain K.




To locate a section, roughly n/j steps at most
To search in a section, j steps at most
So, the worst-case complexity: (n/j)+j, with j selected
properly, (n/j)+j(n)
However, we can use the same strategy in the
small sections recursively
Choose j = n
Binary Search
int binarySearch(int[] E, int first, int last, int K)
if (last<first)
index=-1;
else
int mid=(first+last)/2
if (K==E[mid])
index=mid;
else if (K<E[mid])
index=binarySearch(E, first, mid-1, K)
else if (K<E[mid])
index=binarySearch(E, mid+1, last, K)
return index;
Worst-case Complexity
of Binary Search



Observation: with each call of the
recursive procedure, only at most half
entries left for further consideration.
At most lg n calls can be made if we
want to keep the range of the section left
not less than 1.
So, the worst-case complexity of binary
search is lg n+1=lg(n+1)
Average Complexity
of Binary Search


Observation:
 for most cases, the number of comparison is
or is very close to that of worst-case
 particularly, if n=2k-1, all failure position need
exactly k comparisons
Assumption:
 all success position are equally likely (1/n)
 n=2k-1
Average Complexity
of Binary Search

Average complexity
 Note: We count the sum of st, which is the
number of inputs for which the algorithm does t
comparisons, and if n=2k-1, st=2t-1
Aq ( n)  qA1 ( n)  (1  q ) A0 ( n)
1 k
(k  1) 2 k  1
 st 
t 1
A1 ( n)   t     t 2


n t 1
n
t 1  n 
( k  1)(n  1)  1
 log n 
 lg( n  1)  1  O

n
 n 
A0  lg( n  1)
k
Aq ( n)  lg( n  1)  q
Decision Tree

An algorithms A that can do no other operations on the array entries
except comparison can be modeled by a decision trees.

A decision tree for A and a given input of size n is a binary tree
whose nodes are labeled with numbers between 0 and n-1
 Root: labeled with the index
4
first compared
7
1
 If the label on a particular node
is i, then the left child is labeled
5
the index next compared if K<E[i], 0
2
the right child the index next
3
compared if K>E[i], and no branch
6
for the case of K=E[i].
8
9
Binary Search Is Optimal

If the number of comparison in the worst case is
p, then the longest path from root to a leaf is p-1,
so there are at most 2p-1 node in the tree.

There are at least n node in the tree.
(We can prove that For all i{0,1,…,n-1}, exist a
node with the label i.)

Conclusion: n 2p-1, that is plg(n+1)
Binary Search Is Optimal


For all i{0,1,…,n-1}, exist a node with the label i.
Proof:
 if otherwise, suppose that i doesn’t appear in
the tree, make up two inputs E1 and E2, with
E1[i]=K, E2[i]=K’, K’>K, for any ji, (0jn-1),
E1[j]=E2[j]. (Remember that both array are
sorted). Since i doesn’t appear in the tree, for
both K and K’, the algorithm behave alike
exactly, and give the same outputs, of which at
least one is wrong, so A is not a right algorithm.
Home Assignment

pp.63 –





1.23
1.27
1.31
1.34
1.45