Transcript CS 332: Algorithms

CS 3343: Analysis of
Algorithms
Lecture 3: Asymptotic Notations,
Analyzing non-recursive algorithms
7/21/2015
1
Outline
• Review of last lecture
• Continue on asymptotic notations
• Analyzing non-recursive algorithms
7/21/2015
2
True or false?
1.
2.
3.
4.
5.
6.
7/21/2015
2n2 + 1 = O(n2)
T (also )
Sqrt(n) = O(log n)
F ()
log n = O(sqrt(n))
T (also o)
n2(1 + sqrt(n)) = O(n2 log n)
F ()
3n2 + sqrt(n) = O(n2)
T (also )
sqrt(n) log n = O(n)
T (also o)
3
Questions
•
If f(n)  O(g(n))
1. compare f(n) and f(n) + g(n)
2. compare g(n) and f(n) + g(n)
3. compare h(n) f(n) and h(n) g(n) where
h(n) > 0.
•
7/21/2015
How about f(n)  Ω (g(n)) and
f(n)  Θ (g(n))?
4
Asymptotic notations
•
•
•
•
•
O: <=
o: <
Ω: >=
ω: >
Θ: =
(in terms of growth rate)
7/21/2015
5
Mathematical definitions
• O(g(n)) = {f(n):  positive constants c and
n0 such that 0 ≤ f(n) ≤ cg(n)  n>n0}
• Ω(g(n)) = {f(n):  positive constants c and
n0 such that 0 ≤ cg(n) ≤ f(n)  n>n0}
• Θ(g(n)) = {f(n):  positive constants c1, c2,
and n0 such that 0  c1 g(n)  f(n)  c2 g(n)
 n  n0}
7/21/2015
6
O, Ω, and Θ
The definitions imply a constant n0 beyond which they are
satisfied. We do not care about small values of n.
Use definition to prove big-O (Ω,Θ): find constant c (c1 and
c2) and n0, such that the definition of big-O (Ω,Θ) is satisfied
7/21/2015
7
Big-Oh
• Claim: f(n) = 3n2 + 10n + 5  O(n2)
• Proof by definition:
(Hint: Need to find c and n0 such that f(n) <= cn2 for all n > n0. You
can be sloppy about the constant factors. Pick a comfortably large c
when proving big-O or small one when proving big-Omega.)
(Note: you just need to find one concrete example of c and n0, but
the condition needs to be met for all n > n0. So do not try to plug in a
concrete value of n and show the inequality holds.)
Proof:
3n2 + 10n + 5  3n2 + 10n2 + 5,  n > 1
 3n2 + 10n2 + 5n2, n > 1
 18 n2,  n > 1
If we let c = 18 and n0 = 1, we have f(n)  c n2,  n > n0.
Therefore by definition 3n2 + 10n + 5  O(n2).
7/21/2015
8
Use limits to compare orders of
growth
f(n)  o(g(n))
• lim f(n) / g(n) =
n→∞
0
c >0
∞
f(n)  O(g(n))
f(n)  Θ (g(n))
f(n)  Ω(g(n))
f(n)  ω (g(n))
7/21/2015
9
Examples
• Compare 2n and 3n
n / 3n = lim(2/3)n = 0
• lim
2
n→∞
n→∞
• Therefore, 2n  o(3n), and 3n  ω(2n)
• How about 2n and 2n+1?
2n / 2n+1 = ½, therefore 2n = Θ (2n+1)
7/21/2015
10
L’ Hopital’s rule
7/21/2015
lim f(n) / g(n) = lim f(n)’ / g(n)’
Condition:
n→∞
If both lim f(n) and
lim g(n) =  or 0
n→∞
11
Example
• Compare n0.5 and log n
• lim n0.5 / log n = ?
n→∞
•
•
•
•
•
7/21/2015
(n0.5)’ = 0.5 n-0.5
(log n)’ = 1 / n
lim (n-0.5 / 1/n) = lim(n0.5) = ∞
Therefore, log n  o(n0.5)
In fact, log n  o(nε), for any ε > 0
12
Stirling’s formula
n
n
n!  2 n    2 nn1/ 2en
e
or
n! 
7/21/2015
(constant)
n
n 1/ 2  n
e
13
Examples
• Compare 2n and n!
n
n!
c nn n
n
lim n  lim n n  lim c n    
n 2
n 2 e
n
 2e 
• Therefore, 2n = o(n!)
• Compare nn and n!
n!
c nn n
c n
lim n  lim n n  lim n  0
n  n
n  n e
n  e
• Therefore, nn = ω(n!)
• How about log (n!)?
7/21/2015
14
Example
n
c nn
n 1/ 2
n
log(n!)  log n  C  log n
 log(e )
e
1
 C  n log n  log n  n
2
 (n log n)
7/21/2015
15
Properties of asymptotic notations
• Textbook page 51
• Transitivity
f(n) = (g(n)) and g(n) = (h(n))
=> f(n) = (h(n))
(holds true for o, O, , and  as well).
• Symmetry
f(n) = (g(n)) if and only if g(n) = (f(n))
• Transpose symmetry
f(n) = O(g(n)) if and only if g(n) = (f(n))
f(n) = o(g(n)) if and only if g(n) = (f(n))
7/21/2015
16
About exponential and logarithm
functions
• Textbook page 55-56
• It is important to understand what logarithms are and
where they come from.
• A logarithm is simply an inverse exponential function.
• Saying bx = y is equivalent to saying that
x = logb y.
• Logarithms reflect how many times we can double
something until we get to n, or halve something until we
get to 1.
• log21 = ?
• log22 = ?
7/21/2015
17
Binary Search
• In binary search we throw away half the
possible number of keys after each
comparison.
• How many times can we halve n before
getting to 1?
• Answer: ceiling (lg n)
7/21/2015
18
Logarithms and Trees
• How tall a binary tree do we need until we
have n leaves?
• The number of potential leaves doubles
with each level.
• How many times can we double 1 until we
get to n?
• Answer: ceiling (lg n)
7/21/2015
19
Logarithms and Bits
• How many numbers can you represent
with k bits?
• Each bit you add doubles the possible
number of bit patterns
• You can represent from 0 to 2k – 1 with k
bits. A total of 2k numbers.
• How many bits do you need to represent
the numbers from 0 to n?
• ceiling (lg (n+1))
7/21/2015
20
logarithms
•
•
•
•
•
•
•
•
7/21/2015
lg n = log2 n
ln n = loge n, e ≈ 2.718
lgkn = (lg n)k
lg lg n = lg (lg n) = lg(2)n
lg(k) n = lg lg lg … lg n
lg24 = ?
lg(2)4 = ?
Compare lgkn vs lg(k)n?
21
Useful rules for logarithms
For all a > 0, b > 0, c > 0, the following rules hold
• logba = logca / logcb = lg a / lg b
• logban = n logba
• blogba = a
• log (ab) = log a + log b
– lg (2n) = ?
• log (a/b) = log (a) – log(b)
– lg (n/2) = ?
– lg (1/n) = ?
• logba = 1 / logab
7/21/2015
23
More advanced dominance ranking
1
n  n! 3  2  n  n  n
n
n
n
3
2
 n log n ~ log n!
 n  n / log n  n  n  log3 n  log2 n  log n
 log n / log log n  log log n  log n   (n)  1
( 3)
7/21/2015
24
Analyzing the complexity of an algorithm
7/21/2015
25
Kinds of analyses
• Worst case
– Provides an upper bound on running time
• Best case – not very useful, can always
cheat
• Average case
– Provides the expected running time
– Very useful, but treat with care: what is
“average”?
7/21/2015
26
General plan for analyzing time
efficiency of a non-recursive algorithm
• Decide parameter (input size)
• Identify most executed line (basic operation)
• worst-case = average-case?
• T(n) = i ti
• T(n) = Θ (f(n))
7/21/2015
27
Example
repeatedElement (A, n)
// determines whether all elements in a given
// array are distinct
for i = 1 to n-1 {
for j = i+1 to n {
if (A[i] == A[j])
return true;
}
}
return false;
7/21/2015
28
Example
repeatedElement (A, n)
// determines whether all elements in a given
// array are distinct
for i = 1 to n-1 {
for j = i+1 to n {
if (A[i] == A[j])
return true;
}
}
return false;
7/21/2015
29
• Best case?
• Worst-case?
• Average case?
7/21/2015
30
• Best case
– A[1] = A[2]
– T(n) = Θ (1)
• Worst-case
– No repeated elements
– T(n) = (n-1) + (n-2) + … + 1 = n (n-1) / 2 = Θ (n2)
• Average case?
– What do you mean by “average”?
– Need more assumptions about data distribution.
• How many possible repeats are in the data?
– Average-case analysis often involves probability.
7/21/2015
31
Find the order of growth for sums
•
•
•
•
•
T(n) = i=1..n i = Θ (n2)
T(n) = i=1..n log (i) = ?
T(n) = i=1..n n / 2i = ?
T(n) = i=1..n 2i = ?
…
• How to find out the actual order of growth?
– Math…
– Textbook Appendix A.1 (page 1058-60)
7/21/2015
32
Arithmetic series
• An arithmetic series is a sequence of numbers
such that the difference of any two successive
members of the sequence is a constant.
e.g.: 1, 2, 3, 4, 5
or 10, 12, 14, 16, 18, 20
• In general:
ai  ai 1  d
Or:
7/21/2015
ai  a1  (i  1)d
Recursive definition
Closed form, or explicit formula
33
Sum of arithmetic series
If a1, a2, …, an is an arithmetic series, then
n(a1  an )
ai 

2
i 1
n
e.g. 1 + 3 + 5 + 7 + … + 99 = ?
(series definition: ai = 2i-1)
This is ∑ i = 1 to 50 (ai) = 50 * (1 + 99) / 2 = 2500
7/21/2015
34
Geometric series
• A geometric series is a sequence of numbers
such that the ratio between any two successive
members of the sequence is a constant.
e.g.: 1, 2, 4, 8, 16, 32
or 10, 20, 40, 80, 160
or 1, ½, ¼, 1/8, 1/16
• In general:
ai  rai 1
Or:
7/21/2015
ai  r i a0
Recursive definition
Closed form, or explicit formula
35
Sum of geometric series
(1  r n 1 ) /(1  r )
n
 n 1
i
r  ( r  1) /( r  1)

i 0

n 1

if r < 1
if r > 1
if r = 1
n
i
2
 ?
i 0
n
1
limn  i  ?
i 0 2
n
1
limn  i  ?
i 1 2
7/21/2015
36
Sum of geometric series
(1  r n 1 ) /(1  r )
n
 n 1
i
r  ( r  1) /( r  1)

i 0

n 1

if r < 1
if r > 1
if r = 1
n 1
2
1
i
n 1
n 1
n
2


2

1

2


(
2
)

2 1
i 0
n
n
n
1
1
i
1
limn  i  limn  ( 2 ) 
2
1  12
i 0 2
i 0
n
n
1
i
0
1
1
limn  i  limn   2    2   2  1  1
i 1 2
i 0
7/21/2015
37
Important formulas
3
n
i 2   ( n 3 )

3
i 1
n
n
1  n  (n)
i 1
n
i 
i 1
n( n  1)
 (n 2 )
2
 1  (1) ( r  1)
r 


n

(
r
) ( r  1)
r

1
i 0

n
i
r
n 1
n k 1
k 1
i



(
n
)

k 1
i 1
n
k
n
i
n 1
n
i
2

(
n

1
)
2

2


(
n
2
)

i 1
n
1
 (lg n )

i 1 i
n
 lg i  (n lg n)
i 1
7/21/2015
38
Sum manipulation rules
 (a  b )   a   b
 ca  c a
i
i
i
i
i
i i
i i
i i
n
x
n
a  a  a
i m
i
i
i m
i  x 1
i
Example:
n
i
(
4
i

2
)?

i 1
n
n
?

i
i 1 2
7/21/2015
39
Sum manipulation rules
 (a  b )   a   b
 ca  c a
i
i
i
i
n
i
i i
i i
i i
x
n
a  a  a
i m
i
i m
i
i  x 1
i
Example:
n
n
n
i 1
i 1
i 1
i
i
n 1
(
4
i

2
)

4
i

2

2
n
(
n

1
)

2
2

 
n
n
n
1
 n i  n

i
i 1 2
i 1 2
7/21/2015
40
• i=1..n n / 2i = n * i=1..n (½)i = ?
• using the formula for geometric series:
i=0..n (½)i = 1 + ½ + ¼ + … (½)n = 2
• Application: algorithm for allocating
dynamic memories
7/21/2015
41
• i=1..n log (i) = log 1 + log 2 + … + log n
= log 1 x 2 x 3 x … x n
= log n!
= (n log n)
• Application: algorithm for selection sort
using priority queue
7/21/2015
42
Recursive definition of sum of series
• T (n) = i=0..n i is equivalent to:
Recurrence
T(n) = T(n-1) + n
Boundary condition
T(0) = 0
• T(n) = i=0..n ai is equivalent to:
T(n) = T(n-1) + an
T(0) = 1
Recursive definition is often intuitive and easy to obtain. It
is very useful in analyzing recursive algorithms, and some
non-recursive algorithms too.
7/21/2015
43
Recursive definition of sum of
series
• How to solve such recurrence or more
generally, recurrence in the form of:
• T(n) = aT(n-b) + f(n) or
• T(n) = aT(n/b) + f(n)
7/21/2015
44