Transcript CSCE 3110 Data Structures & Algorithm Analysis Algorithm Analysis II

CSCE 3110
Data Structures &
Algorithm Analysis
Algorithm Analysis II
Reading: Weiss, chap. 2
Algorithm Analysis
We know:
Experimental approach – problems
Low level analysis – count operations
Abstract even further
Characterize an algorithm as a function of the
“problem size”
E.g.
Input data = array  problem size is N (length of
array)
Input data = matrix  problem size is N x M
Asymptotic Notation
Goal: to simplify analysis by getting rid of
unneeded information (like “rounding”
1,000,001≈1,000,000)
We want to say in a formal way 3n2 ≈ n2
The “Big-Oh” Notation:
given functions f(n) and g(n), we say that
f(n) is O(g(n)) if and only if there are
positive constants c and n0 such that f(n)≤ c
g(n) for n ≥ n0
Graphic Illustration
c g(n) = 4n
f(n) = 2n + 6
f(n) = 2n+6
Conf. def:
Need to find a
function g(n) and a
const. c and a
constant n0 such as
f(n) < cg(n) when
n>n0
g(n) = n and c = 4
and n0=3
 f(n) is O(n)
The order of f(n) is n
g(n) = n
n
More examples
What about f(n) = 4n2 ? Is it O(n)?
Find a c and n0 such that 4n2 < cn for any n > n0
50n3 + 20n + 4 is O(n3)
Would be correct to say is O(n3+n)
• Not useful, as n3 exceeds by far n, for large values
Would be correct to say is O(n5)
• OK, but g(n) should be as close as possible to f(n)
3log(n) + log (log (n)) = O( ? )
•Simple Rule: Drop lower order
terms and constant factors
Big-Oh Rules
If f1(n)=O(g1(n)) and f2(n)=O(g2(n))
f1(n)+f2(n)=O(g1(n)+g2(n))=max(O(g1(n)+g2(n))
f1(n)*f2(n)=O(g1(n)*g2(n))
logkN=O(N) for any constant k
The relative growth rate of two functions can always be
determined by computing their limit
But using this method is almost always an overkill
lim f1 (n) / f 2 (n)
n 
Big-Oh and Growth Rate
The big-Oh notation gives an upper bound on the growth rate of
a function.
The statement “f(n) is O(g(n))” means that the growth rate of f(n) is
no more than the growth rate of g(n).
We can use the big-Oh notation to rank functions according to
their growth rate.
f(n) is O(g(n))
g(n) is O(f(n))
g(n) grows faster
Yes
No
f(n) grows faster
No
Yes
Same growth
Yes
Yes
Classes of Functions
Let {g(n)} denote the class (set) of functions that are
O(g(n))
We have
{n}  {n2}  {n3}  {n4}  {n5}  …
where the containment is strict
{n3}
{n2}
{n}
Big-Oh Rules
If is f(n) a polynomial of degree d, then f(n) is O(nd),
i.e.,
1.
2.
Drop lower-order terms
Drop constant factors
Use the smallest possible class of functions
Say “2n is O(n)” instead of “2n is O(n2)”
Use the simplest expression of the class
Say “3n + 5 is O(n)” instead of “3n + 5 is O(3n)”
Inappropriate Expressions
f (n)X
 O( g (n))
f (n)X
 O( g (n))
Properties of Big-Oh
If f(n) is O(g(n)) then af(n) is O(g(n)) for any a.
If f(n) is O(g(n)) and h(n) is O(g’(n)) then f(n)+h(n) is
O(g(n)+g’(n))
If f(n) is O(g(n)) and h(n) is O(g’(n)) then f(n)h(n) is
O(g(n)g’(n))
If f(n) is O(g(n)) and g(n) is O(h(n)) then f(n) is O(h(n))
If f(n) is a polynomial of degree d , then f(n) is O(nd)
nx = O(an), for any fixed x > 0 and a > 1
An algorithm of order n to a certain power is better than an algorithm of
order a ( > 1) to the power of n
log nx is O(log n), for x > 0 – how?
log x n is O(ny) for x > 0 and y > 0
An algorithm of order log n (to a certain power) is better than an algorithm of
n raised to a power y.
Asymptotic analysis terminology
Special classes of algorithms:
logarithmic:
linear:
quadratic:
polynomial:
exponential:
O(log n)
O(n)
O(n2)
O(nk), k ≥ 1
O(an), n > 1
Polynomial vs. exponential ?
Logarithmic vs. polynomial ?
Graphical explanation?
Some Numbers
log n
n
0
1
2
3
4
5
n log n
1
2
4
8
16
32
0
2
8
24
64
160
n2
1
4
16
64
256
1024
n3
2n
1
2
8
4
64
16
512
256
4096
65536
32768 4294967296
Computing Prefix Averages
35
The i-th prefix average of an array X
is average of the first (i + 1) elements 30
of X
25
A[i] = (X[0] + X[1] + … + X[i])/(i+1)
X
A
20
Computing the array A of prefix
averages of another array X has
applications to financial analysis
15
10
5
0
1
2
3
4
5
6
7
Prefix Averages (Quadratic)
The following algorithm computes prefix averages in quadratic
time by applying the definition
Algorithm prefixAverages1(X, n)
Input array X of n integers
Output array A of prefix averages of X #operations
A  new array of n integers
n
for i  0 to n  1 do
n
s  X[0]
n
for j  1 to i do
1 + 2 + …+ (n  1)
s  s + X[j]
1 + 2 + …+ (n  1)
A[i]  s / (i + 1)
n
return A
1
Arithmetic Progression
The running time of
prefixAverages1 is
O(1 + 2 + …+ n)
The sum of the first n integers
is n(n + 1) / 2
There is a simple visual
proof of this fact
Thus, algorithm
prefixAverages1 runs in O(n2)
time
7
6
5
4
3
2
1
0
1
2
3
4
5
6
Prefix Averages (Linear)
The following algorithm computes prefix averages in linear time
by keeping a running sum
Algorithm prefixAverages2(X, n)
Input array X of n integers
Output array A of prefix averages of X
A  new array of n integers
s0
for i  0 to n  1 do
s  s + X[i]
A[i]  s / (i + 1)
return A
#operations
Algorithm prefixAverages2 runs in O(n) time
n
1
n
n
n
1
A table of functions wrt input n, assume that each primitive
operation takes one microsecond (1 second = 106 microsecond).
O(g(n))
1 Second
1 Hour
1 Month
1 Century
log2 n
 10300000
 1010
9
 100.8*10
n
 1012
 1.3*1019
 6.8*1024
9.7*1030
n
106
3.6*109
 2.6*1012
 3.12*1015
n log2 n
 105
 109
 1011
 1014
n2
1000
6*104
 1.6*106
 5.6*107
n3
100
 1500
 14000
 1500000
2n
19
31
41
51
n!
9
12
15
17
12
 1010
15
Running Time Calculations
General Rules
1.
FOR loop
•
2.
The number of iterations times the time of the inside statements.
Nested loops
•
3.
The product of the number of iterations times the time of the
inside statements.
Consecutive Statements
•
4.
•
The sum of running time of each segment.
If/Else
The testing time plus the larger running time of the cases.
Some Examples
Case1: for (i=0; i<n; i++)
for (j=0; j<n; j++)
k++;
O(n2)
Case 2: for (i=0; i<n; i++)
k++;
for (i=0; i<n; i++)
for (j=0; j<n; j++)
k++;
O(n2)
Case 3: for (int i=0; i<n-1; i++)
for (int j=0; j<i; j++)
int k+=1;
O(n2)
Maximum Subsequence Sum
Problem
Given a set of integers A1, A2, …, AN, find the maximum
j
value of

k =i
Ak
For convenience, the maximum subsequence sum is
zero if all the integers are negtive.
Algorithm 1
int MaxSubSum1(const vector <int> & a) {
int maxSum=0;
for (int i=0; i<a.size(); i++)
for (int j=i; j<a.size(); j++) {
int thisSum=0;
for (int k=i; k<=j; k++)
thisSum+=a[k];
if (thisSum>maxSum)
maxSum=thisSum;
}
return maxSum;
}
3
O(n )
Algorithm 2
int MaxSubSum2(const vector <int> & a) {
int maxSum=0;
for (int i=0; i<a.size(); i++) {
thisSum=0;
for (int j=i; j<a.size(); j++) {
thisSum+=a[j];
if (thisSum>maxSum)
maxSum=thisSum;
}
}
return maxSum;
}
2
O(n )
Algorithm 3
int maxSumRec (const verctor<int> & a, int left, int right) {
if (left==right)
if (a[left]>0)
return a[left];
else
return 0;
int center = (left+right)/2;
int maxLeftSum=maxSumRec(a, left, center);
int maxRightSum=maxSumRec(a, center+1, right);
int maxLeftBorderSum=0, leftBorderSum=0;
for (int i=center; i>=left; i--) {
leftBorderSum += a[i];
if (leftBorderSum>maxLeftBorderSum)
maxLeftBorderSum=leftBorderSum;
}
int maxRightBorderSum=0; rightBorderSum=0;
for (int j=center+1; j<=right; j++) {
rightBorderSum+=a[j];
if (rightBorderSum>maxrightBorderSum)
maxRightBorderSum=rightBorderSum;
}
return max3(maxLeftSum, maxRightSum,
maxLeftBorderSum+maxRightBorderSum);
}
int maxSubSum3 (const vector <int> & a) {
return maxSumRec(a, 0, a.size()-1);
}
T(n’)=T(n/2)
O(n)
T(1) = 1
4 = 2*2
T(n)=2 T(n/2) + O(n)
T(2) = ?12 = 4*3
T(4) = ?32 = 8*4
T(8) = ?
…
If n=2k, T(n)=n*(k+1)
k=log n
T(n)=n(log n + 1)
Logarithms in the Running Time
An algorithm is O(log N) if it takes constant time to cut the problem size by
a fraction (usually ½).
On the other hand, if constant time is required to merely reduce the
problem by a constant amount (such as to make the problem smaller by 1),
then the algorithm is O(N)
Examples of the O(log N)
Binary Search
Euclid’s algorithm for computing the greatest common divisor
Analyzing recursive algorithms
function foo (param A, param B) {
statement 1;
statement 2;
if (termination condition) {
return;
foo(A’, B’);
}
Solving recursive equations by
repeated substitution
T(n)
T(n)
=
=
=
=
=
=
T(n/2) + c
T(n/4) + c + c
T(n/8) + c + c + c
T(n/23) + 3c
…
T(n/2k) + kc
substitute for T(n/2)
substitute for T(n/4)
in more compact form
“inductive leap”
= T(n/2logn) + clogn
“choose k = logn”
= T(n/n) + clogn
= T(1) + clogn = b + clogn = θ(logn)
Solving recursive equations by
telescoping
T(n)
=
T(n/2) =
T(n/4) =
T(n/8) =
…
T(4)
=
T(2)
=
T(n)
=
T(n)
T(n/2) + c
initial equation
T(n/4) + c
T(n/8) + c
T(n/16) + c
so this holds
and this …
and this …
T(2) + c
T(1) + c
T(1) + clogn
eventually …
and this …
sum equations, canceling the
terms appearing on both sides
= θ(logn)
Algorithm 4
int MaxSubSum4(const vector <int> & a) {
int maxSum=0, thisSum=0;
for (int j=0; j<a.size(); j++) {
thisSum+=a[j];
if (thisSum>maxSum)
maxSum=thisSum;
else if (thisSum<0)
thisSum=0;
}
return maxSum;
}
O(n)
Problem:
Order the following functions by their
asymptotic growth rates
nlogn
logn3
n2
n2/5
2logn
log(logn)
Sqr(logn)
Back to the original question
Which solution would you choose?
O(n2) vs. O(n)
Some math …
properties of logarithms:
logb(xy) = logbx + logby
logb (x/y) = logbx - logby
logbxa= alogbx
logba=
logxa/logxb
properties of exponentials:
a(b+c) = aba c
abc = (ab)c
ab /ac = a(b-c)
b = a logab
bc = a c*logab
“Relatives” of Big-Oh
“Relatives” of the Big-Oh
 (f(n)): Big Omega – asymptotic lower bound
 (f(n)): Big Theta – asymptotic tight bound
Big-Omega – think of it as the inverse of O(n)
g(n) is  (f(n)) if f(n) is O(g(n))
Big-Theta – combine both Big-Oh and Big-Omega
f(n) is  (g(n)) if f(n) is O(g(n)) and g(n) is  (f(n))
Make the difference:
3n+3 is O(n) and is  (n)
3n+3 is O(n2) but is not  (n2)
More “relatives”
Little-oh – f(n) is o(g(n)) if f(n) is O(g(n)) and
f(n) is not  (g(n))
2n+3 is o(n2)
2n + 3 is o(n) ?
Important Series
N
S ( N ) = 1 + 2 +  + N =  i = N (1 + N ) / 2
i =1
Sum of squares:
Sum of exponents:
N ( N + 1)(2 N + 1) N 3
i =

for large N

6
3
i =1
N
2
N k +1
i 
for large N and k  -1

| k +1|
i =1
N
Geometric series:
Special case when A = 2
k
A N +1  1
A =

A 1
i =0
N
• 20 + 21 + 22 + … + 2N = 2N+1 - 1
i
Problem
Running time for finding a number in a
sorted array
[binary search]
Pseudo-code
Running time analysis
ADT
ADT = Abstract Data Types
A logical view of the data objects together
with specifications of the operations required
to create and manipulate them.
Describe an algorithm – pseudo-code
Describe a data structure – ADT
What is a data type?
A set of objects, each called an instance of the data type.
Some objects are sufficiently important to be provided
with a special name.
A set of operations. Operations can be realized via
operators, functions, procedures, methods, and special
syntax (depending on the implementing language)
Each object must have some representation (not
necessarily known to the user of the data type)
Each operation must have some implementation (also not
necessarily known to the user of the data type)
What is a representation?
A specific encoding of an instance
This encoding MUST be known to implementors
of the data type but NEED NOT be known to
users of the data type
Terminology: "we implement data types using
data structures“
Two varieties of data types
Opaque data types in which the representation is
not known to the user.
Transparent data types in which the representation
is profitably known to the user:- i.e. the encoding
is directly accessible and/or modifiable by the
user.
Which one you think is better?
What are the means provided by C++ for
creating opaque data types?
Why are opaque data types better?
Representation can be changed without affecting
user
Forces the program designer to consider the
operations more carefully
Encapsulates the operations
Allows less restrictive designs which are easier to
extend and modify
Design always done with the expectation that the
data type will be placed in a library of types
available to all.
How to design a data type
Step 1: Specification
Make a list of the operations (just their names)
you think you will need. Review and refine the
list.
Decide on any constants which may be required.
Describe the parameters of the operations in detail.
Describe the semantics of the operations (what
they do) as precisely as possible.
How to design a data type
Step 2: Application
Develop a real or imaginary application to test the
specification.
Missing or incomplete operations are found as a
side-effect of trying to use the specification.
How to design a data type
Step 3: Implementation
Decide on a suitable representation.
Implement the operations.
Test, debug, and revise.
Example - ADT Integer
Name of ADT
Integer
Operation
Create
Description
Defines an identifier with an
undefined value
Assigns the value of one integer
identifier or value to another integer
identifier
Returns true if the values associated
with two integer identifiers are the
same
Assign
isEqual
C/C++
int id1;
id1 = id2;
id1 == id2;
Example – ADT Integer
LessThan
Negative
Sum
Returns true if an identifier integer is
less than the value of the second
integer identifier
Returns the negative of the integer value
Returns the sum of two integer values
Operation Signatures
Create: identifier  Integer
Assign: Integer  Identifier
IsEqual: (Integer,Integer)  Boolean
LessThan: (Integer,Integer)  Boolean
Negative: Integer  Integer
Sum: (Integer,Integer)  Integer
id1<id2
-id1
id1+id2
More examples
We’ll see more examples throughout the
course
Stack
Queue
Tree
And more