Transcript CSC401: Analysis of Algorithms

CSC401 – Analysis of Algorithms
Lecture Notes 1
Introduction
Objectives:
Introduce algorithm and algorithm
analysis
Discuss algorithm analysis methodologies
Introduce pseudo code of algorithms
Asymptotic notion of algorithm efficiency
What is an algorithm ?
Input
Algorithm
Output
An algorithm is a
step-by-step procedure
for solving a problem
in a finite amount of
time.
What is algorithm analysis ?
Two aspects:
• Running time – How much time is taken to
complete the algorithm execution?
• Storage requirement – How much memory is
required to execute the program?
2
Running Time
– Easier to analyze
– Crucial to applications such
as games, finance and
robotics
best case
average case
worst case
120
100
Running Time
Most algorithms
transform input objects
into output objects.
The running time of an
algorithm typically grows
with the input size.
Average case time is
often difficult to
determine.
We focus on the worst
case running time.
80
60
40
20
0
1000
2000
3000
4000
Input Size
3
How Is An Algorithm Analyzed?
Two methodologies
9000
– Experimental analysis
8000
Method
7000
Time (ms)
6000
– Write a program implementing the
5000
algorithm
4000
– Run the program with inputs of
3000
varying size and composition
2000
– Use a method like System.currentTimeMillis() 1000
to get an accurate measure of the
0
actual running time
0
– Plot the results
50
100
Input Size
Limitations
– It is necessary to implement the algorithm, which may
be difficult
– Results may not be indicative of the running time on
other inputs not included in the experiment.
– In order to compare two algorithms, the same
hardware and software environments must be used
4
How Is An Algorithm Analyzed?
– Theoretical Analysis
Method
– Uses a high-level description of the algorithm
instead of an implementation
– Characterizes running time as a function of the
input size, n.
– Takes into account all possible inputs
– Allows us to evaluate the speed of an algorithm
independent of the hardware/software
environment
Characteristics
– A description language -- Pseudo code
– Mathematics
5
Pseudocode
High-level description
Example: find max
of an algorithm
element of an array
More structured than
Algorithm arrayMax(A, n)
English prose
Input array A of n integers
Less detailed than a
Output maximum element of A
program
Preferred notation for
currentMax  A[0]
describing algorithms
for i  1 to n  1 do
Hides program design
if A[i]  currentMax then
issues
currentMax  A[i]
return currentMax
6
Pseudocode Details
Control flow
–
–
–
–
–
–
if … then … [else …]
switch …
while … do …
repeat … until …
for … do …
Indentation replaces
braces
Method declaration
Algorithm method (arg [, arg…])
Input …
Output …
Method call
var.method (arg [, arg…])
Return value
return expression
Expressions
 Assignment
(like  in Java)
 Equality testing
(like  in Java)
n2 Superscripts and other
mathematical
formatting allowed
Array indexing: A[i]
7
The Random Access Machine (RAM) Model
A CPU
An potentially
unbounded bank of
memory cells, each of
which can hold an
arbitrary number or
character
2
0
1
Memory cells are numbered and
accessing any cell in memory takes
unit time.
8
Primitive Operations
Basic computations
performed by an algorithm
Identifiable in pseudocode
Largely independent from
the programming language
Exact definition not
important (we will see why
later)
Assumed to take a
constant amount of time in
the RAM model
Examples:
– Evaluating an
expression
– Assigning a value to
a variable
– Performing an
arithmetic operation
– Comparing two
numbers
– Indexing into an
array
– Following an object
reference
– Calling a method
– Returning from a
method
9
Counting Primitive Operations
By inspecting the pseudocode, we can determine
the maximum number of primitive operations
executed by an algorithm, as a function of the
input size
Algorithm arrayMax(A, n)
currentMax  A[0]
for i  1 to n  1 do
if A[i]  currentMax then
currentMax  A[i]
{ increment counter i }
return currentMax
# operations
2
1+n
2(n  1)
2(n  1)
2(n  1)
1
Total 7n  2
10
Estimating Running Time
Algorithm arrayMax executes 7n  2 primitive
operations in the worst case.
Define:
a = Time taken by the fastest primitive operation
b = Time taken by the slowest primitive operation
Let T(n) be the worst-case time of arrayMax.
Then
a (7n  1)  T(n)  b(7n  1)
Hence, the running time T(n) is bounded by
two linear functions
11
Growth Rate of Running Time
Changing the hardware/ software environment
– Affects T(n) by a constant factor, but
– Does not alter the growth rate of T(n)
– Linear  n
– Quadratic  n2
– Cubic  n3
In a log-log chart, the
slope of the line
corresponds to the
growth rate of the
function
T (n )
The linear growth rate of the running time T(n) is
an intrinsic property of algorithm arrayMax
1E+30
Growth rates of
1E+28
Cubic
1E+26
functions:
Quadratic
1E+24
1E+22
1E+20
1E+18
1E+16
1E+14
1E+12
1E+10
1E+8
1E+6
1E+4
1E+2
1E+0
1E+0
Linear
1E+2
1E+4
1E+6
n
1E+8
1E+10
12
Constant Factors
The growth rate is
not affected by
Examples
– 102n + 105 is a linear
function
– 105n2 + 108n is a
quadratic function
T (n )
– constant factors or
– lower-order terms
1E+26
1E+24
1E+22
1E+20
1E+18
1E+16
1E+14
1E+12
1E+10
1E+8
1E+6
1E+4
1E+2
1E+0
1E+0
Quadratic
Quadratic
Linear
Linear
1E+2
1E+4
1E+6
1E+8
1E+10
n
13
Big-Oh Notation
Given functions f(n) and g(n),
we say that f(n) is O(g(n)) if
there are positive constants
c and n0 such that
f(n)  cg(n) for n  n0
Example: 2n + 10 is O(n)
–
–
–
–
3n
2n+10
1,000
n
100
10
1
1
2n + 10  cn
(c  2) n  10
n  10/(c  2)
Pick c  3 and n0  10
Example: the function
not O(n)
10,000
10
100
1,000
n
1,000,000
n^2
100n
100,000
n2
is
– n2  cn
– n c
– The above inequality cannot
be satisfied since c must be
a constant
10n
n
10,000
1,000
100
10
14
1
1
10
100
1,000
More Big-Oh Examples
7n-2

7n-2 is O(n)
need c > 0 and n0  1 such that 7n-2  c•n for n  n0
this is true for c = 7 and n0 = 1
3n3 + 20n2 + 5



3n3 + 20n2 + 5 is O(n3)
need c > 0 and n0  1 such that 3n3 + 20n2 + 5  c•n3
for n  n0
this is true for c = 4 and n0 = 21
3 log n + log log n



3 log n + log log n is O(log n)
need c > 0 and n0  1 such that 3 log n + log log n 
c•log n for n  n0
this is true for c = 4 and n0 = 2
15
Big-Oh and Growth Rate
The big-Oh notation gives an upper bound on
the growth rate of a function
The statement “f(n) is O(g(n))” means that the
growth rate of f(n) is no more than the growth
rate of g(n)
We can use the big-Oh notation to rank
functions according to their growth rate
g(n) grows more
f(n) grows more
Same growth
f(n) is O(g(n))
g(n) is O(f(n))
Yes
No
Yes
No
Yes
Yes
16
Big-Oh Rules
If is f(n) a polynomial of degree d, then
f(n) is O(nd), i.e.,
1.Drop lower-order terms
2.Drop constant factors
Use the smallest possible class of
functions
– Say “2n is O(n)” instead of “2n is O(n2)”
Use the simplest expression of the class
– Say “3n + 5 is O(n)” instead of “3n + 5 is
O(3n)”
17
Asymptotic Algorithm Analysis
The asymptotic analysis of an algorithm
determines the running time in big-Oh notation
To perform the asymptotic analysis
– We find the worst-case number of primitive operations
executed as a function of the input size
– We express this function with big-Oh notation
Example:
– We determine that algorithm arrayMax executes at
most 7n  2 primitive operations
– We say that algorithm arrayMax “runs in O(n) time”
Since constant factors and lower-order terms are
eventually dropped anyhow, we can disregard
them when counting primitive operations
18