Transcript TCSS 342 Lecture Notes: Algorithm Analysis

TCSS 342 Lecture Notes
Course Overview,
Review of Math Concepts,
Algorithm Analysis and Big-Oh Notation
Weiss book, Chapter 5
These lecture notes are copyright (C) Marty Stepp, 2005. They may not be rehosted, sold, or
modified without expressed permission from the author. All rights reserved.
1
Lecture outline

Introduction and course objectives

Mathematical background review



Algorithm analysis and Big-Oh notation




exponents and logarithms
arithmetic and geometric series
the RAM model
Big-Oh
Big-Omega, Big-Theta, Little-Oh, Little-Omega
Comparison of algorithms to solve the Maximum
Contiguous Subsequence Sum problem
2
Course objectives

(broad) prepare you to be a good software engineer

(specific) learn basic data structures and algorithms



data structures – how data is organized
algorithms – unambiguous sequence of steps to compute
something
algorithm analysis – determining how long an algorithm will take to
solve a problem


Who cares? Aren't computers fast enough and getting faster?
"Data Structures + Algorithms = Programs"
-- Niklaus Wirth, author of Pascal language
3
An example

Given an array of 1,000,000 integers, find the
maximum integer in the array.
…
0

1
2
999,999
Now suppose we are asked to find the kth largest
element. (The Selection Problem)
4
Candidate solutions

candidate solution 1



sort the entire array (from small to large), using Java's
Arrays.sort()
pick out the (1,000,000 – k)th element
candidate solution 2


place the first k elements into a sorted array of size k
for each of the remaining 1,000,000 – k elements,


keep the k largest in the array
pick out the smallest of the k survivors and toss it
5
Is either solution good?




What advantages and disadvantages does each solution
have?
Is there a better solution?
What makes a solution "better" than another? Is it
entirely based on runtime?
How would you go about determining which solution is
better?
 could code them, test them
 could somehow make predictions and analysis of
each solution, without coding
6
Why algorithm analysis?



as computers get faster and problem sizes get bigger,
analysis will become more important
The difference between good and bad algorithms will
get bigger
being able to analyze algorithms will help us identify
good ones without having to program them and test
them first
7
Why data structures?



when programming, you are an engineer
engineers have a bag of tools and tricks – and the
knowledge of which tool is the right one for a given
problem
Examples: arrays, lists, stacks, queues, trees, hash
tables, heaps, graphs
8
Mathematical background
review
Let's review some math material that'll be useful later when
computing the runtime of various algorithms.
9
Math background: exponents

Exponents


XY , or "X to the Yth power";
X multiplied by itself Y times
Some useful identities





XA XB = XA+B
XA / XB = XA-B
(XA)B = XAB
XN+XN = 2XN
2N+2N = 2N+1
10
Math background:Logarithms

Logarithms




definition: XA = B if and only if logX B = A
intuition: logX B means:
"the power X must be raised to, to get B"
In this course, a logarithm with no base implies base 2.
log B means log2 B
Examples


log2 16 = 4
(because 24 = 16)
log10 1000 = 3 (because 103 = 1000)
11
Logarithm identities
Identities for logs with addition, multiplication, powers:
 log (AB) = log A + log B
 log (A/B) = log A – log B
 log (AB) = B log A
Identity for converting bases of a logarithm:

logC B
logA B 
logC A

A, B, C  0, A  1
example:
log432 = (log2 32) / (log2 4)
=5/2
12
Logarithm problem solving

When presented with an expression of the form:

logaX = Y
and trying to solve for X, raise both sides to the a
power.


X = aY
When presented with an expression of the form:

logaX = logbY
and trying to solve for X, find a common base between
the logarithms using the identity on the last slide.

logaX = logaY / logab
13
Logarithm practice problems

Determine the value of x in the following equation.


log7 x + log713 = 3
Determine the value of x in the following equation.

log8 4 - log8 x = log8 5 + log16 6
14
Math review: Arithmetic series

Series
k
 Expr
i j

for some expression Expr (possibly containing i ), means the
sum of all values of Expr with each value of i between j and k
inclusive
4

Example:
 2i  1
i 0

= (2(0) + 1) + (2(1) + 1) + (2(2) + 1)
+ (2(3) + 1) + (2(4) + 1)
=1+3+5+7+9
= 25
15
Series identities

sum from 1 through N inclusive
N ( N  1)
i

2
i 1
N

is there an intuition for this identity?

sum of all numbers from 1 to N
1 + 2 + 3 + ... + (N-2) + (N-1) + N

how many terms are in this sum? Can we rearrange them?
16
More series identities

sum from a through N inclusive
(when the series doesn't start at 1)
N
N
a 1
i a
i 1
i 1
i  i  i

is there an intuition for this identity?

The sum of all numbers from 4 to 10 is just the sum of all
numbers from 1 to 10, minus the sum of 1 through 3.
17
Series of constants

sum of constants
(when the body of the series doesn't contain the
counter variable such as i)
b
b
i a
i a
 k  k 1  k b  a  1

example:
10
10
i 4
i 4
 5  51  510  4  1  35
18
Splitting series
for any constant k,
 splitting a sum with addition
b
b
b
i a
i a
i a
 i  k    i   k

moving out a constant multiple
b
b
i a
i a
 ki  k  i
19
Series of powers

sum of powers of 2
N
2
i
2
N 1
1
i 0


1 + 2 + 4 + 8 + 16 + 32 = 64 - 1 = 63
think about binary representation of numbers...
111111 (63)
+
1 (1)
1000000 (64)

when the series doesn't start at 0:
N
a 1
N
2  2  2
i
i a
i
i 0
i 0
i
20
Series practice problems

Give a closed form expression for the following
summation.

A closed form expression is one without the S or "…".
N 2
 2i
i 0

Give a closed form expression for the following
summation.
N 1
 i  5
i 10
21
Algorithm analysis and Big-oh
notation
22
Algorithm performance



How to determine how much time an algorithm A uses to solve
problem X ?
 Depends on input; use input size "N" as parameter
 Determine function f(N) representing cost
empirical analysis: code it and use timer running on
many inputs
algorithm analysis: Analyze steps of algorithm,
estimating amount of work each step takes
23
RAM model

Typically use a simple model for basic operation costs

RAM (Random Access Machine) model




RAM model has all the basic operations:
+, -, *, / , =, comparisons
fixed sized integers (e.g., 32-bit)
infinite memory
All basic operations take exactly one time unit (one CPU
instruction) to execute
24
Critique of the model

Strengths:




simple
easier to prove things about the model than the real machine
can estimate algorithm behavior on any hardware/software
Weaknesses:


not all operations take the same amount of time in a real
machine
does not account for page faults, disk accesses, limited
memory, floating point math, etc
25
Relative rates of growth



most algorithms' runtime can be expressed as a
function of the input size N
rate of growth: measure of how quickly the graph of
a function rises
goal: distinguish between fast- and slow-growing
functions


we only care about very large input sizes
(for small sizes, most any algorithm is fast enough)
this helps us discover which algorithms will run more quickly or
slowly, for large input sizes
26
Growth rate example
Consider these graphs of functions.
Perhaps each one represents an algorithm:
n3 + 2n2
100n2 + 1000
• Which grows
faster?
27
Growth rate example
• How about now?
28
Review: Big-Oh notation



Defn:
T(N) = O(f(N))
if there exist positive constants c , n0 such that:
T(N)  c · f(N) for all N  n0
idea: We are concerned with how the function grows
when N is large. We are not picky about constant
factors: coarse distinctions among functions
Lingo: "T(N) grows no faster than f(N)."
29
Big-Oh example problems

n = O(2n) ?

2n = O(n) ?

n = O(n2) ?

n2 = O(n) ?

n = O(1) ?

100 = O(n) ?

10 log n = O(n) ?

214n + 34 = O(2n2 + 8n) ?
30
Preferred big-Oh usage

pick tightest bound. If f(N) = 5N, then:
f(N)
f(N)
f(N)
f(N)

=
=
=
=
O(N5)
O(N3)
O(N log N)
O(N)
 preferred
ignore constant factors and low order terms
T(N) = O(N), not T(N) = O(5N)
T(N) = O(N3), not T(N) = O(N3 + N2 + N log N)



Wrong: f(N)  O(g(N))
Wrong: f(N)  O(g(N))
remove non-base-2 logarithms
f(N) = O(N log6 N)
f(N) = O(N log N)
 preferred
31
Big-Oh of selected functions
32
Ten-fold processor speedup


"Who cares about choosing the fastest algorithm. If it
is too slow, we'll just upgrade the computer..."
Example data when computer is sped up by a factor of 10:
33
Big omega, theta

Defn: T(N) = (g(N)) if there are positive constants c
and n0 such that T(N)  c g(N) for all N  n0


Defn: T(N) = (g(N)) if and only if T(N) = O(g(N)) and
T(N) = (g(N)).


Lingo: "T(N) grows no slower than g(N)."
Big-Oh, Omega, and Theta establish a relative ordering among
all functions of N
Defn: T(N) = o(g(N)) if and only if T(N) = O(h(N)) and
T(N)  (h(N)).
34
Intuition about the notations
notation
intuition
O (Big-Oh)

 (Big-Omega)

 (Theta)
=
o (little-Oh)
<
35
More about asymptotics


Fact: If f(N) = O(g(N)), then g(N) = (f(N)).
Proof: Suppose f(N) = O(g(N)).
Then there exist constants c and n0 such that f(N)  c
g(N) for all N  n0
Then g(N)  (1/c) f(N) for all N  n0,
and so g(N) = (f(N))
36
More terminology

T(N) = O(f(N))



T(N) = (g(N))



f(N) is an upper bound on T(N)
T(N) grows no faster than f(N)
g(N) is a lower bound on T(N)
T(N) grows at least as fast as g(N)
T(N) = o(h(N))

T(N) grows strictly slower than h(N)
37
Facts about big-Oh

If T1(N) = O(f(N)) and T2(N) = O(g(N)), then



If T(N) is a polynomial of degree k, then:
T(N) = (Nk)


T1(N) + T2(N) = O(f(N) + g(N))
T1(N) * T2(N) = O(f(N) * g(N))
example: 17n3 + 2n2 + 4n + 1 = (n3)
logk N = O(N), for any constant k
38
Hierarchy of Big-Oh

Functions, ranked in increasing order of growth:











1
log log n
log n
n
n log n
n2
n2 log n
n3
...
2n
n!
nn
39
Techniques


Algebra
ex. f(N) = N / log N
g(N) = log N
same as asking which grows faster, N or log2 N
Evaluate:
f (N )
lim
N  g ( N )
limit is
Big-Oh relation
0
f(N) = o(g(N))
c0
f(N) = (g(N))

g(N) = o(f(N))
no limit
no relation
40
Techniques, cont'd

L'Hôpital's rule:
If
lim
N 
f ( N )   and
lim
N 
g ( N )   , then
f (N )
f '(N )
lim
 lim
N  g ( N )
N  g ' ( N )
example: f(N) = N, g(N) = log N
Use L'Hôpital's rule
f'(N) = 1, g'(N) = 1/N
 g(N) = o(f(N))
41
Program loop runtimes
for (int i = 0; i < n; i += c)
statement(s);

Adding to the loop counter means that the loop runtime grows
linearly when compared to its maximum value n.

Loop executes its body exactly n / c times.
for (int i = 0; i < n; i *= c)
statement(s);

// O(n)
// O(log n)
Multiplying the loop counter means that the maximum value n
must grow exponentially to linearly increase the loop runtime;
therefore, it is logarithmic.

Loop executes its body exactly logc n times.
for (int i = 0; i < n * n; i += c)
statement(s);

// O(n2)
The loop maximum is n2, so the runtime is quadratic.

Loop executes its body exactly (n2 / c) times.
42
More loop runtimes

Nesting loops multiplies their runtimes.
for (int i = 0; i < n; i += c) {
for (int j = 0; j < n; i += c) {
statement;
}
}

// O(n2)
Loops in sequence add together their runtimes, which
means the loop set with the larger runtime dominates.
for (int i = 0; i < n; i += c) {
statement;
}
for (int i = 0; i < n; i += c) {
for (int j = 0; j < n; i *= c) {
statement;
}
}
// O(n)
// O(n log n)
43
Loop runtime problems

Compute the exact value of the variable sum after the
following code fragment, as a closed-form expression in
terms of input size n.
int sum = 0;
for (int i = 1; i <= n; i *= 2) {
sum++;
}
for (int i = 1; i <= 1000; i++) {
sum++;
sum++;
}
44
Loop runtime problems

Compute the exact value of the variable sum after the
following code fragment, as a closed-form expression in
terms of input size n.
int sum = 0;
for (int i = 1; i <= n; i++) {
for (int j = 1; j <= i / 2; j += 2) {
sum++;
}
}
45
"Maximum subsequence sum"
problem
46
Maximum subsequence sum
The maximum subsequence sum problem:
Given a sequence of integers A0, A1, ..., An - 1,
j
find the maximum value of
A
k i
k
for any integers 0  (i, j) < n.
(This sum is zero if all numbers in the sequence are negative.)
For example, the maximum subseq. sum of this list is 33:
[2, 9, -4, 1, -20, 3, 15, -10, 20, 5, -100, 10, 7]
47
Thoughts about the problem

This is a maximization problem.


When solving maximization problems, one possible
technique is to generate all possible solutions, compare
them, and choose the appropriate one.


There are a set of possible answers (sums of subsets of the
original array), and we are being asked to find the largest one.
In the context of this problem, that would mean generating all
possible subsets of the original array, adding them up to
determine each of their sums, and comparing these sums to
choose the best one of them all.
brute force algorithm: A method for solving a
problem that proceeds in a simple and obvious way,
possibly making it require more steps than necessary.
48
First algorithm (brute force)
generate and test all possible subsequences
// implement together
function maxSubsequence(array[]):
max sum = 0
for each starting index i,
for each ending index j,
add up the sum from Ai to Aj
if this sum is bigger than max,
max sum = this sum
return max sum


Using the RAM model, how many steps does this algorithm
require?
What do you believe is the runtime (Big-Oh) of this algorithm?


How could we empirically test this?
What is wasteful about the algorithm? How could it be improved?
49
More observations


The preceding algorithm is a brute force algorithm, but
it isn't even the best brute force algorithm.
We are redundantly re-computing a large number of
values many times.

For example, we compute the sum of the subsequence between
indexes 2 and 5,
A[2] + A[3] + A[4] + A[5]
next we compute the sum of the subsequence between indexes
2 and 6.
A[2] + A[3] + A[4] + A[5] + A[6]


We already had computed the sum of 2--5, but we compute it
again as part of the 2--6 computation.
Assuming from the RAM model that each addition takes 1 unit
of time, we are wasting 4 add operations!
50
Second algorithm (improved)
still try all possible combinations, but don't redundantly add the sums
j

key observation:
A
k i


k
j 1
 Aj   Ak
k i
in other words, we don't need to throw away partial sums
can we use this information to remove one of the loops
from our algorithm?
// implement together
function maxSubsequence2(array[]):

What is the runtime (Big-Oh) of this new algorithm?
Can it still be improved further?
51
Observations: Claim 1


To make our code even faster, we must avoid trying all
possible combinations; to do this, we must find a way
to broadly eliminate many potential combinations from
consideration.
Claim #1: A subsequence with a negative sum cannot
be the start of the maximum-sum subsequence.
52
Claim 1, continued

Claim #1, more formally:
If Ai, j is a subsequence such that
j
A
k i
k
0
,
then there is no q such that Ai,q is the maximum-sum
subsequence.

Proof: (do it together in class)

Can this help us produce a better algorithm?
53
Claim 2

Claim #2: when examining subsequences left-to-right,
for some starting index i, if Ai,j becomes the first
subsequence starting with i ,
j
such that
A
k i
k
0
Then no part of Ai,j can be part of the maximum-sum
subsequence.


(Why is this a stronger claim than Claim #1?)
Proof: (do it together in class)
54
Claim 2, continued

These figures show the possible contents of Ai,j
55
Third (best) algorithm

Can we eliminate another loop from our algorithm?
// implement together
function maxSubsequence3(array[]):


What is its runtime (Big-Oh)?
Is there an even better algorithm than this third
algorithm? Can you make a strong argument about
why or why not?
56
Types of runtime analysis




Express the running time as f(N), where N is the size of
the input
worst case: your enemy gets to pick the input
average case: need to assume a probability
distribution on the inputs
amortized: your enemy gets to pick the
inputs/operations, but you only have to guarantee
speed over a large number of operations
57