Transcript Slides for Rosen, 5th edition

Module #1 - Logic
The Fundamentals: Algorithms
Based on Rosen, Discrete Mathematics & Its Applications. Prepared
by (c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
1
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Overview


Algorithms (Formal procedures)
Complexity of algorithms

Analysis using order-of-growth notation.
2
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Algorithms

The foundation of computer programming.



Most generally, an algorithm just means a definite
procedure for performing some sort of task.
A computer program is simply a description of an
algorithm in a language precise enough for a
computer to understand, requiring only operations
the computer already knows how to do.
We say that a program implements (or “is an
implementation of”) its algorithm.
3
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Algorithms You Already Know

Grade school arithmetic algorithms:





How to add any two natural numbers written
in decimal on paper using carries.
Similar: Subtraction using borrowing.
Multiplication & long division.
Your favorite cooking recipe.
How to register for classes at UBC.
4
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Programming Languages

Some common programming languages:





Newer: Java, C, C++, Visual Basic,
JavaScript, Perl, Pascal
Older: Fortran, Cobol, Lisp, Basic
Assembly languages, for low-level coding.
In this class we will use an informal,
Pascal-like “pseudo-code” language.
You should know at least 1 real language!
5
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Algorithm Example (English)





Task: Given a sequence {ai}=a1,…,an,
aiN, say what its largest element is.
Set the value of a temporary variable v
(largest element seen so far) to a1’s value.
Look at the next element ai in the sequence.
If ai>v, then re-assign v to the number ai.
Repeat previous 2 steps until there are no
more elements in the sequence, & return v.
6
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Executing an Algorithm



When you start up a piece of software, we
say the program or its algorithm is being
run or executed by the computer.
Given a description of an algorithm, you
can also execute it by hand, by working
through all of its steps on paper.
Before ~WWII, “computer” meant a
person whose job was to run algorithms!
7
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Executing the Max algorithm







Let {ai}=7,12,3,15,8. Find its maximum…
Set v = a1 = 7.
Look at next element: a2 = 12.
Is a2>v? Yes, so change v to 12.
Look at next element: a2 = 3.
Is 3>12? No, leave v alone….
Is 15>12? Yes, v=15…
8
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Algorithm Characteristics

Some important features of algorithms:








Input. Information or data that comes in.
Output. Information or data that goes out.
Definiteness. Precisely defined.
Correctness. Outputs correctly relate to inputs.
Finiteness. Won’t take forever to describe or run.
Effectiveness. Individual steps are all do-able.
Generality. Works for many possible inputs.
Efficiency. Takes little time & memory to run.
9
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Our Pseudocode Language










Procedure name(argument: type)
variable := expression
informal statement
begin statements end
{comment}
if condition then statement [else statement]
for variable := initial value to final value statement
while condition statement
procname(arguments) -- call another procedure.
return expression
10
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Max procedure in pseudocode
procedure max(a1, a2, …, an: integers)
v := a1
{largest element so far}
for i := 2 to n
{go thru rest of elems}
if ai > v then v := ai {found bigger?}
{at this point v’s value is the same as
the largest integer in the list}
return v
11
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Inventing an Algorithm

Requires a lot of creativity and intuition


Like writing proofs.
We can’t give you an algorithm for
inventing algorithms.




Just look at lots of examples…
And practice (preferably, on a computer)
And look at more examples…
And practice some more… etc., etc.
12
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
An Example Task

Problem of searching an ordered list.






Given a list L of n elements that are sorted into
a definite order (e.g., numeric, alphabetical),
And given a particular element x,
Determine whether x appears in the list,
and if so, return its index (position) in the list.
This problem occurs often in many contexts.
Let’s find an efficient algorithm!
13
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Search algorithm #1: Linear Search
procedure linear search (x: integer, a1, a2,
…, an: distinct integers)
i := 1
while (i  n  x  ai)
i := i + 1
if i  n then location := i
else location := 0
return location {index or 0 if not found}
14
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Search algorithm #2: Binary Search


Assume that the elements have been sorted
in ascending order
Basic idea: On each step, look at the middle
element of the remaining list to eliminate
half of it, and quickly zero in on the desired
element.
<x
<x
<x
>x
15
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Search algorithm #2: Binary Search
procedure binary search
(x: integer, a1, a2, …, an: sorted integers)
i := 1
{left endpoint of search interval}
j := n
{right endpoint of search interval}
while i<j begin
{while interval has >1 item}
m := (i+j)/2
{midpoint}
if x>am then i := m+1 else j := m
end
if x = ai then location := i else location := 0
return location
16
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Practice exercises


Devise an algorithm that finds the sum of all
the integers in a list. [2 min]
procedure sum(a1, a2, …, an: integers)
s := 0
{sum of elems so far}
for i := 1 to n {go thru all elems}
s := s + ai
{add current item}
{at this point s is the sum of all items}
return s
17
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Review: Algorithms





Characteristics of algorithms.
Pseudocode.
Examples: Max algorithm, linear search & binary
search algorithms.
Intuitively we see that binary search is much faster
than linear search, but how do we analyze the
efficiency of algorithms formally?
Use methods of algorithmic complexity, which
utilize the order-of-growth concepts.
18
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Sorting Algorithms

Sorting is a common operation in many
applications.



It is also widely used as a subroutine in other
data-processing algorithms.
Two sorting algorithms shown in textbook:



E.g. spreadsheets and databases
Bubble sort
Insertion sort
Bubble sort will be discussed here.
19
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Bubble Sort

Smallest elements “float” up to the top of
the list, like bubbles in a container of
liquid.
30
31
1
32
33
2
34
3
30
1
31
32
2
33
3
34
1
30
31
2
32
3
33
34
1
30
2
31
3
32
33
34
1
2
30
3
31
32
33
34
1
2
3
30
31
32
33
34
20
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Algorithm: Bubble Sort
Procedure bubblesort(a1, …, an)
for i := 1 to n-1
for j := 1 to n-i
if aj >aj+1 then interchange aj and aj+1
{a1, …, an is in ascending order}
21
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Orders of Growth



For functions over numbers, we often need
to know a rough measure of how fast a
function grows.
If f(x) is faster growing than g(x), then f(x)
always eventually becomes larger than g(x)
in the limit (for large enough values of x).
Useful in engineering for showing that one
design scales better or worse than another.
22
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Orders of Growth - Motivation



Suppose you are designing a web site to
process user data (e.g., financial records).
Suppose database program A takes
fA(n)=30n+8 microseconds to process any
n records, while program B takes
fB(n)=n2+1 microseconds to process the n
records.
Which program do you choose, knowing
you’ll want to support millions of users?23

On a graph, as you go to the right, a faster
growing function eventually becomes
larger...
Value of
function

Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Visualizing Orders of Growth
fA(n)=30n+8
fB(n)=n2+1
Increasing n 
24
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Concept of order of growth




We say fA(n)=30n+8 is order n, or O(n).
It is, at most, roughly proportional to n.
fB(n)=n2+1 is order n2, or O(n2). It is
roughly proportional to n2.
Any O(n2) function is faster-growing than
any O(n) function.
For large numbers of user records, the
O(n2) function will always take more time.
25
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Definition: O(g), at most order g


Let g be any function RR.
Define “at most order g”, written O(g), to
be: {f:RR | c,k: x>k: f(x)  cg(x)}.



“Beyond some point k, function f is at most a
constant c times g (i.e., proportional to g).”
“f is at most order g”, or “f is O(g)”, or
“f=O(g)” all just mean that fO(g).
Sometimes the phrase “at most” is omitted.
26
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Points about the definition



Note that f is O(g) so long as any values of
c and k exist that satisfy the definition.
But: The particular c, k, values that make
the statement true are not unique: Any
larger value of c and/or k will also work.
You are not required to find the smallest c
and k values that work. (Indeed, in some
cases, there may be no smallest values!)
27
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
“Big-O” Proof Examples

Show that 30n+8 is O(n).

Show c,k: n>k: 30n+8  cn.


Let c=31, k=8. Assume n>k=8. Then
cn = 31n = 30n + n > 30n+8, so 30n+8 < cn.
Show that n2+1 is O(n2).

Show c,k: n>k: n2+1  cn2.

Let c=2, k=1. Assume n>1. Then cn2 = 2n2 =
n2+n2 > n2+1, or n2+1< cn2.
28
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Big-O example, graphically
cn =31n
30n+8
30n+8O(n)
n
n>k=8 
Increasing n 



Note 30n+8 isn’t less than n anywhere (n>0).
It isn’t even less than 31n everywhere.
But it is less than 31n everywhere to the right of n=8.
29
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Useful Facts about Big O




Big O, as a relation, is transitive:
fO(g)  gO(h)  fO(h)
Sums of functions:
If gO(f) and hO(f), then g+hO(f).
c>0, O(cf)=O(f+c)=O(fc)=O(f)
f1O(g1)  f2O(g2) 


f1 f2 O(g1g2)
f1+f2 O(g1+g2)
= O(max(g1,g2))
= O(g1) if g2O(g1)
(Very useful!)
30
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Orders of Growth - So Far

For any g:RR, “at most order g”,
O(g)  {f:RR | c,k x>k |f(x)|  |cg(x)|}.


Often, one deals only with positive functions
and can ignore absolute value symbols.
“fO(g)” often written “f is O(g)” or
“f=O(g)”.
31
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Definition: (g), exactly order g



If fO(g) and gO(f) then we say “g and f
are of the same order” or “f is (exactly)
order g” and write f(g).
Another equivalent definition:
(g)  {f:RR |
c1c2k x>k: |c1g(x)||f(x)||c2g(x)| }
“Everywhere beyond some point k, f(x) lies
in between two multiples of g(x).”
32
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
 example

Determine whether:

Solution:
 n ?
2
  i (n )
 i 1 
33
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Strict Ordering of Functions

Temporarily let’s write fg to mean fo(g),
f~g to mean f(g)
f ( x)
f  g  lim
 0.
x  g ( x )

Note that

Let k>1. Then the following are true:
1  log log n  log n ~ logk n  logk n
 n1/k  n  n log n  nk  kn  n!  nn …
34
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Algorithmic Complexity



The algorithmic complexity of a
computation is some measure of how
difficult it is to perform the computation.
Measures some aspect of cost of
computation (in a general sense of cost).
Common complexity measures:


“Time” complexity: # of ops or steps required
“Space” complexity: # of memory bits
required
35
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Complexity Depends on Input



Most algorithms have different
complexities for inputs of different sizes.
(E.g. searching a long list takes more time
than searching a short one.)
Therefore, complexity is usually expressed
as a function of input length.
This function usually gives the complexity
for the worst-case input of any given
length.
36
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Example 1: Max algorithm

Problem: Find the simplest form of the
exact order of growth () of the worst-case
time complexity (w.c.t.c.) of the max
algorithm, assuming that each line of code
takes some constant time every time it is
executed (with possibly different times for
different lines of code).
37
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Complexity analysis of max
procedure max(a1, a2, …, an: integers)
Times for each execution of each line.
v := a1
t1
for i := 2 to n
t2
if ai > v then v := ai
t3
return v
t4
What’s an expression for the exact total worst-case
time? (Not its order of growth.)
 w.c.t.c.:
 n

t (n)  t1    (t2  t3 )   t4
38
 i 2

Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Complexity analysis of max

Now, what is the simplest form of the exact ()
order of growth of t(n)?
 n

t (n)  t1    (t 2  t3 )   t 4
 i 2

 n

 (1)    (1)   (1)  (1)  (n  1)(1)
 i 2

 (1)  (n)(1)  (1)  (n)  (n)
39
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Example 2: Linear Search
procedure linear search (x: integer, a1, a2,
…, an: distinct integers)
i := 1
while (i  n  x  ai)
i := i + 1
if i  n then location := i
else location := 0
return location
t1
t2
t3
t4
t5
t6
40
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Linear search analysis


Worst case time complexity order:
 n

t (n)  t1    (t2  t3 )   t4  t5  t6  (n)
 i 1

Best case time complexity order:
t (n)  t1  t2  t4  t6  (1)

Average case, if item is present:
 n/2

t (n)  t1    (t2  t3 )   t4  t5  t6  (n)
 i 1

41
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Example 3: Binary Search
procedure binary search (x:integer, a1, a2, …,
an: sorted integers)
i := 1
j := n
while i<j begin
m := (i+j)/2
if x>am then i := m+1 else j := m
end
if x = ai then location := i else location := 0
return location
42
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Binary search analysis






Suppose n=2k.
Original range from i=1 to j=n contains n
elements.
Each iteration: Size ji+1 of range is cut in half.
Loop terminates when size of range is 1=20 (i=j).
Therefore, number of iterations is k = log2n
= (log2 n)= (log n)
Even for n2k (not an integral power of 2),
time complexity is still (log2 n) = (log n).
43
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Names for some orders of growth







(1)
(logc n)
(logc n)
(n)
(nc)
(cn), c>1
(n!)
Constant
Logarithmic (same order c)
Polylogarithmic
Linear
Polynomial
Exponential
Factorial
44
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Problem Complexity


The complexity of a computational problem
or task is (the order of growth of) the
complexity of the algorithm with the lowest
order of growth of complexity for solving
that problem or performing that task.
E.g. the problem of searching an ordered list
has at most logarithmic time complexity.
(Complexity is O(log n).)
45
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Tractable vs. intractable


A problem or algorithm with at most
polynomial time complexity is considered
tractable (or feasible). P is the set of all
tractable problems.
A problem or algorithm that has more than
polynomial complexity is considered
intractable (or infeasible).
46
Based on Rosen, Discrete Mathematics & Its Applications. Prepared by
(c)2001-2004, Michael P. Frank. Modified By Mingwu Chen
Things to Know


Definitions of algorithmic complexity, time
complexity, worst-case complexity; names
of orders of growth of complexity.
How to analyze the worst case, best case,
or average case order of growth of time
complexity for simple algorithms.
47