Transcript Algorithms and data structures 7.11.2015. Protected by http://creativecommons.org/licenses/by-nc-sa/3.0/hr/ Creative Commons  You are free to: share — copy and redistribute the material in any medium.

Algorithms and data structures
7.11.2015.
Protected by http://creativecommons.org/licenses/by-nc-sa/3.0/hr/
Creative Commons

You are free to:
share — copy and redistribute the material in any medium or format
 adapt — remix, transform, and build upon the material


Under the following terms:



Attribution — You must give appropriate credit, provide a link to the license, and
indicate if changes were made. You may do so in any reasonable manner, but
not in any way that suggests the licensor endorses you or your use.
NonCommercial — You may not use the material for commercial purposes.
ShareAlike — If you remix, transform, or build upon the material, you must
distribute your contributions under the same license as the original.
No additional restrictions — You may not apply legal terms or technological
measures that legally restrict others from doing anything the license permits.
Notices:
You do not have to comply with the license for elements of the material in the public domain or where your use is
permitted by an applicable exception or limitation.
No warranties are given. The license may not give you all of the permissions necessary for your intended use. For
example, other rights such as publicity, privacy, or moral rights may limit how you use the material.
Text copied from http://creativecommons.org/licenses/by-nc-sa/3.0/
Algorithms and data structures, FER
7.11.2015.
2 / 36
Algorithms and complexity
Algorithms
Complexity of algorithms
7.11.2015.
Abu Ja'far Muhammad ibn Musa Al-Khwarizmi


Abu Ja'far Muhammad ibn
Musa Al-Khwarizmi (the
sources differ regarding his full
name and transcription!)
‫أبو جعفر محمد بن موسى الخوارزمي‬
Mohammad, father of Jafar, son of
Musa from Khwarezm
Born in Khwarezm, what is today
Khiva, Uzbekistan, about year 780.
Died in Baghdad about year 850.
One of the 10 most appreciated
mathematicians of all times.




Algorithms and data structures, FER
7.11.2015.
4 / 36
Achievements of al Khwarizmi


Encourages using of Hindu/Arabic numbers – introduces zero
About year 825 he wrote the book “Kitab al-jabr w'al-muqabala” in
Baghdad
 ‫الكتاب المختصر في حساب الجبر والمقابلة‬
 The Compendious Book on Calculation by Completion and Balancing
 jabr - moving to the other side of an equation
x - 2 = 12  x = 12 + 2
 muqabala - equal subtraction from both sides of an equation
– x+y=y+7x=7
 al-jabr  algebra - bone-setting
Algorithms and data structures, FER
7.11.2015.
5 / 36
From al Khwarizmi to algorithms...





He believed that any mathematical problem can be split into steps, i.e. to a
sequence of rules.
In the Latin translation of his book (12th century), in front of every rule there is:
 Dixit Algorizmi - said Al Khwarizmi

 The algorithm is as follows...
The very first understanding of the word “algorithm” refers only to calculation
rules related to numbers; later it also refers to rules for solving other tasks in
mathematics
In the 20th century, with emerging of computers, the term is extended to
computing, and later also to other fields
Rules to achieve a required goal
Algorithms and data structures, FER
7.11.2015.
6 / 36
What is an algorithm



A precisely described way to solve a problem
Uniquely defines what has to be done
Initial objects must be defined, belonging to a class of objects on
which operations can be performed
As outcome appear final objects or results
 Final number of steps; every step is described by an instruction
 Execution is an algorithmic process



Applicable, if the result can be obtained in a finite time
Examples for unallowed instructions:


Calculate 5/0
Increase x for 6 or 7
Algorithms and data structures, FER
7.11.2015.
7 / 36
Algorithm

An algorithm must be effective:


Examples:




The result should be achievable by using pencil and paper, in a finite amount time
Addition of integers is effective
Addition of real numbers is not because a number with infinite amount of digits may appear
With programming skills and with understanding the nature of the problem to be
solved, the student can write an effective algorithm
 The goal of this course is to learn how to design and program an
efficient algorithm.
Effective and efficient are not synonyms


effective = the one that brings effect, results
efficient = effective & successful (regarding the consumption of resources - time, processor,
disk, memory)
–
–

Multiplication can be performed as repetitive addition - effective but not efficient!
Solving of a large set of linear equations using the Kramer’s rule - effective but not efficient!
Wikipedia: A simple way of distinguishing between efficiency and effectiveness is the saying, "Efficiency is doing things right,
while Effectiveness is doing the right things. "
Algorithms and data structures, FER
7.11.2015.
8 / 36
Procedure

Procedure has all properties of an algorithm, except it does not
necessarily terminate after a finite number of steps


Examples for procedure:




In language C that may be a void function
Procedure
Operating system
Text editor
Algorithm
Execution time must be “reasonable”
Example:

An algorithm that would play chess choosing at each move all the possible
consequences of it, would require billions of years even on a quickest
conceivable computer
Algorithms and data structures, FER
7.11.2015.
9 / 36
Analysing algorithms



Program - description of an algorithm in a programming language that uniquely
defines what the computer should do
Programming – to learn the syntax of a procedural language and to acquire
basic intuitive skills for algorithmisation of a problem being verbally described
Algorithms + data structures = programs (Wirth)
How do devise algorithms?
 How to structure data?
 How to express the algorithms?
 How to verify the algorithm correctness?
 How to analyse algorithms?
 How to check (test) a program?
The algorithm design procedures are not unique and require creativity. Otherwise, algorithm
generators would already exist. It means that (for the time being?) the content of this course
cannot be completely algorithmised. We shall use the programming language C. For a compact
description of complex algorithms pseudo-code may be used.


Algorithms and data structures, FER
7.11.2015.
10 / 36
Effective but not efficient?
Important!
Due to a new computer program, which has slowed down our
work I beg the patients for patience!
Thanks, Your Doctor
Algorithms and data structures, FER
7.11.2015.
11 / 36
Analysis of complexity
"a priori" and "a posteriori“ analyses
O-notation
Asymptotic running time
Examples
7.11.2015.
Basics

Purpose
Intellectual amusement?
 Prediction of execution time
 Search for more efficient algorithms


Hypotheses:
Sequential single processor computer
 Fixed time (or limited by an upper bound) for retrieval of contents of a
memory location
 Time for processing of an operation (arithmetic, logical, assignment,
function call) is upper bounded by a constant

Algorithms and data structures, FER
7.11.2015.
13 / 36
“a priori” and “a posteriori” analyses

Selection of data sets for exhaustive algorithm testing:
best case scenario
 worst case scenario
 average (typical) behaviour


a priori


Duration of algorithm execution (worst case scenario) expressed as a
function of some relevant arguments (e.g. amount of data)
a posteriori

Statistics achieved by measurements on computer
Algorithms and data structures, FER
7.11.2015.
14 / 36
“a priori” analysis


Estimation of execution time, independent of computer type,
programming language, compiler...
examples:
a) x += y;
1
b) for(i = 1; i <= n; i++) {
x += y;
}
n
c) for(i = 1; i <= n; i++) {
for(j = 1; j <= n; j++) {
x += y;
}
}
n2
Algorithms and data structures, FER
7.11.2015.
15 / 36
Algortihm complexity

A tool is needed for comparison of algorithms efficiency



Application of mathematics
Execution time is observed when the count of input data becomes
sufficiently large
Example: execution of inversion of a n x n matrix, measured in
seconds is in proportion to n3



If the inversion of a 100 x 100 elements requires 1 minute,
then the inversion of a 200 x 200 matrix will take 8 minutes (23=8).
How to express the execution time as a function of n?
–
E.g. execution time grows with the matrix size as n3
Algorithms and data structures, FER
7.11.2015.
16 / 36
O - notation

f(n) = O(g(n)) if there exist two positive constants c and n0 such that is valid f(n)  cg(n)
for all n n0

We look for the minimum g(n) to satisfy the above requirement

In other words, the execution time is estimated as an order of magnitude of f(n), determined by
the quantity of data n, multiplied by some constant c

examples:

n3+5n2+77n = O(n3)

How much work is required to move 1 chair from the room A to the room B?

How much work is required to move n chairs from A to B?

How much work is required to move n chairs from A to B, whereby when each new chair is brought to
B, all the present chairs there have to be shifted, supposing that for shifting of a single chair, the
same effort is required like when bringing it from A to B?
1 + 2 + 3 + 4 + ...+ n = n(n+1)/2 = n2/2 + n/2 = O (n2)
Algorithms and data structures, FER
7.11.2015.
17 / 36
O - notation

By “a priori” analysis, the execution time O(g(n))of an algorithm is obtained

If the number of operations to be performed in an algorithm depends upon an
input argument n in the form of a polynomial of mth degree, then the execution
time for that algorithm equals O(nm).

Proof:
If the execution time follows a polynomial dependency:
A(n) = amnm + ... + a1n + a0
then it is valid:
A(n)  am nm + ... +a1n +a0
A(n)  (am + am-1/ n + ... +a1 /nm-1+a0/ nm) nm
A(n)  (am + ... +a1 +a0) nm , for every n1

with: c = am + ... +a1 +a0 and n0 = 1 the statement is proved.
Algorithms and data structures, FER
7.11.2015.
18 / 36
O - notation

The value for c can be any constant grater then am if n is big
enough:
If an algorithm consists of k segments with respective execution times:
c1nm1, c2nm2,...,cknmk
Then the total time is

c1nm1 + c2nm2 +... + cknmk
This implies that for this algorithm the execution time is equal to
O(nm), where
m = max{mi}, i =1,...,k
Algorithms and data structures, FER
7.11.2015.
19 / 36
O - notation

For any n big enough, it holds:
O(1) < O(log n) < O(n) < O(nlog n) < O(n2) < O(n3) < ...< O(2n) < O(n!)


O(1) means that the execution time is upper bounded by a constant
Other values, until the penultimate, represent polynomial execution times
–

Penultimate expression represents exponential execution time
–

Each subsequent execution time is larger for an order of magnitude
No polynomial can be its upper bound because for n big enough, this function exceeds
any polynomial
Algorithms requiring exponential time can be insolvable in a reasonable
time, regardless to the processing speed of a sequential computer
Algorithms and data structures, FER
7.11.2015.
20 / 36
 - notation

f(n) = (g(n)) if there exist positive constants c and n0
such that is valid f(n)  c g(n) for all n > n0
 We look for the maximum g(n) to satisfy the above requirement
 Lower bound for an algorithm execution time
 In no case the execution can be shorter than 
–
E.g. multiplication of two n x n matrices always requires time
(n3) , O (n3)
–
Addition of n numbers always requires the time (n) = O (n)
–
Retrieval of a number among n unsorted numbers (1)  O (n)
Algorithms and data structures, FER
7.11.2015.
21 / 36
 - notation


f(n) = (g(n)) if there exist positive constants c1, c2 and n0 such
that is valid c1g(n)  f(n)  c2g(n) for all n > n0

f and g grow equally fast for large n

The ratio of f and g is between c1 and c2

In other words, the execution times are equal for the best and the worst
case
Example:

Sort n written examinations lexicographically according to students’ names. The process
starts by finding the first one, than in the remaining set it looks for the first one, and so on.
–
The process takes the same time regardless of possible ordering of the elements:
O(n2) = (n2) =  (n2)
Algorithms and data structures, FER
7.11.2015.
22 / 36
Asymptotic execution time

f(n) ~ g(n)


lim
n 
It is pronounced: "f(n) is asymptotically equal to the function g(n)”
A more precise estimation of the execution time than by O-notation


if
f ( n)
1
g ( n)
The leading member’s order of magnitude is known but also its constant multiplier
If, for example:
f(n) = aknk + ... + a0
then:

f(n) = O(nk) and f(n) ~ aknk
Examples:


3*2x+7 logx+x ~ 3*2x
For the example from the previous slide (sorting of written examinations)
–
Due to linear decrement of the remaining set, where the first member is searched, asymptotic
duration is: ~ n2/2
Algorithms and data structures, FER
7.11.2015.
23 / 36
Asymptotic execution time

While counting repetitions in some algorithms, the summations such
as follows often appear:
for (i = 1; i <= n; i++)
for (j = 0; j < i; i++) {...}
n
 i = n(n+1)/2 = O (n2)
i=1
 n2/2
Algorithms and data structures, FER
7.11.2015.
24 / 36
Different complexities in logarithmic scale
Algorithms and data structures, FER
7.11.2015.
25 / 36
O-notation - examples
 IspisiTrazi.c (PrintSearch.c)

print


The loop is always repeated n times  (n) , O(n) ,  (n)
search

Execution time is O(n)

Lower bound is  (1)
–
–
In the best case, looked for array member is found in the first step
In the worst case, all the array members have to be examined
Algorithms and data structures, FER
7.11.2015.
26 / 36
“a posteriori” analysis

Real time necessary to execute an algorithm on a specific computer
#include <sys\timeb.h> // where it is declared
struct timeb {
time_t time; // elapsed seconds since midnight,
// 01.01.1970, UTC
unsigned short millitm; // milliseconds
short timezone; // difference to UTC, in minutes
short dstflag; // <>0 if daylight saving time is active
};
void ftime(struct timeb *timeptr);
Algorithms and data structures, FER
7.11.2015.
27 / 36
“a posteriori” analysis

For time measurement, the program should include:
struct timeb time1, time2; long durationms;
ftime (&time1);
...
ftime (&time2);
durationms = 1000 * (time2.time - time1.time) +
time2.millitm - time1.millitm;

Universal Time Co-ordinated (UTC)

The old term was Greenwich Mean Time (GMT)
Algorithms and data structures, FER
7.11.2015.
28 / 36
Exercises

For the algorithm
for (i = 0; i < n; i++) {
if (a[i] == b) {
printf ("i = %d\n", i);
break;
}
}
determine:
a) a priori execution time
b) asymptotic estimate for average execution time
c) asymptotic estimate for the best case execution time
d) asymptotic estimate for the worst case execution time
e) for what value of the variable b happens the worst case?
Algorithms and data structures, FER
7.11.2015.
29 / 36
Exercises

Write the function to calculate the sum of digits for a given natural
number N. What is the function complexity?
int sumDigits (int N) {
int sum = 0;
while (N > 0) {
sum += N % 10;
N /= 10;
}
return sum;
}
Algorithms and data structures, FER
7.11.2015.
30 / 36
Exercises
Actual duration
O
~
5n2+7n+4
3n2+700n+2
n2+7logn+40
0,1*2n+100n2
(2n+1)2
6nlogn + 10n
8logn + 1000
3n! + 2n + n10
3*5n + 6*2n + n100
n2
n2
n2
2n
n2
nlogn
logn
n!
5n
5n2
3n2
n2
0,1*2n
4n2
6nlogn
8logn
3n!
3*5n
Algorithms and data structures, FER
7.11.2015.
31 / 36
Examples of dependency of execution time on the amount of data and complexity
n
T(n) = n
T(n) = n lg(n)
T(n) = n2
T(n) = n3
T(n) = 2n
5
0.005 s
0.01 s
0.03 s
0.13 s
0.03 s
10
0.01 s
0.03 s
0.1 s
1 s
1 s
20
0.02 s
0.09 s
0.4 s
8 s
1 ms
50
0.05 s
0.28 s
2.5 s
125 s
13 days
100
0.1 s
0.66 s
10 s
1 ms
4 x 1013 years
Algorithms and data structures, FER
7.11.2015.
32 / 36
What can be solved in a given time
The size of the largest problem case that can be solved in 1 hour using...
Time complexity
currently available
computer
100 times faster
computer
1000 times faster
computer
n
N1
100N1
1000N1
n2
N2
10 N2
31.6 N2
n3
N3
4.64 N3
10 N3
2n
N4
N4+6.64
N4+9.97
3n
N5
N5+4.19
N5+6.29
Algorithms and data structures, FER
7.11.2015.
33 / 36
Relative contribution of components in the expression: n4 + n3log
n + 2 (n-1)
O ( 2n )
n
n4
n3 log n
2(n-1)
1
1
0
1
10
10 000
1000
512
17
83 521
6 046
65536
18
104 976
7 321
131072
50
6 250 000
212 372
562 949 953 421 312
100
100 000 000
2 000 000
6,3 x 10 29
Algorithms and data structures, FER
7.11.2015.
34 / 36
Influence of the value of the constant K = 1 000 000
n
Kn
K n2
K n4
K 2n
1
106
106
106
2 x 106
10
107
108
1010
1,0 x 108
100
108
1010
1014
1,2 x 1036
1 000
109
1012
1018
~ 10306
Algorithms and data structures, FER
7.11.2015.
35 / 36
What does it mean “n big enough”, i.e. n > n0
Source: Cormen, Leiserson & Rivest: Introduction to algorithms, 2/e,MIT Press, 2001
Algorithms and data structures, FER
7.11.2015.
36 / 36