Transcript Testing

Lecture 2: testing
Book: Chapter 9
What is testing?
Testing is not showing that there are no errors
in the program.
 Testing cannot show that the program
performs its intended goal correctly.
So, what is testing?
Testing is the process of executing the program in
order to find errors.
A successful test is one that finds an error.

Some drawbacks of testing




There are never sufficiently many test
cases.
Testing does not find all the errors.
Testing is hard and takes a lot of time.
Testing is still a largely informal task.
Black-Box (data-driven, input-output)
testing
The testing is not based on the structure of
the program (which is unknown).
In order to ensure correctness, every
possible input needs to be tested - this is
impossible!
The goal: to maximize the number of
errors found.
White-Box testing
Is based on the internal structure of the
program.
There are several alternative criterions for
checking “enough” paths in the program.
Even checking all paths (highly impractical)
does not guarantee finding all errors
(e.g., missing paths!)
Some testing principles






A programmer should not test his/her own program.
One should test not only that the program does what it
is supposed to do, but that it does not do what it is not
supposed to.
The goal of testing is to find errors, not to show that
the program is errorless.
No amount of testing can guarantee error-free
program.
Parts of programs where a lot of errors have already
been found are a good place to look for more errors.
The goal is not to humiliate the programmer!
Inspections and Walkthroughs





Manual testing methods.
Done by a team of people.
Performed at a meeting (brainstorming).
Takes 90-120 minutes.
Can find 30%-70% of errors.
Code Inspection





Team of 3-5 people.
One is the moderator. He
distributes materials and
records the errors.
The programmer explains
the program line by line.
Questions are raised.
The program is analyzed
w.r.t. a checklist of
errors.
An example: Sieve of
Erathosthenes
2
3
5
6 - is dropped because it is divisible by 2.
7 - is added to the next additional slot.
8 - dropped (divisible by 2).
9 - dropped (divisible by 3).
...
The Sieve program
int P[101];
int V=1; int N=1; int I;
main() {
while (N<101) {
for (I=1;I<=N;I++)
if (P[I]=0) {
P[I]=V++ ;
N++ ; }
else if (V%P[I]==0)
I=N+1;
else I++}
}
Is this program correct?
Checklist for inspections
Data declaration
All variables declared?
Default values
understood?
Arrays and strings
initialized?
Variables with similar
names?
Correct initialization?

Control flow
Each loop terminates?
DO/END statements
match?

Input/output
OPEN statements correct?
Format specification
correct?
End-of-file case handled?

Walkthrough




Team of 3-5 people.
Moderator, as
before.
Secretary, records
errors.
Tester, play the role
of a computer on
some test suits on
paper and board.
Selection of test cases (for white-box
testing)
The main problem is to select a good coverage
criterion. Some options are:





Cover all paths of the program.
Execute every statement at least once.
Each decision has a true or false value at least
once.
Each condition is taking each truth value at least
once.
Check all possible combinations of conditions in
each decision.
Cover all the paths of the program
Infeasible.
Consider the flow diagram
on the left.
It corresponds to a loop.
The loop body has 5 paths.
If the loops executes 20
times there are 5^20
different paths!
May also be unbounded!
How to cover the executions?
IF (A>1)&(B=0) THEN X=X/A;
END;
IF (A=2)|(X>1) THEN X=X+1;
END;
 Choose values for A,B,X.
 Value of X may change, depending on A,B.
 What do we want to cover? Paths? Statements?
Conditions?
Edge coverage
Execute every statement at least once
By choosing
A=2,B=0,X=3
each statement will be
chosen.
The case that the tests
fail is not checked!
IF (A>1)&(B=0)
THEN X=X/A;
END;
IF (A=2)|(X>1)
THEN X=X+1;
END;
Decision coverage
Each decision has a true and false outcome
at least once.
Can be achieved using


A=3,B=0,X=3
A=2,B=1,X=1
Problem: Does not test
individual conditions.
E.g., when X>1 is
erroneous in second
decision.
IF (A>1)&(B=0)
THEN X=X/A;
END;
IF (A=2)|(X>1)
THEN X=X+1;
END;
Condition coverage
Each condition has a true and false
value at least once.
For example:


A=1,B=0,X=3
A=2,B=1,X=0
lets each condition be
true and false once.
Problem:covers only the
path where the first
test fails and the
second succeeds.
IF (A>1)&(B=0)
THEN X=X/A;
END;
IF (A=2)|(X>1)
THEN X=X+1;
END;
Multiple Condition Coverage
Test all combinations of all conditions
in each test.








A>1,B=0
A>1,B≠0
A<=1,B=0
A<=1,B≠0
A=2,X>1
A=2,X<=1
A≠2,X>1
A≠2,X<=1
IF (A>1)&(B=0)
THEN X=X/A;
END;
IF (A=2)|(X>1)
THEN X=X+1;
END;
A smaller number of cases:
A=2,B=0,X=4
 A=2,B=1,X=1
 A=1,B=0,X=2
 A=1,B=1,X=1
Note the X=4 in the first
case: it is due to the fact
that X changes before
being used!

IF (A>1)&(B=0)
THEN X=X/A;
END;
IF (A=2)|(X>1)
THEN X=X+1;
END;
How to find values for coverage?
•Put true at end of path.
•Propagate path backwards.
•On assignment, relativize
expression.
(A=2 | X>1) & ¬(A>1 & B=0)
A>1 & B=0
no
A=2 | X>1
•On “yes” edge of decision, add
decision as conjunction.
A=2 | X>1
•On “no” edge, add negation of
decision as conjunction.
no
•May need to give more specific
choice of covering condition,
which ones are true or false.
true
yes
X=X/A
yes
true
X=X+1
How to find values for coverage?
(A≠2 & X/A>1) & (A>1 & B=0)
A>1 & B=0
Need to find a satisfying
assignment:
no
yes
A≠2 & X/A>1
X=X/A
A=3, X=6, B=0
A≠2 & X>1
A=2 | X>1
no
yes
true
X=X+1
true
Test case design for black box
testing



Equivalence partition
Boundary value analysis
Cause-effect graphs
Test cases based on data-flow
analysis



Partition the program into
pieces of code with a
single entry/exit point.
For each piece find which
variables are
set/used/tested.
Various covering criteria:
 from each set to each
use/test
 From each set to some
use/test.
Equivalence partition


Goals:
 Find a small number of test cases.
 Cover as much possibilities as you can.
Try to group together inputs for which the program would
likely to behave the same.
Specification
condition
Valid equivalence
class
Invalid equivalence
class
Example: A legal variable



Begins with A-Z
Contains [A-Z0-9]
Has 1-6 characters.
Specification
condition
Valid equivalence
class
Invalid equivalence
class
Starting char
Starts A-Z 1
Starts other 2
Chars
[A-Z0-9]
Has others
Length
1-6 chars 5
3
4
0 chars, >6 chars
6
7
Equivalence partition (cont.)


Add a new test case until all valid equivalence classes
have been covered. A test case can cover multiple such
classes.
Add a new test case until all invalid equivalence class
have been covered. Each test case can cover only one
such class.
Specification
condition
Valid equivalence
class
Invalid equivalence
class



AB36P
(1,3,5)
1XY12
(2)
A17#%X (4)


(6)
VERYLONG (7)
Specification
condition
Valid equivalence
class
Invalid equivalence
class
Starting char
Starts A-Z 1
Starts other 2
Chars
[A-Z0-9]
Has others
Length
1-6 chars 5
3
4
0 chars, >6 chars
6
7
Boundary value analysis

In every element class, select values that
are closed to the boundary.


If input is within range -1.0 -- +1.0, select
values -1.001, -1.0, -0.999, 0.999, 1.0,
1.001.
If needs to read N data elements, check
with
N-1, N, N+1. Also, check with
N=0.