Transcript Software Life Cycle (I. Sommerville: Software Engineering

Software Testing
1
Software Life Cycle

Sommerville , 1992: Development efforts are
typically distributed as follows:
Specifications / Design
30% - 40%
Implementation
Testing
15%
-
25%
30%
-
50%
2
Remarks by Bill Gates
17th Annual ACM Conference on Object-Oriented Programming, Seattle,
Washington, November 8, 2002

“… When you look at a big commercial software company
like Microsoft, there's actually as much testing that
goes in as development. We have as many testers as we
have developers. Testers basically test all the time, and
developers basically are involved in the testing process
about half the time…

… We've probably changed the industry we're in. We're
not in the software industry; we're in the testing
industry, and writing the software is the thing that
keeps us busy doing all that testing.”
3
Remarks by Bill Gates (cont’d)

“…The test cases are unbelievably expensive; in
fact, there's more lines of code in the test
harness than there is in the program itself.
Often that's a ratio of about three to one.”

“… Well, one of the interesting questions is,
when you change a program, … what portion of
these test cases do you need to run?“
4
Remarks by Bill Gates (cont’d)


“We also went back and say, OK, what is the state-ofthe-art in terms of being able to prove programs, prove
that a program behaves in a certain way? …And although
a full general solution to that is in some ways
unachievable, for many very interesting properties this
idea of proof has come a long way in helping us quite
substantially just in the last year…
“We use model-checking… You describe the constraints,
such as ‘nobody should acquire the lock if they've already
acquired it’ … then it's trying to go through to determine
is there some path through the program that violates the
constraints. … The initial domain we applied this in was in
device drivers… “
5
The V-model of development
Requir ements
specification
System
specification
System
integration
test plan
Acceptance
test plan
Service
System
design
Acceptance
test
Detailed
design
Sub-system
integration
test plan
System
integration test
Module and
Implementation
unit code
and unit
testing
and tess
Sub-system
integration test
6
From V to Y: automatic code generation
...
Specification
...
...
Detailed design
Implementation
...
Automatic
Code-generation
Unit-test
7
Some popular testing categories
Black box
/ white box
Static
/ dynamic
8
Testing methodologies

Specification-based
testing


Reflects true
intention of testing
Does not
propagate errors
from previous
versions

Regression testing



Does not need a
specification
Easy to implement
Finds subtle errors
9
Black Box Testing
(Behavioral testing)
Input

System under test
Output
How shall we check the I/O relation ?



Manually (specification-based)
Table of expected results ( a simple version
of non-temporal specification-based)
Compare results to previous version
(Regression testing)
10
Input
Black Box Testing
Output
(Behavioral testing)

Testing Input-Output relationships only
 Pros



This is what the product is about.
Implementation independent.
Cons



For complicated products it is hard to identify erroneous
output.
It is hard to estimate whether the product is error-free.
Practically: Choosing input with high probability of error
detection is very difficult (e.g. division of two
numbers).
11
White Box Testing
(Operational Testing)

Testing how input becomes output (including
algorithms)

Pros




Easier to detect errors.
Enables to find better tests (direct the tests)
The only way to check coverage.
Cons


Implementation weaknesses are not necessarily those of
the product.
Code is not always available
12
Static and Dynamic Testing

Dynamic testing (Run your program)

Predefined tests





Good for Regression Testing (comparing an old version against a
new one)
Testing the product under extreme conditions (stree-testing)
Random tests
“real life” tests
Static testing (Inspect your code)



Code analyzers (e.g., tools like lint and purify)
Inspection (code review)
Proofs (by tools, or by mathematical arguments): “formal
methods” (incl. Model-checking).
13
Special Testing Methods


.
Stress Testing
A product that will work under heavy load (e.g,
on-line banking system) should be tested under
increasing load - much heavier than expected.
15
Static Testing

Code analysis







Unreachable code
Objects declared and never used
Parameters type/number mismatch
Variable used before initialization
Variable is assigned to twice, without using
the first value
Function results not used
Possible array bound violations
16
Static Testing

Code inspection
 Self - The default choice.



Subtle errors and micro-flaws may be
overlooked.
Wrong conceptions propagate to review…
Code review by others - Very efficient
!
17
One more quote…

Dijkstra:
“Testing can only prove the existence
of bugs, not their absence…”
18
… So why not try to prove correctness?




In general – it is undecidable, i.e. can’t be done.
In most cases it is possible, but with manual
assistance – the same way we would prove a math
theorem.
In some cases properties of software can be proved
automatically.
Chances for errors increase with length of text


Write short code (e.g, divide into more functions).
Provide short proofs for correctness (even if they are informal).
19
Estimate how clean is your software

Error Implantation (For measuring the effectiveness of
testing)



Introduce errors.
See how many of them are detected.
This gives us an “educated guess” about testing
quality.
20
Estimate how much of the software’s behavior is covered


Coverage is a mean to estimate how
rigorous is the testing effort
We can use coverage information in order
to guide the process of test generation
(some times even automatically)
21
Statement Coverage
Example 1
int a, b, sum;
int list1[10] = {0,1,2,3,4,5,6,7,8,9};
int list2[10] = {9,8,7,6,5,4,3,2,1,0};
cin >> a >> b;
if (a >= 0 && a <= 9)
sum = list1[a];
if (b >= 0 && b <= 9)
sum = sum + list2[b];
cout << sum << "\n";
22
Statement Coverage
Example 1
if (a >=
sum =
if (b >=
sum =

0 && a <= 9)
list1[a];
0 && b <= 9)
sum + list2[b];
Statement coverage may be achieved by __
test case(s):
23
Statement Coverage
Example 1
if (a >=
sum =
if (b >=
sum =
0 && a <= 9)
list1[a];
0 && b <= 9)
sum + list2[b];
But statement coverage may not cater for all
conditions...
 ... such as when a and b are beyond the
array size.
24
Branch Coverage
Same Example 1
if (a >=
sum =
if (b >=
sum =

0 && a <= 9)
list1[a];
0 && b <= 9)
sum + list2[b];
Branch coverage may be achieved by __ test
cases.
25
Branch coverage: Example
switch (x){
case 1: x = 1; break;
case 2: switch (y){
case 1: x = 3; break;
case 2: x = 2; break;
otherwise: x = 1; }
otherwise: 4;}

branch coverage may be achieved by __ test cases
26
Path Coverage
Same Example 2
if (a >=
sum =
else sum
if (b >=
sum =
else sum
0 && a <= 9)
list1[a];
= 0;
0 && b <= 9)
sum + list2[b];
= 0;
Path coverage may be achieved by __ test cases:
28
Subsumption Relationships

subsumes
subsumes



Path coverage
Branch coverage
Statement coverage
But can we always demand path coverage?
29
Branch Coverage
Example 3
if (a >=
sum =
else sum
if (b >=
sum =
else sum
...
if (z >=
sum =
else sum

0 && a <= 9)
list1[a];
= 0;
0 && b <= 9)
sum + list2[b];
= 0;
0 && z <= 9)
sum + list26[b];
= 0;
Branch coverage may be achieved by __ test cases
30
Path Coverage
Same Example 3
if (a >=
sum =
else sum
if (b >=
sum =
else sum
...
if (z >=
sum =
else sum

0 && a <= 9)
list1[a];
= 0;
0 && b <= 9)
sum + list2[b];
= 0;
0 && z <= 9)
sum + list26[b];
= 0;
Path coverage may be achieved by __ test cases
31
Statement Coverage
Example 4
sum = 0;
while (a >= 0 && a <= 9) {
sum = list1[a];
a = a + 1;
};
if (b >= 0 && b <= 9)
sum = list2[b];
else sum = 0;

Statement coverage may be achieved by __ test
cases:
32
Branch Coverage
Same Example 4
sum = 0;
while (a >= 0 && a <= 9) {
sum = list1[a];
a = a + 1;
};
if (b >= 0 && b <= 9)
sum = list2[b];
else sum = 0;

Branch coverage may be achieved by __ test cases:
33
Loop coverage




Skip the loop entirely
Only 1 pass through the loop
2 passes through the loop
n–1, n and n+1 passes through the loop,
where n is the maximum number of allowable
passes
34
Loop Coverage
Same Example 4
sum = 0;
while (a >= 0 && a <= 9) {
sum = list1[a];
a = a + 1;
};
if (b >= 0 && b <= 9)
sum = list2[b];
else sum = 0;

Loop coverage may be achieved by __ test cases
35
Path Coverage
Same Example 4
sum = 0;
while (a >= 0 && a <= 9) {
sum = list1[a];
a = a + 1;
};
if (b >= 0 && b <= 9)
sum = list2[b];
else sum = 0;

Path coverage may be achieved by __ test cases
36
Subsumption Relationships

subsumes
subsumes

subsumes



Path coverage
Branch coverage
Statement coverage
Path coverage
Loop coverage
But can we always demand path coverage?
37
Path Coverage
Example 5
sum = 0;
while (a >= 0 && a <= 9) {
sum = list1[a];
a = a + 1;
};
while (b >= 0 && b <= 9) {
sum = list2[a];
b = b + 1;
};
...
while (z >= 0 && z <= 9) {
sum = list26[z];
z = z + 1;
};

Path coverage may be achieved by __ test cases
38
Path Coverage is not Everything
Example 1
z = x + y;



Path coverage may be
achieved by 1 test case
x = 8, y = 0
 Cannot detect
z = x - y;
x = 2, y = 2
 Cannot detect
z = x * y;

x = 8, y = 9
 Cannot detect
z = 8 + y;


Cannot detect
z = x + 9;
We need 2 test cases:
 x = 8, y = 9
 x = 29, y = 18.
39
Path Coverage is not Everything
Example 2
int a[10];
if (b > 0)
a[b] = 1;
else …
Same path with b = 5 and b = 12 behave differently
40
Condition coverage
Example
if (b1 ||(b2 && b3))
a = 1;
else …
Every sub-expression in a predicate should be evaluated
to TRUE and FALSE

Sub-expression coverage may be achieved by __ test cases
41
Multiple condition coverage
Example
if (b1 ||(b2 && b3))
a = 1;
else …
Every Boolean combination of sub-expressions in a predicate
should be evaluated to TRUE and FALSE

Multiple condition coverage may be achieved by __ test cases
42
Condition/Branch coverage (MC / DC)
(Modified Condition/Decision Coverage)
Modified – due to some constructs not present in C / C++.
Decision coverage = Branch coverage
Union of Condition coverage and Branch coverage
This is an industry standard, introduced by Boeing
if (!b1 || b2)
a = 1;
else …

MC/DC coverage may be achieved by __ test cases
43
Specification-based testing




We will see an example of a system for specificationbased testing of real-time applications.
It is a Run-time verification tool.
The testing system is called “Logic Assurance”
It monitors the specification, and can also intervene
in the execution of the program.
44
Overview
Monitoring
control
Report state
Logic
Assurance
Event
Reporting
System
Under
Development
Report State
Environment
control
Optional
=
45
The user should:
1. Specify rules:


Formalize parts of the specification by writing
rules in the specification Language.
The rules should refer to events, the System
Under Development (SUD)'s variables, external
data etc..
46
A typical specification language: The LA Language (LAL)
Derived from Temporal Logic and C, LAL enables specification of:





Event order.
Relative frequency ("fairness").
Timing demands.
Logical and mathematical operators.
More...
47
Examples (1/4):
1. Using event names, time directives and
messages:
OPEN_DOOR follows CLOSE_DOOR after not
more than 10 sec ?: message("Open
door is late");
48
...Examples
(2/4)
2. Logical operators and functions:
when time>20: time([CLOSE_DOOR])>
time([OPEN_DOOR])+10?:
message("CLOSE_DOOR is early");
49
...Examples (3/4)
3. User-defined functions are used to enhance the
language and enable external control:
if JOB_DONE(10) then HEAT(3,5) < 30?:
REDUCE_HEAT(5);
50
...Examples (4/4)
4. Locators are used to scan the log and retrieve event
index:
[2nd SEND s.t. Time>=10, packet=5] > 0 ?:
REDUCE_HEAT(5);
51
2. Report events:

From inside the SUD by using the LA Report
Facility.


From outside the SUD by using black-box
techniques (e.g. OS events)
From the environment (Sensors, etc.)
52
The LA Report Facility:
The following directive should be inserted where the
event occurs in the program:
LA_report_event (
int
identifier,
float
time-stamp,
user additional data...
)
53
LA will:
1. Keep an event log.
2. Analyze the rules in real time* (during execution)
using the LA Analyzer.
* Off-line analysis is also possible
54
3. When a rule is violated, do one of the
following:

Report the exact place and time the
rule was violated.

Invoke a user-defined function.
55
Demo
A basic communication protocol ("Stop & Wait"):
2
2
2
3
time-out
time-out
Rules:
1. Resend packet after time-out…
2. Do not resend packet that was acknowledged…
…
56
Incoming events:
# event time
messages:
message index arguments
-- ------ ------- ---------- ------- -----------1: send 10
A
1
7, 10, 21..
2: Tout 15
A
1
8, 9 ,10..
3: send 16
B
2
20,30,21..
:
Rule 1, event 3: failed to resend packet
user screen:
‘A’ sent at 10, Ack expected at 13..
Tout for A activated at 15, must send A again..
‘B’ sent at 16, Ack expected at 19..
.
57
Advantages of real time monitoring:

Tests can be planned, maintained, expanded and
applied throughout the development process.

Problems can be detected sooner.

The product is ‘tied’ to the specification.
58
Advantages of real time monitoring:


A generic tool and methodology.
By receiving input from different sources, it
enables testing of:
 Multiprocessor systems
 Dynamic environment systems.
59
Advantages of real time monitoring:


Enables temporal tests.
Enables smarter tests by using information
from inside the program.
60