Transcript Document

ECE 453 – CS 447 – SE 465
Software Testing &
Quality Assurance
Lecture 10
Instructor
Paulo Alencar
1
Overview
Structural Testing
Introduction – General Concepts
Flow Graph Testing
DD-Paths
Test Coverage Metrics
Basis Path Testing
Guidelines and Observations
Data Flow Testing
Hybrid Methods
Retrospective on Structural Testing
2
Path Testing
• Path Testing is focusing on test techniques that are based
on the selection of test paths though a program graph. If
the set of paths is properly chosen, then we can claim that
we have achieved a measure of test thoroughness.
• The fault assumption for path testing techniques is that
something has gone wrong with the software that makes it
take a different path than the one intended.
• Structurally, a path is a sequence of statements in a
program unit. Semantically, a path is an execution instance
of the program unit. For software testing we are interested
in entry-exit paths.
3
Path Testing Process
• Input:
– Source code and a path selection criterion
• Process:
–
–
–
–
–
Generation of a CFG
Selection of Paths
Generation of Test Input Data
Feasibility Test of a Path
Evaluation of Program’s Output for the Selected Test
Cases
4
Flow Graphs – Use in determining
Paths for Testing - Revisited
V(G) = 3
1
Basis set:
1, 2, 3, 4, 6, 7
1, 2, 3, 4, 5, 4, 6, 7
1, 2, 6, 7
x = z+5
z = 4*3-y
x>z
2
t
x = z+5
z = 4*3-y
if(x > z) goto A;
for( u=0; u < x; u++) {
z = z+1;
};
A: y = z + k
6
f
R1
R2
3
5
z = z+1
u++
t
R3
u=0
4
y = z+k
f
7
u<x
Example of a simple control flowgraph.
5
Program Graph to DD-Path Graph
first
4
5
6
7
8
B
11
10
D
C
9
E
12
F
13
21
22
23
14
G
15
20
16
17
18
19
O
H
I
N
J
K
L
M
last
6
Topological Paths
• We can identify eight possible paths through the
previous diagram
1. first – A – B – C – E – F – G – O – last
2. first – A – B – D – E – F – G – O – last
3. first – A – B – C – E – F – H – I – N – O – last
4. first – A – B – D – E – F – H – I – N – O – last
5. first – A – B – C – E – F – H – J – K – M – N – O – last
6. first – A – B – D – E – F – H – J – K – M – N – O – last
7. first – A – B – C – E – F – H – J – L – M – N – O – last
8. first – A – B – D – E – F – H – J – L – M – N – O – last
7
Feasible Paths
• If we assume that the logic of the program dictates that “if
node C is traversed, then we must traverse node H, and if
node D is traversed, then we must traverse Node G”
– These constraints will eliminate Paths 1, 4, 6 and 8
– We are left to consider four feasible paths:
2. first – A – B – D – E – F – G – O – last
3. first – A – B – C – E – F – H – I – N – O – last
5. first – A – B – C – E – F – H – J – K – M – N – O – last
7. first – A – B – C – E – F – H – J – L – M – N – O – last
8
•
Let’s consider the Triangle program, its CFG, and its DD-Path
graph.
[Jorgensen: Software Testing A Craftsman’s Approach]
1. program triangle (input, output);
2. VAR a,b,c: integer;
3. IsATriangle: boolean;
4. BEGIN
5. writeln(“Enter integers: “);
6. readln(a,b,c);
7. Writeln(“Side A is: “, a, “Side B is: “, b, “Side C is: “, c);
8. IF(a <b+c) AND (b < a+c) AND (c<a+b) THEN
9.
IsATriangle = True;
10. ELSE IsATriangle = False;
11. IF IsATriangle)
12. THEN
13.
BEGIN
14.
IF (a=b) XOR (a=c) XOR (b=c) AND NOT((a=b) AND (a=c)
15.
THEN writeln(“Equilateral”);
16.
IF (a=b) AND (b=c)
17.
THEN writeln(“Equilateral”);
18.
IF(a <> b) AND (a <> c) AND (a<>c)
19.
THEN writeln(“Scalene”);
20.
END
21. ELSE Writeln(“Not a Triangle”);
22. END
first
A
A
B
P1: A-B-D-E-G-I-J-K-Last
P2: A-C-D-E-G-I-J-K-Last
P3: A-B-D-L-Last
P4: A-B-D-E-F-G-I-J-K-Last
P5: A-B-D-E-F-G-H-I-J-K-Last
P6: A-B-D-E-F-G-H-I-K-Last
E
C
D
G
F
I
H
L
K
J
9
last
Node B is traversed  D, E are traversed
Node C is traversed  D, L are traversed
Node E is traversed  F, H, J are traversed
Node F is traversed  H, J are traversed
Node H is traversed  F, J are traversed
Node J is traversed  F, I are traversed
•
1. program triangle (input, output);
2. VAR a,b,c: integer;
3. IsATriangle: boolean;
4. BEGIN
A5. writeln(“Enter integers: “);
A6. readln(a,b,c);
A7. Writeln(“Side A is: “, a, “Side B is: “, b, “Side C is: “, c);
A8. IF(a < b+c) AND (b < a+c) AND (c < a+b) THEN
B9.
IsATriangle = True;
C10. ELSE IsATriangle = False;
D11. IF IsATriangle)
D12. THEN
E13.
BEGIN
E14.
IF (a=b) XOR (a=c) XOR (b=c) AND NOT((a=b) AND (a=c)
F15.
THEN writeln(“Equilateral”);
G16.
IF (a=b) AND (b=c)
H17.
THEN writeln(“Equilateral”);
I18.
IF(a <> b) AND (a <> c) AND (a<>c)
J19.
THEN writeln(“Scalene”);
K20.
END
L21. ELSE Writeln(“Not a Triangle”);
22. END
first
A
A
B
E
C
D
G
F
I
H
L
J
last
Let’s consider the Triangle program, its CFG, and its DD-Path graph.
[Jorgensen: Software Testing A Craftsman’s Approach]
Logically feasible paths:
K
P1: A-C-D-L-Last
P2: A-B-D-E-F-G-I-K-Last
P3: A-B-D-E-G-H-I-K-Last
P4: A-B-D-E-G-I-J-K-Last
10
Structured Programming Constructs
s2
s3
s3
s2
s1
s1
s1
s1
s4
Sequence if-then-else
s2
s3
s2
s4
s5
s3
case
if-then
s1
s1
s2
s2
loop-pre-test loop-post-test
11
Violations of Structured
Programming Constructs
s1
s1
s1
s1
s0
s0
s2
s2
s3
s3
Branching into loop
Branching out of loop
s2
s2
s0
s3
Branching into decision
s3
s1
Branching out of decision
12
Essential Complexity
• Essential complexity. This measures how much
unstructured logic exists in a module (e.g., a loop
with an exiting GOTO statement)
• The essential complexity can be computed on a
program graph that has been condensed by
collapsing structured programming constructs to a
single node, until no more structured constructs
can be found
13
Condensing A Program Graph
first
first
a
a
A
A
B
b
E
C
D
d
b
H
G
F
L
I
H
L
c
c
last
K
J
last
first
a
e
first
L
d
V(G) = 1
e
last
14
Test Coverage Metrics - Revisited
Metric
Description of Coverage
C0
Every Statement
C1
Every DD-Path
C1P
Every predicate to each outcome
C2
C1 Coverage + loop coverage
Cd
C1 Coverage + every dependent pair of
DD-Paths
CMCC
Multiple condition coverage
Cik
Every program path that contains up to
k repetitions of a loop (usually k=2)
Cstat
“Statistically significant” fraction of
paths
C∞
All possible execution paths
15
Some Fundamental Path Selection
Criteria
• Exercise every statement or instruction at least once
(statement coverage) C0
• Exercise every branch and case statement, in each
direction, at least once (branch coverage) C1
• Exercise each condition in a decision (condition coverage)
• Exercise each condition in a decision with all possible
outcomes C1P
• Exercise every compound predicate outcome (MCC) CMCC
• Exercise every path from entry to exit C∞
16
Example
1
2
F
Statement Coverage C0:
T
SCPath1: 1-2-3(F)-10(F)-11-13
SCPath2: 1-2-3(T)-4(T)-5-6(T)7(T)-8-9-3(F)-10(T)-12-13
3
F
10
4
F
T
T
5
12
11
Branch or Decision Coverage C1:
F
6
T
13
F
7
T
8
9
BCPath1: 1-2-3(F)-10(F)-11-13
BCPath2: 1-2-3(T)-4(T)-5-6(T)7(T)-8-9-3(F)-10(T)-12-13
BCPath3: 1-2-3(T)-4(F)-10(F)-11-13
BCPath4: 1-2-3(T)-4(T)-5-6(F)-9-3(F)10(F)-11-13
BCPath5: 1-2-3(T)-4(T)-5-6(T)-7(F)-93(F)-10(F)-11-13
17
Condition Coverage
• Requires that each condition in a compound
decision takes on all possible outcomes at least
once
• Condition Coverage does not require that each
decision takes on all possible outcomes
• Example: if Not (A or B) then C
Setting A = True, B = False, then A = False and B = True
satisfies Condition Coverage, but statement C will
never be executed
18
Multiple Condition Coverage
• Requires that each possible combination of inputs
be tested for each decision.
• Example: “if (A and B)” requires 4 test cases:
A = True, B = True
A = True, B = False
A = False, B = True
A = False, B = False
• For n conditions, 2n test cases are needed, and this
grows exponentially with n
19
Modified MCC
• Requires that each condition be shown to independently
affect the outcome of a decision
• Generally, for n conditions, MC/DC requires only n + 1
test cases
• Example: if (A or B) then C
3 test cases:
A = True, B = False (A dominates)
A = False, B = True (B dominates)
A = False, B = False (false outcome)
20
Examples
• To test if (A or B)
A:
T
F
B:
F
T
F
F
• To test if (A and B)
A:
F
T
T
B:
T
F
T
• To test if (A xor B)
A:
T
T
F
B:
T
F
T
21
Guidelines and Observations
•
Functional testing focuses on properties that are “too far” or disassociated
from the source code being tested
•
Path testing focuses too much on the graph and not the logic of the code
•
Path testing very useful to measure the quality of the testing (coverage criteria)
•
Basis path testing is giving us a lower bound of how much testing is necessary
•
Path testing is providing a set of metrics that act as cross checks to functional
testing
– When a program path is traversed by several functional test cases we suspect
redundancy
– When we fail to attain DD-Path graph coverage, we know that there are gaps is the
functional test cases
•
Coverage metrics can be used as
– Setting minimum testing standards and as
– Mechanism to selectively test portions of the code more rigorously than others
22
Guidelines and Observations
• We can distinguish between
– Specified program behavior (S),
– Programmed behavior (feasible paths) (P), and
– Topologically feasible paths (T)
S
P
T
23
Guidelines and Observations
•
Region 1 contains specified behavior that is
implemented by feasible paths – most
desirable
•
Every feasible path is topologically possible
so the parts 2,6 of set P should be empty
•
Region 3 contains feasible paths that
corresponds to unspecified behavior. Such
extra functionality must be inspected
•
Regions 4 and 7 contain the infeasible paths.
Region 4 is problematic as it corresponds to
specified behavior that has almost been
implemented (topologically possible, yet
infeasible program paths) – coding errors
•
Region 7 refers to unspecified, infeasible, yet
topologically possible paths – potential
maintenance problem – could become
Region3
S
P
2
6
5
1
4
3
T
7
24
Software Testing Standards
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
IEEE 1008 Software Unit Testing
IEEE 1044 Classification for Software Anomalies
ISO/IEC 12207 Information Technology--Software Life Cycle Processes
ISO/IEC TR 15271 Guide for ISO/IEC 12207 (Software Life Cycle Processes)
AECL CE-1001-STD REV.1 Standard for Software Engineering of Safety Critical Software
BSI BS-7738 Specification for Information Systems Products Using SSADM
(Structured Systems Analysis and Design Method)
BSI BS-7925-1 Software Testing - Vocabulary
BSI BS-7925-2 Standard for Software Component Testing
DEF 00-55 (Part 1)/1 The Procurement of Safety Critical Software in Defence Equipment-Requirements
DIN VDE 0801 Principles for Computers in Safety-Related Systems
ESA PSS-05-01 Guide to the Software Engineering Standards, Issue 1
German Process-Model (V-Model) Software Life-Cycle Process Model
IAEA TRS-372 Development and Implementation of Computerized Operator Support Systems in Nuclear
Installations
IEC 60601-1-4 Medical Electrical Equipment--Part 1: General Requirements for Safety-4.Collateral Standard:
Programmable Electrical Medical Systems
IEC 60880 Software for Computers in the Safety Systems of Nuclear Power Stations
IEE 5 Software Inspection Handbook
NCC STARTS Purchasers' Handbook--Procuring Software-Based Systems
NRC NUREG/BR-0167 Software Quality Assurance Program and Guidelines
RTCA DO-178B/ED-12B Software Considerations in Airborne Systems and Equipment Certification
25