Lecture 2 White-box Testing and Structural Coverage

• • • • • •

White-box Testing (aka. Glass-box or structural testing)

An error may exist at a location – Line numbers – Boolean tests – Expressions etc.

If tests don’t exercise that location then error can never be observed

So identify and exercise locations No loops – finitely many locations – good!

Loops – infinitely many locations – bad! Loops + branches – with loop depth – exponential growth in locations very bad !!!!

Structural Testing

Structural testing is the process of exercising software with test scenarios written from the source code, not from the requirements.

Usually structural testing has the goal to exercise a minimum collection of (combinations of) locations.

How many locations and combinations are enough?

Coverage

• • • • Coverage refers to the extent to which a given testing activity has satisfied its objectives. “

Enough ” testing is defined in terms of

coverage.

A major advantage of structural testing is that coverage can be easily and accurately defined.

Structural coverage measures

Structural Testing – Problems!

• • • • • What about sins of omission ? – no path to go down! Unimplemented requirements !

What about dead code – is a path possible? How to avoid redundant testing ?

What about testing the user requirements ?

How to handle combinatorial explosion of locations and their combinations?

Requirements-based testing versus Structural testing (DO-178B)

Software Requirements-based Test Generation Unit tests Integration tests HW/SW integration tests Software Requirements Coverage analysis End Software Structure Coverage analysis

Problems with requirements-based testing

• • • • A test set that meets requirements coverage is not necessarily a thorough test set Requirements may not contain a complete and accurate specification of all code behaviour Requirements may be too coarse to assure that all implemented behaviours are tested Requirements-based testing alone cannot confirm that code doesn’t include unintended functionality. Need structural testing too!

Coverage Criteria 1: Control flow

• • • • Measure the flow of control between statements and sequences of statements Examples: node coverage, edge coverage.

Mainly measured in terms of statement invocations (line numbers).

Exercise main flows of control.

Coverage Criteria 2: Data Flow

• • • Data flow criteria measure the flow of data between variable assignments (writes) and variable references (reads).

Examples: all-definitions, all-uses Exercise paths between definition of a variable and is subsequent use.

Coverage Criteria 3: Logic

• • • Analyse the influence of all Boolean variables Examples: predicate coverage, clause coverage, MCDC (FAA DO178B) Exercise Boolean variables at control points.

Control Flow Requirements

• • • • • These are structural requirements on a test suite They ignore functionality!

Easy to define using graph theory Possible to automate generation by constraint solving Easy to measure coverage !

Starting Point: a Condensation Graph

Assignments1 Cond1 Assignments2 Cond2 Assignments4 Assignments3 A node is either an assignment block or a Boolean test. Every program statement is associated with exactly one node.

Building condensation graphs

• • Boolean tests can be from: – If-then-else statements if (bexp) then .. else ..

– If statements if (bexp) … – Loops (of any kind) while (bexp) … An assignment block contains consecutive – assignments x = exp – return statements return exp – procedure/method calls myFunc(…) – expressions e.g. i++

Graph Coverage

• • A path is a sequence of nodes n 0 ,…, n k (condensation) graph G , such that each in a adjacent node pair, (n i , n i+1 ) forms an edge in G .

For control flow testing a test requirement tr(.) is a predicate on paths – Node n ( p ) iff p passes through node n – Edge e (p) iff p has edge e

Graph Coverage

Definition: Given a set TR of test requirements for a graph criterion C , A test suite T satisfies C on graph G if, and only if for every test requirement tr(.)  TR , there is at least one test t in T such that for the path p in G taken by t , tr(p) is true (i.e. p satisfies tr(.) )

Why so Formal?

• Answer: Sometimes coverage properties become very technical to define for efficiency reasons.

Structural Coverage Criteria

(Amman and Offut, Chapter 2) 2.1. Node Coverage (NC) Set TR reachable node in G .

contains each Myers “NC is so weak that it is generally considered useless” 2.2. Edge Coverage (EC) Set TR reachable path of length ≤ 1 .

contains each

x < y n 1 n 0 x >= y n 2

Node vs. Edge Coverage

p 1 = n 0 , n 1 , n 2 p 2 = n 0 , n 2 TS 1 =  p 1  satisfies node coverage Abstract test cases TS 2 =  p 1 , p 2  edge coverage satisfies

• 2.3 Edge-Pair Coverage (EPC = EC

2

) Set TR contains each reachable path of length ≤ 2 .

• Clearly we can continue this beyond 1,2 to

EC n

• • Combinatorial explosion!

Doesn’t deal with loops , which have unbounded length.

x < y n 1 w < y n 4 n 0 n 3 x >= y n 2 w >= y n 5 p 1 = n 0 , n 1 , n 3, n 4 p 2 = n 0 , n 2 , n 3, n 5 p 3 = n 0 , n 2 , n 3, n 4 p 4 = n 0 , n 1 , n 3, n 5 TS 1 =  p 1, p 2  coverage satisfies edge TS 2 =  p 1, p 2 , p 3, p 4  edge-pair coverage satisfies

Simple Paths

• • • • How to deal with code loops?

A path node.

p is simple if it has no repetitions of nodes other than (possibly) the first and last

So a simple path p may itself be a loop has no internal loops, but

Problem: there are paths.

too many simple paths, since many are just sub-paths of longer simple

Prime Paths

• • A path p i.e. p is prime iff p is a maximal simple path cannot be extended without losing simplicity.

This cuts down the number of cases to consider 2.4. Prime Path Coverage (PPC) Set TR contains each prime path in G

n 0 n 1 n 3 Prime Paths = (n 0 , n 1 , n 2 ), (n 0 ,n 1 ,n 3 ,n 4 ), (n 1 ,n 3 ,n 4 ,n 1 ), (n 3 ,n 4 ,n 1 ,n 3 ), (n 4 ,n 1 ,n 3 ,n 4 ), (n 3 ,n 4 ,n 1 ,n 2 ) n 4 n 2 P 2 P 1 = (n 0 , n 1 , n 2 ) = (n 0 , n 1 , n 3 , n 4 , n 1 , n 2 ) TS 1 =  p 1, p 3  satisfies prime path coverage P 3 = (n 0 , n 1 , n 3 , n 4 , n 1 , n 3 , n 4 , n 1 , n 2 ) TS 2 =  p 1, p 2  doesn’t satisfy prime path coverage! (why?)

Computing Prime Paths

• • • One advantage is that the set of all prime paths can be computed by a simple dynamic programming algorithm See Amman and Offut for details Then test cases can be derived manually (heuristic: start from longest paths?) or automatically.

2.7. Complete Path Coverage (CPC) Set TR all paths in G .

Infeasible if G has infinitely many paths contains 2.8. Specified Path Coverage (SPC) Set a set S of test paths, where parameter.

TR contains S is supplied as a

Example.

S contains paths that traverse every loop free path p in G and every loop in G both 0 and 1 times.

Data Flow Criteria

• • • A definition of a variable v that writes to v in memory is any statement A use of v memory.

is any statement that reads v from A path p = (n 1 , .., n k ) from a node n 1 n k is def-clear for v if for each to a node 1 < j < k node n j has no statements which write to v

Definition/Use Paths (du-paths)

• • • • A du-path w.r.t. v is a simple path p = (n 1 , .., n k ) Such that: 1. A statement in n 1 writes to v 2. Path p is def-clear for v 3. A statement in n k du(n,v) reads v = set of all du-paths wrt v starting at n du(m,n,v) = set of all du-paths wrt m and ending at n v starting at

Data flow coverage models

• • • 2.9. All-defs Coverage (ADC) For each def path set S = du(n, v) the set TR contains at least one path d in S .

2.10 All-uses Coverage (AUC) For each def pair set S = du(m,n,v) the set TR contains at least one path d in S .

2.11 All-du-paths Coverage (ADUPC) For each def-pair set S = du(m,n,v) the set TR contains every path d in S .

n 1 read v n 4 All-defs =  (n 0 , n 1 , n 3 , n 4 )  All-uses =  (n 0 , n 1 , n 3 , n 4 ) (n 0 , n 1 , n 3 , n 5 )  n 0 write v n 2 n 3 n 5 read v All-du-paths =  (n 0 , n 1 , n 3 , n 4 ) (n 0 , n 1 , n 3 , n 5 ), (n 0 , n 2 , n 3 , n 4 ), (n 0 , n 2 , n 3 , n 5 ) 

Logic Coverage

• • • • Graph and data flow coverages force execution of certain paths (branches) through code.

They don’t necessarily exercise different ways of taking the same branch .

This number of ways may be finite and coverable.

Since a Boolean condition may contain many Boolean and data ( int, float, object ) variables, we may wish to consider exercising the condition in different ways .

Clauses and Predicates

• • • • • • A clause is a Boolean valued expression with no Boolean valued sub-expression Examples p, myGuard, x==y, x<=y, x>y A predicate using &, I, !, is a Boolean combination of clauses Let P be a set of predicates For p  P , let C p be the set of all clauses in p .

For logic coverage, a test requirement predicate on data values tr is a

Logic Coverage Measures

• • 3.12 Predicate Coverage (PC) For each predicate p  P , the set TR contains two requirements: p is reached and evaluates to true , and p is reached and evaluates to false .

3.13 Clause Coverage (CC) For each predicate p  P , and each clause c  C p the set TR contains two requirements: c is reached and evaluates to true , and c is reached and evaluates to false .

Distributive vs. non-distributive

• • • • Note: this definition of PC is non-distributive , ie. We don’t take all combinations of all predicate values.

PC doesn’t necessarily achieve edge coverage!

Non-distributive PC – linear growth .

Distributive PC exponential growth .

Logic Coverage (cont)

• 3.14 Combinatorial Coverage (CoC) For each predicate p  P , the set TR contains requirements so that p clauses c  C p is reached, and the evaluate to every possible combination of truth values.

• CoC is also called multiple condition coverage (MCC) .

Active Clause Coverage

• • • • Clearly PC and CC grow linearly the Boolean predicate p in the size of CoC grows exponentially .

Can we find something more expressive but still linear in-between?

Use notion of active variables which determine value or influence the overall predicate

Determination

Given a clause determines c i p c i in a predicate changes the truth value of p.

p we say that iff the other clauses in p c i can be assigned values so that changing the truth value of 3.43 Active Clause Coverage (ACC) For each predicate p  P and each clause c  C p which determines p , the set TR contains two requirements for c : c is reached and evaluates to true , and c is reached and evaluates to false .

Ambiguity

• • • • • ACC can be seen as ambiguous .

Do the other clauses get the same assignment when c is true and c is false, or can they have different assignments?

Problem of masking , overlap between clauses Consider e.g. c  !b ( c iff not or b ) side-effects Here b cannot have the same assignment for c = true and c = false

Different Values

•

3.15 General Active Clause Coverage

(GACC)For each predicate p  P and each clause c  C p which determines p , the set TR contains two requirements for c : c is reached and evaluates to true , and c is reached and evaluates to false . The values chosen for the other clauses d  C p , d  c , need not be the same in both cases.

Same Values

3.15 Restricted Active Clause Coverage (RACC)

For each predicate p  P and each clause c  C p which determines p , the set TR contains two • requirements for c : c is reached and evaluates to true , and c is reached and evaluates to false . The values chosen for the other clauses d  C p , d  c , must be the same in both cases.

So RACC is infeasible for c in c  !b

Correlation

• • • • • • Another problem is that GACC does not imply PC. Let p be the predicate a  b Clearly a determines p (so does b ) If a = true then b = true so p = true If a = false then b = false so p = true Thus p never becomes false so PC fails!

Handling Correlation

•

3.16 Correlated Active Clause Coverage (CACC)

For each predicate p  P and each clause c  C p which determines p , the set TR requirements for c : c contains two is reached and evaluates to true , and c is reached and evaluates to false . The values chosen for the other clauses d  c , must cause p to be true C p , d  in one case and false in the other.

Now CACC implies PC

Safety Critical Testing: MCDC

• In DO-178B , the Federal Aviation Authority (FAA) has mandated a minimum level of logic coverage for level A (highest safety) avionic software.

• • • • “

Modified Condition Decision Coverage

” (MCDC) Has been some confusion about this definition “

Unique Cause MCDC

” (original definition) is RACC “

Masking MCDC

” (new definition) is CACC