#### Transcript Prime Path Coverage

### Lecture 2 White-box Testing and Structural Coverage

• • • • • •

### White-box Testing (aka. Glass-box or structural testing)

An error may exist at a location – Line numbers – Boolean tests – Expressions etc.

*If tests don’t exercise that location then error can never be observed*

So identify and exercise locations No loops – finitely many locations – good!

Loops – infinitely many locations – bad! Loops + branches – with loop depth – exponential growth in locations very bad !!!!

### Structural Testing

Structural testing is the process of exercising software with test scenarios written from the source code, not from the requirements.

Usually structural testing has the goal to exercise a minimum collection of (combinations of) locations.

How many locations and combinations are enough?

### Coverage

• • • • Coverage refers to the extent to which a given testing activity has satisfied its objectives. “

*Enough ” testing is defined in terms of *

*coverage*.

A major advantage of structural testing is that coverage can be easily and accurately defined.

Structural coverage measures

### Structural Testing – Problems!

• • • • • What about sins of omission ? – no path to go down! Unimplemented requirements !

What about dead code – is a path possible? How to avoid redundant testing ?

What about testing the user requirements ?

How to handle combinatorial explosion of locations and their combinations?

### Requirements-based testing versus Structural testing (DO-178B)

Software Requirements-based Test Generation Unit tests Integration tests HW/SW integration tests Software Requirements Coverage analysis End Software Structure Coverage analysis

### Problems with requirements-based testing

• • • • A test set that meets requirements coverage is not necessarily a thorough test set Requirements may not contain a complete and accurate specification of all code behaviour Requirements may be too coarse to assure that all implemented behaviours are tested Requirements-based testing alone cannot confirm that code doesn’t include unintended functionality. Need structural testing too!

### Coverage Criteria 1: Control flow

• • • • Measure the flow of control between statements and sequences of statements Examples: *node coverage, edge coverage*.

Mainly measured in terms of statement invocations (line numbers).

Exercise main flows of control.

### Coverage Criteria 2: Data Flow

• • • Data flow criteria measure the flow of data between variable assignments (*writes*) and variable references (*reads*).

Examples: *all-definitions, all-uses* Exercise paths between definition of a variable and is subsequent use.

### Coverage Criteria 3: Logic

• • • Analyse the influence of all Boolean variables Examples: *predicate coverage, clause * *coverage, MCDC *(FAA DO178B) Exercise Boolean variables at control points.

### Control Flow Requirements

• • • • • These are structural requirements on a test suite They ignore functionality!

Easy to define using graph theory Possible to automate generation by constraint solving Easy to measure coverage !

### Starting Point: a Condensation Graph

Assignments1 Cond1 Assignments2 Cond2 Assignments4 Assignments3 A node is either an assignment block or a Boolean test. Every program statement is associated with exactly one node.

### Building condensation graphs

• • Boolean tests can be from: – If-then-else statements if (bexp) then .. else ..

– If statements if (bexp) … – Loops (of any kind) while (bexp) … An assignment block contains consecutive – assignments x = exp – return statements return exp – procedure/method calls myFunc(…) – expressions e.g. i++

### Graph Coverage

• • A path is a sequence of nodes n 0 ,…, n k (condensation) graph G , such that each in a adjacent node pair, (n i , n i+1 ) forms an edge in G .

For control flow testing a test requirement tr(.) is a predicate on paths – Node n ( p ) iff p passes through node n – Edge e (p) iff p has edge e

### Graph Coverage

**Definition**: Given a set TR of test requirements for a graph criterion C , A test suite T satisfies C on graph G if, and only if for every test requirement tr(.) TR , there is at least one test t in T such that for the path p in G taken by t , tr(p) is true (i.e. p satisfies tr(.) )

### Why so Formal?

• **Answer: **Sometimes coverage properties become very technical to define for efficiency reasons.

### Structural Coverage Criteria

(Amman and Offut, Chapter 2) 2.1. **Node Coverage (NC) **Set TR reachable node in G .

contains each Myers “NC is so weak that it is generally considered useless” 2.2. **Edge Coverage (EC) **Set TR reachable path of length ≤ 1 .

contains each

x < y n 1 n 0 x >= y n 2

### Node vs. Edge Coverage

p 1 = n 0 , n 1 , n 2 p 2 = n 0 , n 2 TS 1 = p 1 satisfies node coverage Abstract test cases TS 2 = p 1 , p 2 edge coverage satisfies

• 2.3 **Edge-Pair Coverage (EPC = EC**

**2**

**) **Set TR contains each reachable path of length ≤ 2 .

• Clearly we can continue this beyond 1,2 to

**EC n**

• • Combinatorial explosion!

Doesn’t deal with loops , which have unbounded length.

x < y n 1 w < y n 4 n 0 n 3 x >= y n 2 w >= y n 5 p 1 = n 0 , n 1 , n 3, n 4 p 2 = n 0 , n 2 , n 3, n 5 p 3 = n 0 , n 2 , n 3, n 4 p 4 = n 0 , n 1 , n 3, n 5 TS 1 = p 1, p 2 coverage satisfies edge TS 2 = p 1, p 2 , p 3, p 4 edge-pair coverage satisfies

### Simple Paths

• • • • How to deal with code loops?

A path node.

p is simple if it has no repetitions of nodes other than (possibly) the first and last

*So a simple path p may itself be a loop has no internal loops, but *

**Problem**: there are paths.

too many simple paths, since many are just sub-paths of longer simple

### Prime Paths

• • A path p i.e. p is prime iff p is a maximal simple path cannot be extended without losing simplicity.

This cuts down the number of cases to consider 2.4. **Prime Path Coverage (PPC) **Set TR contains each prime path in G

n 0 n 1 n 3 Prime Paths = (n 0 , n 1 , n 2 ), (n 0 ,n 1 ,n 3 ,n 4 ), (n 1 ,n 3 ,n 4 ,n 1 ), (n 3 ,n 4 ,n 1 ,n 3 ), (n 4 ,n 1 ,n 3 ,n 4 ), (n 3 ,n 4 ,n 1 ,n 2 ) n 4 n 2 P 2 P 1 = (n 0 , n 1 , n 2 ) = (n 0 , n 1 , n 3 , n 4 , n 1 , n 2 ) TS 1 = p 1, p 3 satisfies prime path coverage P 3 = (n 0 , n 1 , n 3 , n 4 , n 1 , n 3 , n 4 , n 1 , n 2 ) TS 2 = p 1, p 2 doesn’t satisfy prime path coverage! (why?)

### Computing Prime Paths

• • • One advantage is that the set of all prime paths can be computed by a simple dynamic programming algorithm See Amman and Offut for details Then test cases can be derived manually (heuristic: start from longest paths?) or automatically.

**2.7. Complete Path Coverage (CPC) **Set TR all paths in G .

Infeasible if G has infinitely many paths contains **2.8. Specified Path Coverage (SPC) **Set a set S of test paths, where parameter.

TR contains S is supplied as a

**Example. **

S contains paths that traverse every loop free path p in G and every loop in G both 0 and 1 times.

### Data Flow Criteria

• • • A definition of a variable v that writes to v in memory is any statement A use of v memory.

is any statement that reads v from A path p = (n 1 , .., n k ) from a node n 1 n k is def-clear for v if for each to a node 1 < j < k node n j has no statements which write to v

### Definition/Use Paths (du-paths)

• • • • A du-path w.r.t. v is a simple path p = (n 1 , .., n k ) Such that: 1. A statement in n 1 writes to v 2. Path p is def-clear for v 3. A statement in n k du(n,v) reads v = set of all du-paths wrt v starting at n du(m,n,v) = set of all du-paths wrt m and ending at n v starting at

### Data flow coverage models

• • • **2.9. All-defs Coverage (ADC) **For each def path set S = du(n, v) the set TR contains at least one path d in S .

**2.10 All-uses Coverage (AUC) **For each def pair set S = du(m,n,v) the set TR contains at least one path d in S .

**2.11 All-du-paths Coverage (ADUPC) **For each def-pair set S = du(m,n,v) the set TR contains every path d in S .

n 1 read v n 4 All-defs = (n 0 , n 1 , n 3 , n 4 ) All-uses = (n 0 , n 1 , n 3 , n 4 ) (n 0 , n 1 , n 3 , n 5 ) n 0 write v n 2 n 3 n 5 read v All-du-paths = (n 0 , n 1 , n 3 , n 4 ) (n 0 , n 1 , n 3 , n 5 ), (n 0 , n 2 , n 3 , n 4 ), (n 0 , n 2 , n 3 , n 5 )

### Logic Coverage

• • • • Graph and data flow coverages force execution of certain paths (branches) through code.

They don’t necessarily exercise different ways of taking the same branch .

This number of ways may be finite and coverable.

Since a Boolean condition may contain many Boolean and data ( int, float, object ) variables, we may wish to consider exercising the condition in different ways .

### Clauses and Predicates

• • • • • • A clause is a Boolean valued expression with no Boolean valued sub-expression Examples p, myGuard, x==y, x<=y, x>y A predicate using &, I, !, is a Boolean combination of clauses Let P be a set of predicates For p P , let C p be the set of all clauses in p .

For logic coverage, a test requirement predicate on data values tr is a

### Logic Coverage Measures

• • **3.12 Predicate Coverage (PC) **For each predicate p P , the set TR contains two requirements: p is reached and evaluates to true , and p is reached and evaluates to false .

**3.13 Clause Coverage (CC) **For each predicate p P , and each clause c C p the set TR contains two requirements: c is reached and evaluates to true , and c is reached and evaluates to false .

### Distributive vs. non-distributive

• • • • Note: this definition of PC is non-distributive , ie. We don’t take all combinations of all predicate values.

PC doesn’t necessarily achieve edge coverage!

Non-distributive PC – linear growth .

Distributive PC exponential growth .

### Logic Coverage (cont)

• **3.14 Combinatorial Coverage (CoC) **For each predicate p P , the set TR contains requirements so that p clauses c C p is reached, and the evaluate to every possible combination of truth values.

• CoC is also called multiple condition coverage (MCC) .

### Active Clause Coverage

• • • • Clearly PC and CC grow linearly the Boolean predicate p in the size of CoC grows exponentially .

Can we find something *more expressive *but *still linear *in-between?

Use notion of active variables which determine value or influence the overall predicate

### Determination

Given a clause determines c i p c i in a predicate changes the truth value of p.

p we say that iff the other clauses in p c i can be assigned values so that changing the truth value of **3.43 Active Clause Coverage (ACC) **For each predicate p P and each clause c C p which determines p , the set TR contains two requirements for c : c is reached and evaluates to true , and c is reached and evaluates to false .

### Ambiguity

• • • • • ACC can be seen as ambiguous .

Do the other clauses get the same assignment when c is true and c is false, or can they have different assignments?

Problem of masking , overlap between clauses Consider e.g. c !b ( c iff not or b ) side-effects Here b cannot have the same assignment for c = true and c = false

### Different Values

•

**3.15 General Active Clause Coverage **

**(GACC)**For each predicate p P and each clause c C p which determines p , the set TR contains two requirements for c : c is reached and evaluates to true , and c is reached and evaluates to false . The values chosen for the other clauses d C p , d c , need not be the same in both cases.

### Same Values

**3.15 Restricted Active Clause Coverage (RACC) **

For each predicate p P and each clause c C p which determines p , the set TR contains two • requirements for c : c is reached and evaluates to true , and c is reached and evaluates to false . The values chosen for the other clauses d C p , d c , must be the same in both cases.

So RACC is infeasible for c in c !b

### Correlation

• • • • • • Another problem is that GACC does not imply PC. Let p be the predicate a b Clearly a determines p (so does b ) If a = true then b = true so p = true If a = false then b = false so p = true Thus p never becomes false so PC fails!

### Handling Correlation

•

**3.16 Correlated Active Clause Coverage (CACC)**

For each predicate p P and each clause c C p which determines p , the set TR requirements for c : c contains two is reached and evaluates to true , and c is reached and evaluates to false . The values chosen for the other clauses d c , must cause p to be true C p , d in one case and false in the other.

Now CACC implies PC

### Safety Critical Testing: MCDC

• In DO-178B , the Federal Aviation Authority (FAA) has mandated a minimum level of logic coverage for level A (highest safety) avionic software.

• • • • “

*Modified Condition Decision Coverage*

” (MCDC) Has been some confusion about this definition “

*Unique Cause MCDC*

” (original definition) is RACC “

*Masking MCDC*

” (new definition) is CACC