Transcript Hard Real-Time Test Tools - Concepts and Implementation

Integrated Automated Test Case
Generation and Static Analysis
Prof. Dr. Jan Peleska
Centre for Computing Technologies, University of Bremen, Germany
Dr. Ing. Cornelia Zahlten
Verified Systems International GmbH, Bremen, Germany
QA&TEST 2007 – 17—19 October – Bilbao, Spain
Jan Peleska, Cornelia Zahlten
Motivation ...
... for an integrated approach to automated module
testing and static analysis:
 Verification specialist’s perspective:
 static analysis gives insight with respect to useful test
cases and expected module behaviour
 Module testing can help to verify or falsify fault hypotheses
found in static analysis
 Tool builder’s perspective:
 Algorithms needed for automated test case generation are
also useful for automated static analysis and vice versa
Tool capabilities required are illustrated by means of
RT-Tester, developed by Verified Systems
Jan Peleska, Cornelia Zahlten
2007-10-18
Overview
1. The objectives of module testing + static analysis
2. The tool capabilities required
3. Fundamental techniques
 Structural test case generation enables functional
test case generation
 Tool architecture
 Static analysis by abstract interpretation
 Test data generation
4. Optional integration of techniques into model-
driven development cycle
Jan Peleska, Cornelia Zahlten
2007-10-18
Objectives of Module Testing
 Given a functional specification of the required
module behaviour, find test cases such that
 every aspect of functional requirements is checked
at least once (functional testing)
 the module’s code is fully covered according to the
applicable coverage requirements (structural
testing), while again checking compliance with
functional requirements
 the module is robust against illegal inputs, fulfils
performance requirements, ...
(non-functional tests)
Jan Peleska, Cornelia Zahlten
2007-10-18
Objectives of Static Analysis
 Verify the module’s conformance with non-
functional requirements:
 Absence of run-time errors, e.g.:







Correctness of memory access (array boundaries, pointer
utilisation, string handling, memory copies)
Well-definedness of arithmetic operations
Absence of unintended endless loops
Absence of unreachable code
Module complexity is acceptable (SW metrics ...)
Module code conforms to coding standards
...
Jan Peleska, Cornelia Zahlten
2007-10-18
The Tool Capabilities Required
 Capability 1 – Module specification support
 Capability 2 – Stub specification support
 Capability 3 – Specification support for Module



internal assertions
Capability 4 – Automated test data generation for
structural coverage
Capability 5 – Automated test data generation for
functional coverage
Capability 6 – Run-time error detection
Capability 7 – Advanced debugging support
Jan Peleska, Cornelia Zahlten
2007-10-18
Side Remark: The Objectives of Automation
The tool capabilities listed above allow verification
specialists to focus on their most important tasks:
 Specify expected module behaviour
 Specify meaningful conditions about the
environment behaviour – for module testing, this is
expressed by preconditions and stub
specifications
 Specify module-internal assertions
 Verify / falsify potential failures
 Identify error locations in module code
Jan Peleska, Cornelia Zahlten
2007-10-18
Capability 1 – Module specification support (1)
 The RT-Tester tool allows module specification by
means of
 Preconditions: Logical conditions about the legal
input parameters, global variable / object attribute
values (pre-states) to be met when calling the
function / method
 Postconditions: Logical conditions about the return
values, output parameters and resulting global
variable / object attribute values (post-states) on
function / method return
Jan Peleska, Cornelia Zahlten
2007-10-18
Capability 1 – Module specification support (2)
Example of Unit Under Test (UUT) specification
with RT-Tester:
double globx;
@uut double f(double x, double y, int i){
@pre: 0 < x and x < 100 and
-10 < y and exp(y) < x and
0 <= i and i <= 10;
@post: @rttAssert( globx == globx@pre );
if (-10 < y and exp(y) < x )
{ @rttAssert( f == 1/(x-exp(y)) );}
else { @rttAssert( f < 0 ); }
};
Jan Peleska, Cornelia Zahlten
2007-10-18
Capability 1 – Module specification support (3)
Keywords for module specification, pre- / post conditions
double globx;
@uut double f(double x, double y, int i){
Conditions may contain arbitrary
parameter/variable relations
@pre: … and exp(y) < x … ;
References to variable pre-state
@post: @rttAssert( globx == globx@pre );
Checked only
if condition
-10 < y and …
holds
};
if (-10 < y and exp(y) < x )
{ @rttAssert( f == 1/(x-exp(y)) );}
else { @rttAssert( f < 0 ); }
Checked only if not(-10 < y and exp(y) < x)
Jan Peleska, Cornelia Zahlten
2007-10-18
Capability 2 – Stub specification support (1)
Suppose UUT f() calls sub-function double g(double *w):
Keyword for input/output
Keyword for stub declaration
reference parameter
@stub double g(@inout double *w){
Condition to be checked
@assert: w and *w > 0;
whenever UUT calls g() –
signal UUT failure if violated
@constraint: g > (*w)@pre and
(*w)@pre < *w;
};
Test data generator only
generates data satisfying
constraint
Jan Peleska, Cornelia Zahlten
2007-10-18
Capability 2 – Stub specification support (2)
Further stub specification capabilities:
 Number and sequence of stub calls performed by
UUT can be referenced in postconditions
 Pre- / poststates of stub calls can be stored and
referenced in postconditions
 Stub body may also be explicitly programmed –
test data generator takes explicit stub code into
account
Jan Peleska, Cornelia Zahlten
2007-10-18
Fundamental Techniques: Tool Architecture
Memory model allows
evaluation of array /
pointer handling
Selection of paths through the
program / specification model
(1) C/C++ interpretation
semantics
(2) Abstract interpretation
semantics
Internal encoding of programs and
specifications: Hierarchic Hybrid Transition
Systems
Solution of path constraints
results in concrete test data
Jan Peleska, Cornelia Zahlten
2007-10-18
Fundamental Techniques: Structural Test Case Generation
Enables Functional Test Case Generation
 Consider UUT f(…)
 Assume f(…) is to be tested against precondition
P(v)
 Assume expected results are specified by
postcondition
Q(v,v@pre) ≡
(C_1(v,v@pre) → Q_1(v,v@pre))
and … and (C_k(v,v@pre) → Q_k(v,v@pre))
Jan Peleska, Cornelia Zahlten
2007-10-18
Structural coverage of augmented function f_aug()
results in functional coverage of f():
1. void f_aug(…) {
2.
if ( P(v) ) { // Test data meets precondition
3.
f(…); // UUT is invoked
4.
if ( C_1(v,v@pre) ) {
5.
// First functional feature has been tested
6.
assert(Q_1(v,v@pre));}
7.
if ( C_2(v,v@pre) ) …
8.
}
9.
else {
10.
f(…); } // Robustness test
11.}
Jan Peleska, Cornelia Zahlten
2007-10-18
Static Analysis by Abstract Interpretation (1)
Static analysis derives program properties from
program abstractions:
 Suppose you wish to analyse C/C++function/method f(…) and prove property P
about f().
 Instead of proving P directly, using C/C++
operational semantics and analysing all possible
execution states of f(), we analyse an abstracted
function A(f) of f() and an abstracted property
A(P) such that

(A(f) satisfies A(P)) → (f() satisfies P)
Jan Peleska, Cornelia Zahlten
2007-10-18
Static Analysis by Abstract Interpretation (2)
Static analysis derives program properties from
program abstractions:
 If property
(A(f) satisfies A(P)) → (f() satisfies P)
holds then A(f) is called an
abstract interpretation of f().
 For static analysis of functions/methods, A(f)
often
 keeps the same control structures as f()
 but operates on abstracted program variables
Jan Peleska, Cornelia Zahlten
2007-10-18
Static Analysis by Abstract Interpretation (3)
 Using abstract interpretations speeds up the
analysis process, but false alarms may occur,
that is


(A(f) satisfies A(P)) does NOT hold, but
(f() satisfies P) holds
 Therefore, if abstract input data A(x) can be
found such that
A(f)(A(x)) runs into an abstract state satisfying
not(A(P))
we look for concrete input data x such that f(x)
runs into a state satisfying not(P)
Jan Peleska, Cornelia Zahlten
2007-10-18
Abstract Interpretation Example:
Interval Analysis
Interval analysis uses interval ranges for variable
valuations instead of concrete values:
 Instead of investigating concrete input values x
and calculating concrete results, say, y = f(x), we
use
 input interval ranges A(x) = [x0 ,x1 ] and calculate
 output interval ranges A(y) = [y0, y1 ] such that
For all x in [x0 ,x1 ] : f(x) in [y0, y1 ]
Jan Peleska, Cornelia Zahlten
2007-10-18
Abstract interpretation example:
Interval Analysis (continued)
 All operations x ω y on program variables x,y
can be abstracted to corresponding interval
analysis operations I [ω] J on intervals I, J by
setting
 I [ω] J = [ a0 , a1 ] with


a0 = Infimum({ a ω b | a in I and b in J })
a1 = Supremum({ a ω b | a in I and b in J })
Jan Peleska, Cornelia Zahlten
2007-10-18
Abstract interpretation example:
Interval Analysis (continued)
 For many basic operations this leads to simple
and easy-to-implement interval abstractions, e.g.

[x1 , x2 ] [+] [y1 , y2 ] = [x1 + y1 , x2 + y2 ]
 Abstract interpretation by interval analysis gives
rise to 3-valued logic: For example,

[x1 , x2 ] [<] [y1 , y2 ]
= true for x2 < y1
= false for y2 ≤ x1
= undecided otherwise: x1 < y2 and y1 ≤ x2
Jan Peleska, Cornelia Zahlten
2007-10-18
C-Sample-Function f()
1. double globx, globy;
2. double f(double x, double y, int i) {
3.
double z;
4.
int j, k, error0 = 0;
5.
if ( i < 0 )
6.
k = 0;
7.
else
8.
k = i;
9.
if ( x < 5 and y < exp(x) )
10.
x = x - y - globy;
11. else
12.
x = y - y - globx;
Jan Peleska, Cornelia Zahlten
2007-10-18
C-Sample-Function (continued)
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
for ( k = k + 1;
k < x and error0 == 0;
k = k*2)
{ if ( k < 0 )
error0 = 1; } // reachable ?
if ( error0 == 0 )
z = log((double)k-x);
else
z = 0;
return z;}
Jan Peleska, Cornelia Zahlten
2007-10-18
Example 1: Unreachable code in line 17 ?
Assume precondition x,y,globx,globy,i in [-10,10]
Perform abstract interpretation by interval analysis with input
value abstraction A(x) = A(y) = ... = A(i) = [-10,10] :
double f(double x, double y, int i) {
// A(x) = A(y) = ... = A(i) = [-10,10]
double z;
// A(z) = [-∞,+∞], since stack values are undefined
int j, k, error0 = 0;
// A(j) = A(k) = [-∞,+∞], A(error0) = [0,0]
if ( i < 0 ) k = 0; else k = i;
// A(k) = [-10,10], because A(i)[<]0 = undecided
// Result can be improved to A(k) = [0,10],
// because k = i; is only assigned for 0 <= i
Jan Peleska, Cornelia Zahlten
2007-10-18
Example 1: Unreachable code in line 17 ?
For loops, calculate interval interpretation fixpoint
// A(k) = [0,10], A(error0) = [0,0], A(x) = [-30,30]
for ( k = k + 1;
k < x and error0 == 0;
k = k*2)
{ if ( k < 0 )
error0 = 1; } // line 17
// Loop interpretation fixpoint
// A(k) = [1,58], A(error0) = [0,0], A(x) = [-30,30]
As a consequence, line 17 is unreachable  Static analysis
provides information for structural coverage test data
generation process
Jan Peleska, Cornelia Zahlten
2007-10-18
Example 2: Structural coverage test data generation
Suppose we wish to cover if-branches line 6 and line 10, and skip
the for-loop in lines 13 –17 in the C-sample function f()
introduced above – with precondition as given above. This
results in the following constraints to be solved:
x,y,globx,globy,i in [-10,10] // Precondition
and (i < 0)
// if-condition line 5 true
and (x < 5.0)
// if-condition line 9 true, 1st conjunct
and (y < exp(x))
// if-condition line 9 true, 2nd conjunct
and (1 >= x – y – globy)
// loop condition line 14, k < x, false
which evaluates to true if
A(i) = [-10,-1] and (A(x) subset [-10,5[) and
(A(y), A(globx), A(globy) subset [-10,10]) and
(A(y) [<] [exp](A(x))) and ([1,1] [≥] A(x) [-] A(y) [-] A(globy)) is true
Jan Peleska, Cornelia Zahlten
2007-10-18
Example 2: Structural coverage test data generation
Solution technique for interval constraints above:
1. Start with initial interval valuations:
A(i) = [-10,-1] and
(A(x) = [-10,5[ ) and
(A(y), A(globx), A(globy) = [-10,10])
2. If one conjunct evaluates to false, no solution exists, and
the path is infeasible, i.e. cannot be covered
3. If all conjuncts evaluate to true, interval solution has been
found, select any x in A(x), y in A(y), ..., i in A(i): These
inputs will lead to coverage of the desired path
4. If one conjunct evaluates to undecided, perform bipartitioning on interval with largest diameter
Jan Peleska, Cornelia Zahlten
2007-10-18
Example 2: Structural coverage test data generation
Bi-partitioning example: Suppose (A(y) [<] [exp](A(x)))
evaluates to undecided, and that
diameter(A(y)) < diameter(A(x))


Define A1(x) = [Inf(A(x)),(Inf(A(x))+Sup(A(x)))/2]
A2(x) = [(Inf(A(x))+Sup(A(x)))/2,Sup(A(x))]
Re-perform steps 2 – 4 above with two possible solution
candidates



A1(x),A(y),A(globx),A(globy),A(i)
A2(x),A(y),A(globx),A(globy),A(i)
Use forward-backward constraint propagation to avoid too
many bi-partitioning steps
Jan Peleska, Cornelia Zahlten
2007-10-18
Example 2: Structural coverage test data generation




Test data generator uses the same abstract interpretation by
interval analysis technique as static analysis
Static analysis applies over approximation: Possible concrete
executions are a subset of possible executions identified by
static analysis  fast, but may lead to false alarms
Test data generation applies under approximation: Every tuple
(x,y,...) selected from solution intervals A(x), A(y),... covers the
desired path  slower, but guaranteed reachability
As a consequence, the test data generator can be used to verify
potential errors indentified by static analysis, by means of
constructing concrete inputs (x,y,...) leading to the error situation
Jan Peleska, Cornelia Zahlten
2007-10-18
Example 3: Test Case Generator Supports Static Analysis
Consider modified for-loop in example above:
1. // A(k) = [0,10], A(error0) = [0,0], A(x) = [-30,30]
2. int kOld = k;
// A(kOld) = [0,10]
3. for ( k = k + 1;
4.
k < x and error0 == 0;
5.
k = k*2) {
6.
if ( kOld > k ) // (A(kOld) [>] A(k)) = [0,1]
7.
error0 = 1;
//
= undecided
8.
else
9.
kOld = k;
10.
}
11. // A(k) = [1,58], A(kOld) = [1,29]
12. // A(error0) = [0,1], A(x) = [-30,30]
Jan Peleska, Cornelia Zahlten
2007-10-18
Example 3: Test Case Generator Supports Static Analysis


Conventional static analysis tool might stop with false alarm
“error0 = 1 (line 7) potentially reachable”
Path selector, constraint generator and solver can falsify this
result as follows:






Path selector unwinds loop for n = 0,1,2, ... cycles
Constraint generator collects conditions to be fulfilled in order to
stimulate n loop cycles
Solver establishes that at most n = 5 cycles are possible for initial
valuations
A(k) = [0,10], A(x) = [-30,30]
at loop entry
Solver establishes that if-condition (kOld > k) cannot be fulfilled
for n ≤ 5.
Test case generation can confirm that “error0 = 1 (line 7) is
reachable” if A(i) = [230, 230], A(x) = [231,∞] at program start
Jan Peleska, Cornelia Zahlten
2007-10-18
Integration of Techniques Into Model-Based
Development Cycle (1)
 In model-driven development code is generated
from structural and functional models, for
example in UML:
 Composite structure diagrams, class diagrams for
SW architecture
 Statecharts for reactive behaviour
 Method specifications for transformational
behaviour
 Transformational behaviour is typically coded
directly in the target programming language
Jan Peleska, Cornelia Zahlten
2007-10-18
Model-Based Development ... (2)
 The techniques described in this talk are applied
to the test+analysis of these hand-coded
methods
 RT-Tester also performs automated test case
generation from UML-Statecharts:
 Test cases are paths through the Statechart
 RT-Tester calculates the necessary input data to
be passed to the system under test whenever an
input method is invoked
 Intermediate model representation allows to use
the same techniques as sketched for module
testing
Jan Peleska, Cornelia Zahlten
2007-10-18
Conclusion (1)
1. We have described the main objectives of
modules testing and static analysis
2. The utilisation of combined testing+analysis has
been illustrated, using the RT-Tester tool as an
example
3. The fundamental techniques for implementing the
capabilities described have been sketched
Jan Peleska, Cornelia Zahlten
2007-10-18
Conclusion (2)
4. The integration of the methods and techniques
described in this presentation into the modeldriven development process have been sketched
5. We expect that a variety of tools supporting
testing+analysis will become available in the near
future
6. All features are currently applied and evaluated in
verification and test projects for railway control
and avionic systems
Jan Peleska, Cornelia Zahlten
2007-10-18
Conclusion (3)
7. Road map for RT-Tester:
 Available NOW:
 Module testing support
 Test case generation from UML Statecharts – only in
combination with case tool Borland Together
 January 2008: Static analysis and light-weight
functional verification available
 March 2008: Full model-based testing support
available – support for various case tools
Acknowledgements: This work has been supported by BIG
Bremer Investitions-Gesellschaft under research grant
2INNO1015B
Jan Peleska, Cornelia Zahlten
2007-10-18