Transcript Qualitative and quantitative validation of software systems

Qualitative and Quantitative
Analysis of Software Systems:
a Model-based View on Integrated
Analysis
Ed Brinksma
University of Twente, Netherlands
ISSTA/Wosp
Rome, July 24th, 2002
Formal methods & Tools
MOTIVATION
Central Issue
Can the qualitative and quantitative aspects of
reactive systems be modelled and analysed within
one compositional framework?




July 24, 2002
increasing importance of quantitative behaviour
need for integrated design disciplines
cross-fertilization
theory of approximate correctness
ISSTA/Wosp, Rome
2
Contents
 Analysis, validation, and modelling
 Design, composition, and algebra
 Model-based test generation
 Integrated performance analysis
 Conclusions & perspective
July 24, 2002
ISSTA/Wosp, Rome
3
Modelling
 science vs. engineering
 software programs as models
 models of software systems
 the role of experimentation
July 24, 2002
ISSTA/Wosp, Rome
4
The Empirical Cycle
Physical world
Formal world
induction
analyse
experiment
model
phenomenon
validation
July 24, 2002
ISSTA/Wosp, Rome
5
Methodology of Science
(Vienna Circle: Carnap et al.)
 a normative procedure:
when is a theory is acceptable?
 models are refined by iteration
 models play a descriptive role
July 24, 2002
ISSTA/Wosp, Rome
6
Engineering
 models (specs) are prescriptive
 deals with artefacts
July 24, 2002
ISSTA/Wosp, Rome
7
The Design Cycle
Physical world
Formal world
realization
analyse
experiment
model
artefact
validation
July 24, 2002
ISSTA/Wosp, Rome
8
Methodology of Engineering
 design cycle is normative, a posteriori
 artefact is refined by iteration
 can employ theory of underlying
empirical cycles
July 24, 2002
ISSTA/Wosp, Rome
9
The formal nature of
software
Physical world
Formal world
compilation
implementation
specification
program
validation
verification
July 24, 2002
behaviour
ISSTA/Wosp, Rome
10
Programming as mathematics
(Dijkstra, Hoare et al.)
 programs are formal models of their
realizations.
 program derivation and verification are
mathematical activities, subject to proofs
 validation can be achieved through reliable
compilation/interpretation
July 24, 2002
ISSTA/Wosp, Rome
11
Limits of the Mathematical
Paradigm
Physical world
Formal world
realization
implementation
incomplete
combinatorial
explosion
specification
verification
July 24, 2002
large &
incomplete
software
system
ISSTA/Wosp, Rome
unreliable
validation
physical
system
12
System Modelling
Physical world
Formal world
validation
verification
specification
fragment
July 24, 2002
validation
system
model
validation
software
system
ISSTA/Wosp, Rome
physical
system
13
System Modelling
 Abstract models of software
 Interpolation specs and programs
 Basis for verification
 Basis for validation
like traditional engineering models
July 24, 2002
ISSTA/Wosp, Rome
14
System Model Validation
 Top-down: specification refinement
 Bottom-up: implementation abstraction
 Formally: by proof or by construction
 Empirically: by experimentation
July 24, 2002
ISSTA/Wosp, Rome
15
Empirical Validation
Non-formal world
Formal world
induction
analyse
system description
system design
physical system
system
model
validation
query
experimentation
& modification
July 24, 2002
ISSTA/Wosp, Rome
16
Contents
 Analysis, validation, and modelling
 Design, composition, and algebra
 Model-based test generation
 Integrated performance analysis
 Conclusions & perspective
July 24, 2002
ISSTA/Wosp, Rome
17
Design and Complexity
 Divide and conquer: distribute functionality
 Hierarchical design: different abstraction levels
Modelling formalism must support
compositionality and abstraction
July 24, 2002
ISSTA/Wosp, Rome
18
Navigating the Design Graph
Abstraction
level
Modelling formalism must
support model transformation
Alternatives
July 24, 2002
ISSTA/Wosp, Rome
19
Process Algebra
abstract process modelling
compositionality
abstraction
transformation laws
operational interpretation
July 24, 2002
ISSTA/Wosp, Rome
20
Basic Process Algebraic
Operators
inaction:
 action-prefix:
 choice:
 composition:
 hiding:
 definition:
 application:

July 24, 2002
ISSTA/Wosp, Rome
0
a.B , .B
B + C , I Bi
B ||AC
B\A
p =def B
p
21
A very basic example
A simple one place buffer
in
in
out
out
Buf
Buf =def in . out . Buf
Two instances of this buffer
in
Buf
mid
out
Buf
hide mid in
in
Buf[out/mid]
||
|[mid]|
{mid}
out
Buf[in/mid]
mid

out
in
July 24, 2002
ISSTA/Wosp, Rome
22
A very basic example II
A two place buffer
in
Buf2
out
Buf2 =def in.Half
Half =def in.Full + out.Buf2
Full =def out.Half
July 24, 2002
ISSTA/Wosp, Rome
out
out
in
in
23
Equivalence
Two ways to represent a two place buffer:
in
Buf2
out
 by enumerating the
detailled behaviour
in
Buf
mid
Buf
out
 by coupling two one
place buffers
in
out
in
out
out

out
in
in
examples for the need to study equivalences
July 24, 2002
ISSTA/Wosp, Rome
24
Equivalence
Process algebraic equivalences are based on different answers to
the question:
What is an observable part of the behaviour of a process ?
Various notions have been studied (see [van Glabbeek])
trace equivalences
testing equivalences
bisimulation equivalences
July 24, 2002
ISSTA/Wosp, Rome
25
Algebraic Laws
Equivalences (congruences) induce algebraic laws
B +C=C +B
(B + C) + D = B + (C + D)
B +0=B
B ||A C = C ||A B
B +B=B
(B ||A C) ||A D = B ||A (C ||A D)
July 24, 2002
ISSTA/Wosp, Rome
26
Expansion Laws
Parallelism can be removed step by step:
Let B = k ak .Bk and C = l cl .Cl
B
||A
C = {ak .(Bk ||A C ) | ak A } +
{cl .(B ||A Cl ) | cl A } +
{d .(Bk ||A Cl ) | d = ak =cl A }
Example:
a.0|| c.0 = a.c.0 + c.a.0
July 24, 2002
ISSTA/Wosp, Rome
a
c
c
a
27
Expansion Laws
These laws are crucial for the success
of process algebraic modelling
 move between abstraction levels
 (de)compose functionality
 stepwise simulation
July 24, 2002
ISSTA/Wosp, Rome
28
Contents
 Analysis, validation, and modelling
 Design, composition, and algebra
 Model-based test generation
 Integrated performance analysis
 Conclusions & perspective
July 24, 2002
ISSTA/Wosp, Rome
29
Testing With Formal Methods
formal model
correctness
relation
specification
test
generator
tests
implements
implementation
test
executor
verdict
(pass/fail)
Advantages formal approach:
 algorithmic generation of sound tests
 formal validation of tests
July 24, 2002
ISSTA/Wosp, Rome
30
Correctness Relations
synchronous communication :
 testing preorder
 conformance preorder
 refusal testing
(De Nicola & Hennessy, 1984)
(Brinksma 1987)
(Phillips 1987, Langerak 1990)
asynchronous communication :
 queue-based testing
 I/O-based testing
 repetitive quiescence
 multiple I/O
July 24, 2002
(Brinksma & Tretmans 1992)
(Phalippou,Tretmans,Segala 1993)
(Tretmans 1996)
(Heerink & Tretmans 1997)
ISSTA/Wosp, Rome
31
Asynchronous Test Contexts
output
test
engine
implementation
under test
interaction
input
formal model:
( IUT || Env)/A
compositionality
July 24, 2002
test environment
operating system
test method
communication buffers
ISSTA/Wosp, Rome
32
Formal Test Generation
Nondeterministic algorithm
Generate a test case t(S) from a transition system model with
S a set of states (initially S = {s0})
Apply the following steps recursively, nondeterministically
1. end test case
2.
supply input
supply ?a
3.
PASS
observe output
!y1 !y
2
FAIL FAIL
forbidden
outputs
!x3 !x2
!x1
allowed outputs
t(S after !x1)
t(S after !x2)
t(S after !x3)
t(S after ?a)
July 24, 2002
ISSTA/Wosp, Rome
34
Test Generation Example
Equation solver for y2=x
Model
? x (x < 0)
! -x
Note: to cope with non deterministic
behaviour, tests are not linear
traces, but trees
July 24, 2002
!9
otherwise
? x (x >= 0)
! x
Test
ISSTA/Wosp, Rome
? -3
?3
FAIL PASS
otherwise
!4
? -2
?2
FAIL PASS
PASS
35
Validity of Test Generation
For every set of tests T generated with the algorithm:
Soundness
generated test will never fail with ioco-correct implementation
I ioco S
implies
I passes T
Exhaustiveness
for each ioco-incorrect implementation a test can be
generated that detects it
I ioco S
July 24, 2002
implies
 T such that I fails T
ISSTA/Wosp, Rome
36
TorX Tool Architecture
on the fly
states
explorer
abstract
primer
driver
transition
transitions
bits
adapter
actions
IUT
bytes
Expansion laws
batch test generation
July 24, 2002
TTCN
concentrate on on-the-fly testing
ISSTA/Wosp, Rome
batch test
execution
38
TorX Application
Highway Tolling System
July 24, 2002
ISSTA/Wosp, Rome
39
Results
Test results :
Errors found during model validation (design error)
and during testing (coding error)
Automated testing :
beneficial: high volume and reliability
very flexible: adaptation and many configurations
Real-time :
How to cope with real time constraints ?
Efficient computation for on-the-fly testing ?
Lack of theory: quiescence vs. time-out
July 24, 2002
ISSTA/Wosp, Rome
40
Contents
 Analysis, validation, and modelling
 Design, composition, and algebra
 Model-based test generation
 Integrated performance analysis
 Conclusions & perspective
July 24, 2002
ISSTA/Wosp, Rome
41
Markovian Process Algebra
basic idea:
 incorporate delays that follow
exponential distributions into
process algebra
Two distinct approaches:
 associate delays to actions
 MEMORYLESS
TIPP, PEPA, EMPA, ...
1
 introduce delays as
0.9
orthogonal entities
0.8
0.7
IMC (also MLOTOS)
0.6
0.5
0.4
0.3
0.2
0.1
0
0
1
2
July 24, 2002
3
4
5
6
7
8
9
10
11
12
13
ISSTA/Wosp, Rome
14
15
42
Interactive Markov Chains







inaction:
prefix:
(). B ,
choice:
composition:    
hiding:

definition:
application:
supports phase type
July 24, 2002
ISSTA/Wosp, Rome
0
a.B,.B
B + C or I Bi
 ||AC
B
B \ A or hide A in B
p =def B
pdistributions
43
Algebraic Laws for IMC
B +C=C +B
 (B + C) + D = B + (C + D)
B +0=B
 a.B + a.B = a.B
July 24, 2002
 ().B + ().B = ( +
ISSTA/Wosp, Rome
).B
44
Interleaving Law
The interleaving law holds for IMC :
Example :
().B || ().C = ().().(B || C) + ().().(B || C)
This does not hold for
general distributions!
July 24, 2002
ISSTA/Wosp, Rome




45
A telephony system






 Original specification
developed by P. Ernberg
(SICS), further studied in the
French/Canadian Eucalyptus
project: more than 1500 lines
of LOTOS.
July 24, 2002



 Extensively verified using
state-of-the-art techniques
ISSTA/Wosp, Rome
model checking
equivalence checking
49
Performance analysis of the
telephony system

 Takes the original
specification without changes.






 Stochastic delays are incorporated


 in a compositional way, i.e. as additional constraints
imposed on the specification.
 exponential, Erlang and phase-type distributions.
 Weak bisimulation is used to factor out nondeterminism.
 State space >107 leads to a Markov Chain
of 720 states with a highly irregular structure.
July 24, 2002
ISSTA/Wosp, Rome
50
Time constraints
ringT_on
 A particular phone:
 The time it takes to
pick up the phone:


ringT_on
 The phone with
time constraints:
on
ringT_on
delay pick_phone
in

by

ringT_off












51
Analysis results
 14 different time
constraints
incorporated.
 Compositional
minimisation to avoid
state space
explosion.
 Here: two
subscribers phoning
each other.
July 24, 2002
ISSTA/Wosp, Rome
52
Tools used
 CAESAR/ALDEBARAN
 original specification,
 first minimisation steps.
 TIPPtool
 time constraints,
 final minimisations,
 numerical analysis.
July 24, 2002
ISSTA/Wosp, Rome
53
Contents
 Analysis, validation, and modelling
 Design, composition, and algebra
 Model-based test generation
 Integrated performance analysis
 Conclusions & perspective
July 24, 2002
ISSTA/Wosp, Rome
54
Conclusions
 software engineering & analysis need modelling, like
in traditional engineering
 modelling formalisms must support composition,
abstraction & transformation
 process algebra provides theoretical framework &
tool support (expansion laws)
 integrated quantitative & qualitative modelling &
analysis by embedding models in specialized
contexts (extending theory where needed)
July 24, 2002
ISSTA/Wosp, Rome
55
Analytic Contexts
model
performance
other
testing
July 24, 2002
ISSTA/Wosp, Rome
56
Perspectives
Extending specialized contexts
Existing: testing, soft & hard real-time
Future: dependability, security
Extending scope of analysis techniques
Model checking CTMCs (ETMCC tool)
Non-Markovian analysis
Hard real-time testing (STRESS = TorX + Uppaal)
Soft real-time testing
Integrated modelling and tool support
MoDest language and tools project
July 24, 2002
ISSTA/Wosp, Rome
57