Transcript Automated Model Based Testing From Theory via Tools to

Testing with Formal Methods
A Formal Framework
Ed Brinksma
course 2004
Software Testing
Testing is:
But also:
 important
 ad-hoc, manual, error-prone
 much practiced
 hardly theory / research
 30% - 50% of project effort
 no attention in curricula
 expensive
 not cool :
“if you’re a bad programmer
you might be a tester”
 time critical
 not constructive
(but sadistic?)
Attitude is changing:
 more awareness
 more professional
© Ed Brinksma/Jan Tretmans
TT 2004, Formal Framework
2
Model Based Testing
with Formal Methods
Claims :
 Formal specifications are
 more precise,
 more complete,
 unambiguous
 Formal models can be formally verified
 Formal models allow automatic generation of tests
© Ed Brinksma/Jan Tretmans
TT 2004, Formal Framework
3
Types of Testing
Level of detail
system
integration
module
unit
security
robustness
performance
Accessibility
white box
black box
usability
reliability
functional
behaviour
Characteristics
© Ed Brinksma/Jan Tretmans
TT 2004, Formal Framework
4
Formal Methods
 Use mathematics to model
relevant parts of software
 Precise, formal semantics: no
room for ambiguity or
misinterpretation
 Allow formal validation and
reasoning about systems
 Amenable to tools: automation
© Ed Brinksma/Jan Tretmans
 Examples:










Z
Temporal Logic
First order logic
SDL Specification and
Description Language
LOTOS
Promela
Labelled Transition Systems
Finite state machine
Process algebra
......
TT 2004, Formal Framework
5
Testing & Formal Methods
A Perfect Couple ?
Formal methods:
 proving properties
 research
 sound theories
 “clean”
Testing :
 trial & error
 practice
 heuristics
 “dirty hands”
“Testing is not necessary after formal verification”
“Testing can only detect the presence of errors, not their absence”
“Formal methods are toys for boys”
“Formal methods have extreme potential - but not for my project”
© Ed Brinksma/Jan Tretmans
TT 2004, Formal Framework
6
Testing with Formal Methods
 Testing with respect to a formal specification
 Precise, formal definition of correctness :
good and unambiguous basis for testing
 Formal validation of tests
 Algorithmic derivation of tests :
tools for automatic test generation
 Allows to define measures expressing coverage
and quality of testing
© Ed Brinksma/Jan Tretmans
TT 2004, Formal Framework
7
Testing & Formal Methods
Claims :
 Combining the “mathematically clean” world of formal methods
and the “dirty hands” world of testing
 Testing and formal methods are both necessary in software
development
 Formal methods improve the testing process
 Formal testing stimulates the use of formal methods
© Ed Brinksma/Jan Tretmans
TT 2004, Formal Framework
8
Specification Based
Functional Testing
with Formal Methods
formal
model
Goal:
Testing functional behaviour
formal testing
system
implementation
© Ed Brinksma/Jan Tretmans
of black-box implementation
with respect to specification
in formal language
based on formal definition
of conformance
TT 2004, Formal Framework
9
Test Automation
Traditional test automation
= tools to execute and manage test cases
 Why not generate tests automatically ?
specification
test
TTCN
cases
pass
implementation
under test
test
tool
fail
© Ed Brinksma/Jan Tretmans
TT 2004, Formal Framework
10
Formal Testing
i ioco s
S
correctness
criterion
implementation
relation
test
generation
exhaustive
  sound

specification
i passes Ts
test suite TS
implementatio
i
n
test
execution
© Ed Brinksma/Jan Tretmans
TT 2004, Formal Framework
pass / fail
11
Formal relations
 conforms-to  IMPS  SPECS
 iut conforms-to spec: implemetation under test obeys specification
 hard: relates formal specification and real world system
 imp  MODS  SPECS
 iIUT imp spec: model of the iut obeys specification
 relates formal objects
 test hypothesis: iut conforms-to spec  iIUT imp spec
 iut passes t =def v ( EXEC ( t, iut )) = pass
 execute a test: EXEC: TESTS  IMPS  P OBS
 verdict function: v: P OBS  { pass, fail }
 iut passes Ts   t  Ts . iut passes t
© Ed Brinksma/Jan Tretmans
TT 2004, Formal Framework
12
Formal Testing : Conformance
specification
S
IUT conforms-to s
correctness
criterion
implementatio
n
IUT
s  SPECS Specification
IUT
Implementation under Test
IUT is concrete, physical object
Model the physical world
But IUT is black box ! ?
Assume that model iIUT exists
© Ed Brinksma/Jan Tretmans
TT 2004, Formal Framework
13
Formal Testing : Conformance
specification
S
formal
correctness
criterion
iIUT imp s
implementatio
iIUT
n
s  SPECS
Specification
iIUT  MODS model of IUT
Test hypothesis :
each concrete IUT can be modelled
by some iIUT  MODS
Conformance : iIUT imp s
iIUT is not known ;
testing to learn about iIUT
© Ed Brinksma/Jan Tretmans
TT 2004, Formal Framework
14
Conformance testing
 conforms-to  IMPS  SPECS
 iut passes t =def v ( EXEC ( t, iut )) = pass
 iut passes Ts   t  Ts . iut passes t
 conformance:
 complete:
 sound:
 exhaustiveness:
iut conforms-to spec  iut passes Ts
iut conforms-to spec  iut passes Ts
iut conforms-to spec  iut passes Ts
 ioco: input output conformance
© Ed Brinksma/Jan Tretmans
TT 2004, Formal Framework
15
Formal Testing
Test hypothesis :
s SPECS
imp
der : SPECS 
(TESTS)
Ts  TESTS
IUTIMPS . iIUT MODS .
tTESTS . exec(t,IUT) = obs(t,iIUT)
Proof soundness and exhaustivess:
iMODS .
( tder(s) . t(obs(t,i)) = pass )
 i imp s
pass
iIUT
IUT MODS
IMPS
obs ::
exec
TESTS 
MODS

IMPS 
(OBS)
© Ed Brinksma/Jan Tretmans
t:
OBS
(OBS)
{fail,pass}
fail
TT 2004, Formal Framework
16
Formal Testing : Test Derivation
specification
S
test
generation
Test generation :
der : SPECS   ( TESTS )
test suite TS
Test suite - set of test cases : T  TESTS
Test case :
t  TESTS
© Ed Brinksma/Jan Tretmans
TT 2004, Formal Framework
17
Formal Testing : Test Execution
Test execution leads to a set of observations :
exec : TESTS  IMPS   ( OBS )
Model of test execution :
obs : TESTS  MODS   ( OBS )
test suite T
implementatio
n
IUT
test
i
OBS
execution
IUT
© Ed Brinksma/Jan Tretmans
TT 2004, Formal Framework
18
Formal Testing : Test Hypothesis
Observational framework : TESTS, OBS, exec, obs
Test hypothesis :
 IUT  IMPS .  iIUT  MODS .
 t  TESTS . exec ( t, IUT ) = obs ( t, iIUT )
IUT
i
IUT
© Ed Brinksma/Jan Tretmans
test suite T
test
exec
exec
obs
execution
TT 2004, Formal Framework
OBS
19
Formal Testing : Verdicts
Observations are interpreted :
t :  (OBS)  { fail, pass }
pass
test
execution
OBS
t
fail
© Ed Brinksma/Jan Tretmans
TT 2004, Formal Framework
20
Testing for Conformance
IUT passes
Ts
?

IUT conforms-to s
IUT passes Ts

passes
Ts t def  t  Ts .
 t  TsIUT
. IUT
passes

 t  TIUT
( exec (t t, 
IUT
= ( exec
pass ( t, IUT ) ) = pass
def) ) t
s . tpasses

 tTest
 Tshypothesis
. t ( obs :( t, iIUT ) )

iProof
s  t : TESTS . exec ( t, IUT ) = obs ( t, iIUT )
obligation
IUT imp

 i  MODS .
IUT conforms-to s
Definition : IUT conforms-to s
(  t  Ts . t ( obs ( t, i ) ) = pass )  i imp s
© Ed Brinksma/Jan Tretmans
TT 2004, Formal Framework
IUT passes t
= pass
21
Testing for Conformance
Proof obligation :
 i  MODS .
(  t  Ts . t ( obs ( t, i ) ) = pass )

i imp s
Proof of completeness on models
leads to completeness for tested systems :
exhaustive
IUT passes
Ts



i conforms-to s
sound
Test derivation: find such a Ts, der : SPECS  P TESTS
© Ed Brinksma/Jan Tretmans
TT 2004, Formal Framework
22
Formal Testing
Test hypothesis :
s SPECS
imp
der : SPECS 
(TESTS)
Ts  TESTS
IUTIMPS . iIUT MODS .
tTESTS . exec(t,IUT) = obs(t,iIUT)
Proof soundness and exhaustivess:
iMODS .
( tder(s) . t(obs(t,i)) = pass )
 i imp s
pass
iIUT
IUT MODS
IMPS
obs ::
exec
TESTS 
MODS

IMPS 
(OBS)
© Ed Brinksma/Jan Tretmans
t:
OBS
(OBS)
{fail,pass}
fail
TT 2004, Formal Framework
23
Approaches to Formal Testing
 Finite State Machine based approaches:
 language acceptance
 transition tours
 distinguishing sequences
 Labelled Transition Systems
 concurrency theory
 implementation relations
 Abstract Data Type testing
© Ed Brinksma/Jan Tretmans
TT 2004, Formal Framework
24
Labelled Transition Systems
Labelled Transition System
 S, L, T, s0 
initial state
states
Discon
ConReq
actions,
labels
Discon
T  S  (L{})  S
Data
ConConf
© Ed Brinksma/Jan Tretmans
transitions
TT 2004, Formal Framework
25
s0  S
Labelled Transition Systems
s
S0
dub
S0
dub
transition
S1
kwart
S1
S2
coffee
S3
S0

S4
tea
S0
S0
dub coffee
S3
kwart tea
executable
sequence
kwart soup
non-executable
sequence
S5
L = { dub,kwart,
coffee,tea,soup}
© Ed Brinksma/Jan Tretmans
all transition
systems over L
LTS(L)
TT 2004, Formal Framework
transition
composition
26
Labelled Transition Systems
Sequences of observable actions:
s
S0
dub
traces (s) = {   L* | s
dub
S1
S2
coffee
S3
tea
S4
}
traces(s) = { , dub, dub coffee, dub tea }
Reachable states:

s after  = { s’ | s
s after dub = {
S1, S2
s after dub tea = {
© Ed Brinksma/Jan Tretmans

TT 2004, Formal Framework
27
}
S4
}
s’ }
Formal Testing with Transition
Systems
Test hypothesis :
s  LTS
ioco
der : LTS 
(TTS)
Ts  TTS
IUTIMPS . iIUT IOTS .
tTTS . exec(t,IUT) = obs(t,iIUT)
Proof soundness and exhaustivess:
iIOTS .
( tder(s) . t(obs(t,i)) = pass )
 i ioco s
pass
iIUT
IUT IOTS
IMPS
obs
: TTS
exec
:
TESTS
IOTS 
IMPS


(traces)
(OBS)
© Ed Brinksma/Jan Tretmans
t:
traces
TT 2004, Formal Framework
(traces)
{fail,pass}
28
fail