Transcript Real-time Testing: From Practice to Theory

Model-Based Testing
Ed Brinksma
University of Twente
Dept. of Computer Science
Formal Methods & Tools group
Enschede
The Netherlands
ARTIST2 Summer School
Nässlingen
Contents
 introduction & background
 testing pre-orders
 input/output & quiescence
 ioco implementation relation
 test generation
 TorX
 test case study
 real-time testing
October 1, 2005
ARTIST2 Summer School
2
Contents
 introduction & background
 testing pre-orders
 input/output & quiescence
 ioco implementation relation
 test generation
 TorX
 test case study
 real-time testing
October 1, 2005
ARTIST2 Summer School
3
Practical problems of testing
Testing is:
But also:
 important
 ad-hoc, manual, error-prone
 much practiced
 hardly theory / research
 30% - 50% of project effort
 no attention in curricula
 expensive
 not cool :
“if you’re a bad programmer
you might be a tester”
 time critical
 not constructive
(but sadistic?)
Attitude is changing:
 more awareness
 more professional
October 1, 2005
ARTIST2 Summer School
4
Types of Testing
Level
system
integration
unit
Accessibility
robustness
performance
white box
black box
usability
reliability
functional
behaviour
Aspect
October 1, 2005
ARTIST2 Summer School
5
Test Automation
Traditional test automation
Why
not
= tools to execute and manage test
cases
generate
test automatically?!
specification
test
TTCN
cases
pass
implementation
under test
test
tool
fail
October 1, 2005
ARTIST2 Summer School
6
Verification and Testing
Verification :
 formal manipulation
 prove properties
 performed on model
Testing :
 experimentation
 show error
 concrete system
formal
world
concrete
world
Verification is only as good as
the validity of the model on
which it is based
October 1, 2005
Testing can only show the
presence of errors, not their
absence
ARTIST2 Summer School
7
Testing with Formal Methods
 Testing with respect to a formal specification
 Precise, formal definition of correctness :
good and unambiguous basis for testing
 Formal validation of tests
 Algorithmic derivation of tests :
tools for automatic test generation
 Allows to define measures expressing coverage
and quality of testing
October 1, 2005
ARTIST2 Summer School
8
Challenges of Testing Theory
 Infinity of testing:
 too many possible input combinations -- infinite breadth
 too many possible input sequences
-- infinite depth
 too many invalid and unexpected inputs
 Exhaustive testing never possible:
 when to stop testing ?
 how to invent effective and efficient test cases with high
probability of detecting errors ?
 Optimization problem of testing yield and invested
effort
 usually stop when time is over ......
October 1, 2005
ARTIST2 Summer School
9
Formal Testing
i imps
S
correctness
criterion
implementation
relation
imp
exhaustive
test
generation
  sound

specification
i passes Ts
test suite TS
implementatio
i
n
test
execution
October 1, 2005
ARTIST2 Summer School
pass / fail
10
Formal Testing : Conformance
specification
S
IUT conforms-to s
correctness
criterion
implementatio
n
IUT
s  SPECS Specification
IUT
Implementation under Test
IUT is concrete, physical object
Model the physical world
But IUT is black box ! ?
Assume that model iIUT exists
October 1, 2005
ARTIST2 Summer School
11
Contents
 introduction & background
 testing pre-orders
 input/output & quiescence
 ioco implementation relation
 test generation
 TorX
 test case study
 real-time testing
October 1, 2005
ARTIST2 Summer School
12
Testing Preorders
on Transition Systems

implementation
i
specification
s
environment
e
environment
e
For all environments e
all observations of an implementation i in e
i s 
 e  Env . obs ( e, i )  obs (e, s )
should be explained by
observations

of the specification 
s in e. 
?
October 1, 2005
?
ARTIST2 Summer School
?
13
Classical Testing Preorder
implementation
i
te
specification
s
environment
e
i
te
s

environment
e
 e  E . obs ( e, i )  obs (e, s )


LTS(L) Deadlocks(e||s)
October 1, 2005
ARTIST2 Summer School
14
Classical Testing Preorder
implementation
i
te
environment
e
i
te
s

specification
s
environment
e
 e  LTS(L) .    L* .
e
{ ||i
 |deadlocks
e||i deadlocks
afterafter
 } e
||s deadlocks after 
{  | e||s deadlocks after }

October 1, 2005
FP (i)  FP (s)
FP (p) = {  , A  | p afer  refuses A }
ARTIST2 Summer School
15
Quirky Coffee Machine
[Langerak]
Can we distinguish between these machines?
coin
tea
coin
bang bang
coffee
coin
coffee
te
tea
tea
ARTIST2 Summer School
bang bang
coffee
coffee
tea
They are
testing equivalent!
October 1, 2005
coin
16
Refusal Preorder
implementation
i
rf
specification
s
Deadlocks δ(e||i) =
{environment
(L{δ})* |
e
e||i deadlocks
after }
environment
e
i
rf
s

 e  E . obs ( e, i )  obs (e, s )

e observes with δ
deadlock on all
alternative actions
October 1, 2005
LTS(L{δ})

Deadlocks δ(e||i)
ARTIST2 Summer School
17
Quirky Coffee Machine
Revisited
coin
tea
coin
bang bang
coffee
coffee
tea
coin
te
tea
rf
tea
coin
coffee

bang bang
October 1, 2005
δ only enabled
if coffee is not
tester
ARTIST2 Summer School
coffee
coffee
bang
coffee
coin
18
Contents
 introduction & background
 testing pre-orders
 input/output & quiescence
 ioco implementation relation
 test generation
 TorX
 test case study
 real-time testing
October 1, 2005
ARTIST2 Summer School
19
I/O Transition Systems
 testing actions are usually directed, i.e. there are inputs and
outputs
L=LinLout with LinLout=
 systems can always accept all inputs
(input enabledness)
a
for all states s, for all aLin s 
 testers are I/O systems
 output (stimulus) is input for the SUT
 input (response) is output of the SUT
October 1, 2005
ARTIST2 Summer School
20
Quiescence

Because of input enabledness S||T deadlocks iff T
produces no stimuli and S no responses. This is known as
quiescence

Observing quiescence leads to two implementation relations
for I/O systems I and S:
1.
I iote S iff for all I/O testers T:

Deadlocks(I||T)  Deadlocks(S||T)
(quiescence)
2.
I iorf S iff for all I/O testers T:

Deadlocksδ(I||T)  Deadlocksδ(S||T)
(repetitive quiescence)
October 1, 2005
ARTIST2 Summer School
21
Input-Output QCM
states must
be saturated
with input
coin?
coffee?
loops for
input
tea?
enabledness
bang?
coin!
coin?
bang?
tea?
coffee?
coffee?
coffee!
tea !
coffee?
coffee!
October 1, 2005
quiescen
t states
iote
tea?
iorf
ARTIST2 Summer School
22

bang!
coffee !
tea !
coffee!
coffee?
Contents
 introduction & background
 testing pre-orders
 input/output & quiescence
 ioco implementation relation
 test generation
 TorX
 test case study
 real-time testing
October 1, 2005
ARTIST2 Summer School
23
Implementation Relation
ioco
δ
By adding a transition p  p to every quiescent state of a system
we treat quiescence as an observable (synchronizable) action:
i
iorf
s   I/O tests T: Deadlocksδ(i||T)  Deadlocksδ(s||T)
   ( L  {  } )*: out ( i after )  out ( s after )
To allow under-specification we restrict the set of traces:
i ioco s    Tracesδ( s ) : out ( i after )  out ( s after )
October 1, 2005
ARTIST2 Summer School
24
Implementation Relation
ioco
Correctness expressed by implementation relation ioco:
i ioco s =def   Tracesδ( s ) : out (i after )  out (s after )
Intuition:
i ioco-conforms to s, iff
if i produces output x after trace ,
then s can produce x after 
if i cannot produce any output after trace ,
then s cannot produce any output after  ( quiescence  )
October 1, 2005
ARTIST2 Summer School
25
Implementation Relation ioco
i ioco s =def   Straces (s) : out (i after )  out (s after )
i

?kwart
?dub
!coffee
October 1, 2005
= {}
out ( i after ?dub )
= { !coffee }
out ( i after ?dub.?dub )
= { !coffee }
out ( i after ?dub.!coffee) = {  }
?dub
?kwart
?dub
?kwart
out ( i after e )

out ( i after ?kwart )
= {}
out ( i after !coffee )
= 
out ( i after ?dub.!tea )
= 
out ( i after  )
= {}
ARTIST2 Summer School
26
Implementation Relation ioco
i ioco s =def   Straces (s) : out (i after )  out (s after )
i
s
?dub
?dub
?dub
!tea
!coffee
ioco
?dub
out (i after ?dub) = { !coffee }
October 1, 2005
!coffee
out (s after ?dub) = { !coffee, !tea }
ARTIST2 Summer School
27
Implementation Relation ioco
i ioco s =def   Straces (s) : out (i after )  out (s after )
i
s
?dub
?dub
?dub
!tea
?dub
!coffee
ioco
?dub
out (i after ?dub) = { !coffee, !tea }
October 1, 2005
!coffee

ARTIST2 Summer School
out (s after ?dub) = { !coffee}
28
Implementation Relation ioco
i ioco s =def   Straces (s) : out (i after )  out (s after )
i
?dub
s
?kwart
?dub
?dub
?kwart
!coffee
out (i after ?dub)
!tea
= { !coffee }
out (i after ?kwart) = { !tea }
!coffee
ioco
out (s after ?dub)
= { !coffee }
out (s after ?kwart) = 
But ?kwart  Tracesδ( s )
October 1, 2005
ARTIST2 Summer School
29
Contents
 introduction & background
 testing pre-orders
 input/output & quiescence
 ioco implementation relation
 test generation
 TorX
 test case study
 real-time testing
October 1, 2005
ARTIST2 Summer School
30
Formal Testing
i iocos
specification
correctness
criterion
implementation
relation
ioco

S
test
generation
i passes Ts
test suite TS
implementatio
i
n
test
execution
October 1, 2005
ARTIST2 Summer School
?
pass / fail
31
Test Cases
Test case t  TTS
coin?
TTS - Test Transition System :
coin ?
 labels in L  {  }
 tree-structured
 finite, deterministic
coffee!
 final states pass and fail
 from each state  pass, fail
 or all outputs o! and 
tea!
pass
October 1, 2005
ARTIST2 Summer School
fail
coin?
 either one input i?
32
tea!

coffee!
pass fail

fail
Test Cases
coin?
test case t
!coin
coin ?
!coin ; Start timer1
?tea
fail
?timer1
fail
coffee!
tea!

?coffee
!coin ; Start timer2
?tea
pass
?timer2
pass
?coffee
fail
coin?
tea!
pass
October 1, 2005
ARTIST2 Summer School
fail
33

coffee!
pass fail
fail
Test Generation Algorithm
Algorithm
To generate a test case t(S) from a transition system
specification with S set of states ( initially S = {s0} )
Apply the following steps recursively, non-deterministically
1
end test case
3
PASS
2
observe output
o!
supply input

allowed outputs o!
FAIL FAIL
forbidden outputs
t(S after o!)
supply i?
t(S after i?)
October 1, 2005
ARTIST2 Summer School
34
Test Generation Example
Equation solver for y2=x
specification
test
!9
? x (x < 0)
? x (x >= 0)
! x
otherwise
! -x
?3
FAIL PASS
otherwise
To cope with non-deterministic behaviour,
tests are not linear traces, but trees
October 1, 2005
ARTIST2 Summer School
FAIL
35
?-3
!4
?2
PASS
?-2
PASS
Validity of Test Generation
For every test t generated with algorithm :
 Soundness :
t will never fail with correct implementation
i ioco s
implies
i passes t
 Exhaustiveness :
each incorrect implementation can be detected
with a generated test t
i ioco s
October 1, 2005
implies
 t : i fails t
ARTIST2 Summer School
36
Contents
 introduction & background
 testing pre-orders
 input/output & quiescence
 ioco implementation relation
 test generation
 TorX
 test case study
 real-time testing
October 1, 2005
ARTIST2 Summer School
37
Formal Testing with Transition Systems
s  LTS
ioco
der : LTS 
(TTS)
Ts  TTS
pass
iIUT IOTS
obs : TTS
 IOTS 
(traces)
October 1, 2005
ARTIST2 Summer School
traces
38
t:
(traces)
{fail,pass}
fail
Test Generation Tools for
ioco
 TVEDA (CNET - France Telecom)
 derives TTCN tests from single process SDL specification
 developed from practical experiences
 implementation relation R1  ioco
 TGV (IRISA - Rennes)
 derives tests in TTCN from LOTOS or SDL
 uses test purposes to guide test derivation
 implementation relation: unfair extension of ioco
 TestComposer
 Combination of TVEDA and TGV in ObjectGeode
 TestGen (Stirling)
 Test generation for hardware validation
 TorX (Côte de Resyste)
October 1, 2005
ARTIST2 Summer School
39
A Test Tool : TorX

On-the-fly test generation and test execution

Implementation relation: ioco

Specification languages: LOTOS, Promela, FSP, Automata
user:
manual
automatic
next
input
specification
check
output
offer
input
TorX
observe
output
pass
fail
inconclusive
October 1, 2005
ARTIST2 Summer School
40
IUT
TorX Tool Architecture
spec.
explorer
specification
specification
text
October 1, 2005
primer
states
transitions
TorX
driver
abstract
actions
ARTIST2 Summer School
adapter
abstract
actions
41
IUT
concrete
actions
IUT
On-the-Fly  Batch Testing
on the fly
spec.
explorer
batch test generation
October 1, 2005
primer
driver
adapter
TTCN
test
taal
ARTIST2 Summer School
IUT
batch test execution
42
spec
On-the-Fly Testing
New
New
menu
Newmenu
menu
Concreteaction
action
Action
Choice
!! xCheck
Action
Choice
Concrete
action
(x
Check
Abstractaction
actionConcrete
!xx(x
(x<< 0)
<0)
0)
Abstract
action
Abstract
Concrete
action
!(quiescence)
93
00001001
!! 3
-1
!! x?
? ?(quiescence)
9
?!?!11111111
(timeout)
00000011
(x
>=>=0)
0)
?(x
3>=
?
(quiescence)
!xx(x
0)
?! -1
explorer
states
primer
transitions
driver
transition
abstract
actions
adapter
specification
bytes
IUT
implementation
? x (x < 0)
? x (x < 0)
? x (x >= 0)
? x (x >= 0)
! x
bits
! x
! -x
?x
October 1, 2005
ARTIST2 Summer School
43
TorX
October 1, 2005
ARTIST2 Summer School
44
Contents
 introduction & background
 testing pre-orders
 input/output & quiescence
 ioco implementation relation
 test generation
 TorX
 test case study
 real-time testing
October 1, 2005
ARTIST2 Summer School
45
TorX Case Studies
academic
 Conference Protocol
Philips
 EasyLink TV-VCR protocol
CMG
 Cell Broadcast Centre component
Interpay
 Road Toll Payment Box protocol
Lucent
 V5.1 Access Network protocol
CMG
 Easy Mail Melder
academic
 FTP Client
 “Oosterschelde” storm surge barrier-control
 TANGRAM: testing VLSI lithography machine
October 1, 2005
ARTIST2 Summer School
46
CMG
ASML
Interpay
Highway Tolling System
October 1, 2005
ARTIST2 Summer School
47
Highway Tolling Protocol
Characteristics :
 Simple protocol
 Parallellism :
many cars at the same time
 Encryption
 System passed traditional testing
phase
October 1, 2005
ARTIST2 Summer School
48
Highway Tolling System
Payment
Box
Onboard
Unit
(PB)
Road Side
Equipment
UDP/IP
Wireless
October 1, 2005
ARTIST2 Summer School
49
Highway Tolling:
Test Architecture
spec
Test Context
TorX
ObuSim
PB
+
ObuSim
+
TCP/IP
+
UDP/IP
PCO
TCP/IP
UDP/IP
Payment
Box
IAP
SUT
October 1, 2005
ARTIST2 Summer School
50
Highway Tolling: Results
 Test results :
 1 error during validation (design error)
 1 error during testing (coding error)
 Automated testing :
 beneficial: high volume and reliability
 many and long tests executed ( > 50,000 test events )
 very flexible: adaptation and many configurations
 Real-time :
 interference computation time on-the-fly testing
 interference quiescence and time-outs
 Step ahead in formal testing of realistic systems
October 1, 2005
ARTIST2 Summer School
51
Contents
 introduction & background
 testing pre-orders
 input/output & quiescence
 ioco implementation relation
 test generation
 TorX
 test case study
 real-time testing
October 1, 2005
ARTIST2 Summer School
52
RT TorX Hacking Approaches
1. Ignore RT functionality:
 test pure functional behaviour
 analyse timing requirements using TorX log files & assumed timing constraints
2. Add timestamps to observations
 adapter adds timestamps to observations when they are made and passed on
to the driver
 timestamps are used to analyse TorX log files
3. Add timestamps to stimuli & observations
 adapter add timestamps to observations when they are made and passed on to
the driver
 adapter adds timestamps to stimuli when they are applied and returned to the
driver
 analysis:
October 1, 2005
a.
timing error logging: observed errors are written to TorX log file
b.
timing error failure: observed errors cause fail verdict of test case
ARTIST2 Summer School
53
Real-time Testing
and I/O Systems
 can the notion of repetitive quiescence be combined
with real-time testing?
 is there a well-defined and useful conformance
relation that allows sound and (relative) complete
test derivation?
 can the TorX test tool be adapted to support Realtimed conformance testing?
October 1, 2005
ARTIST2 Summer School
54
Do We Still
Need Quiescence?
Yes!
the example
processes
should also
be distinct
in a real-time
context
October 1, 2005
coffee?
tea?
coin?
bang?
coin?
bang?
tea?
coffee?
coffee!
tea !
coffee?
tea?
coffee!
ARTIST2 Summer School
tea !
55
Real-Time and Quiescence
 s is quiescent iff:
a(d)
for no output action a and delay d: s 
 special transitions:
δ s for every quiescent system state s
s
 testers observing quiescence take time:
TestM: set of test processes having only δ(M)-actions
to observe quiescence
 assume that implementations are M-quiescent:
for all reachable states s and s’:
e(M)
if s  s’ then s’ is quiescent
October 1, 2005
ARTIST2 Summer School
56
Real-Time and Quiescence
i
M
tiorf
s
 T  TestM:
Deadlocksδ(i||T)  Deadlocksδ(s||T)
   ( L  { (M) } )*:
outM ( i after )  outM ( s after )
i tiocoM s
   Tracesδ(M)( s ) :
outM ( i after )  outM ( s after )
October 1, 2005
ARTIST2 Summer School
57
Properties
1.
for all M1  M2:
M1
M2
i tiorf
s implies i tiorf
s
2. for all time-independent i,s and M1,M20
M2
M1
i tiorf
s iff i tiorf
s iff i iorf s
October 1, 2005
ARTIST2 Summer School
58
A limitation
states are
saturated
with input
loops that
reset the
clocks
coin?
tea?
x=k
tea !
coin?
coffee?
tea?
x
x
xk
x<M
bang?
this
this process
process cannot
cannot
be
be distinguished
distinguished
fromthe
theprevious
next
from
xM
xM
bang? bang?
coffee?
xk
x=k
x<M coffee!
bang?
xk
xk
x=k
coffee!
October 1, 2005
ARTIST2 Summer School
x=k
tea !
59
Test Cases
x:= 0
Test case t  TTA
xk
TTA – Test Timed Automata :
 labels in L  {  }, G(d)
off!
fail
 tree-structured
off!
 finite, deterministic
x=5
on?
x:=0
xM
off!
 final states pass and fail
x:=0
 from each state  pass, fail
x<5
xM
fail
 choose an input i? and a time k and
wait for the time k accepting all
outputs o! and after k time unit
provide input i?

pass
 or wait for time M accepting all
outputs o! and 
October 1, 2005
ARTIST2 Summer School
60
off!
fail

x=M
fail
Timed test Generation Algorithm
can be
calculated
To generate a test case t(S) from a timed
transition
system
specification with S set of states ( effectively
initially S = {s0only
} ) for
subclasses of timed
transition systems!
apply the following steps recursively, non-deterministically
PASS
1. end test case
2. choose k  (0, M) and input μ
3. wait for observing possible output
x:=0
forbidden outputs oi!
after d’ time-units
o1!
x=d’1
on’!
x=d’n’
FAIL FAIL
October 1, 2005
x:=0
xk
μ?
x=k
tμ
allowed outputs oj!
after d time-units
o1!
on!
x=d1
x=dn
t1
forbidden outputs oi!
after d’ time-units
x=d’1
tn
o1!
on’!
x=d’n’
FAIL FAIL
ARTIST2 Summer School
xM

x=M
tδ
61
allowed outputs oj!
after d time-units
o1!
x=d1
on!
t1
x=dn
tn
Example
spec:
δ
t!
m?
c?
t?
m?
b?
b?
t?
c!
x1
c!
fail
c!
fail
c?
c!
c?
impl:
M=k
t!
x<k
pass
c!
m?
b?
t?
c?
c!
x1
c?
x=1
x:=0
:test
t!
fail
t!
fail
xM
δ
x=M
x:=0
t!
fail
x1
b?
x<k t!
October 1, 2005
t!
m?
c?
t?
c!
t?
m?
x=1
x:=0
t?
c!
c?
x<k
x<k
ARTIST2 Summer School
fail
b?
x=1
x:=0
t!
fail
x1
c!
fail
c?
x=1
x:=0
t!
fail
xM
c!
pass
62
δ
x=M
fail
t!
fail
Soundness & Completeness
 the non-timed generation algorithm can be shown to generate only
non-spurious
sound real-time test cases
errors
 test generation is complete
=
for every erroneous trace it can generate a
errors with a
positive
test that exposes it
probability of
 test generation is not limit complete
occurring
because of continuous time there are uncountably many timed error
traces and only countably many test are generated by repeated runs
 test generation is almost limit complete
repeated test geration runs will eventually generate a test case that will
expose one of the non-spurious errors of a non-conforming
implementation
October 1, 2005
ARTIST2 Summer School
63
Current Work
 Extension of the framework
 M as a function of the specification state/output channel
 integration with symbolic data generation
 test action refinement
 robustness & tolerance in real-time testing
 Extending TorX environment using CORBA IDL
 generate abstract TorX actions
 generate TTCN-3 signatures
 generate adapter code
 Practical application
 TANGRAM project: testing control software for VLSI lithography
machines (ASML)

smooth transition between timed & untimed testing
October 1, 2005
ARTIST2 Summer School
64
Future Work
 stochastic systems
 quality of service
 hybrid systems
 coverage measures
 integration white/black box spectrum
 ...
October 1, 2005
ARTIST2 Summer School
65
For more information
fmt.cs.utwente.nl/research/testing
October 1, 2005
ARTIST2 Summer School
66