Transcript Automated Model Based Testing From Theory via Tools to

Test Generation
for
Input/Output Transition Systems
Ed Brinksma
Course 2004
Test Cases
!dub
Test case t  TTS
TTS - Test Transition System :
!kwart
 labels in L  {  }
 tree-structured
?coffee
 finite, deterministic
 final states pass and fail
 either one input !a
 or all outputs ?x and 
?tea
pass
© Ed Brinksma/Jan Tretmans
TT 2004, Test Derivation
2

fail
!dub
 from each state  pass, fail :
?tea

?coffee
pass fail
fail
Test Cases
!dub
test case t
!kwart
?coffee
?tea

!dub
!kwart ; Start timer1
?tea
fail
?timer1
fail
?coffee
fail
!dub
?tea
pass

?coffee
fail
!dub ; Start timer1
?tea
pass
?timer2
pass
?coffee
fail
pass fail
© Ed Brinksma/Jan Tretmans
TT 2004, Test Derivation
3
Observations and Verdicts
OBS

( L  {  } )*
obs

obs : TTS  IOTS   ( ( L  {  } )* )
obs (t,i) =

t
t (O) =
=
© Ed Brinksma/Jan Tretmans
{   ( L  {  } )* |
t  i

pass  i' or
t  i

fail  i'
t :  ( ( L  {  } )* )  { fail, pass }
pass
if    O . t
fail
otherwise

TT 2004, Test Derivation
pass
4
}
Test Generation
Look for test derivation algorithm der : LTS  (TTS)
such that der is sound and exhaustive:
 i  IOTS .
(  t  der ( s ) . t ( obs ( t, i ) ) = pass )

i ioco s
i ioco s =def   Straces (s) : out (i after )  out (s after )
obs (t,i) = {   (L{})* |
t (O) =
=
t  i

pass  i' or
t  i

fail  i'
pass if    O . t
fail
© Ed Brinksma/Jan Tretmans

pass
otherwise
TT 2004, Test Derivation
5
}
Test Generation
i ioco s =def   Straces (s) : out (i after )  out (s after )
s
i
test

!x
!y


!x
?x
!z
pass
out (s after )
out (i after )
= { !x, !y }
= { !x, !z }
© Ed Brinksma/Jan Tretmans
TT 2004, Test Derivation
?y
pass
?z
fail
out (test after ) = LU
6
Test Generation
i ioco s =def   Straces (s) : out (i after )  out (s after )
s
i
test

!x
!y


!x

!z
?x

pass
?y
?z

pass fail
pass
out (s after )
out (i after )
out (test after )
= { !x, !y,  }
= { !x, !z,  }
= LU  {  }
© Ed Brinksma/Jan Tretmans
TT 2004, Test Derivation
7
Test Generation Algorithm
Algorithm
To generate a test case t(S) from a transition system
specification with S, with S set of states ( initially S = s0 after  )
Apply the following steps recursively, non-deterministically:
1
end test case
pass
2
supply input
3
observe output
forbidden outputs
?y
allowed outputs
fail fail
t ( S after !x )
!a
t ( S after ?a   )
© Ed Brinksma/Jan Tretmans
?x

allowed outputs or :
!x  out ( S )
forbidden outputs or : !y  out ( S )
TT 2004, Test Derivation
8
Test Generation Example
s

test
?dub
?dub
!dub

?coffee
!coffee

?tea
?coffee
?tea
fail
© Ed Brinksma/Jan Tretmans
TT 2004, Test Derivation
fail
9


fail
pass
pass
 Test run :
Test Execution

t  i
pass  i'
or
t  i

fail  i'
 t  i equal to t || i , with extra rule :

pass
t
then
t  i


t'

t
?tea
?coffee
fail
!dub
?tea
if
fail
i
and
t'  i
i
?dub
!tea
© Ed Brinksma/Jan Tretmans
Two test runs :
!coffee
t  i
?coffee
pass fail

i fails t
 dub tea
t  i  dub coffee
TT 2004, Test Derivation
10
pass  i'
fail  i'
Test Execution Example
i
test
?dub
?dub
!dub
?dub
?coffee
!coffee
?tea
?dub
?coffee
?tea
Two test runs :
t  i
dub 
t  i dub coffee 
© Ed Brinksma/Jan Tretmans
pass  i'
pass  i'
TT 2004, Test Derivation
fail
fail
i passes t
11


fail
pass
pass
Test Generation Example
Equation solver for y2=x
specification
test
!9
? x (x < 0)
otherwise
? x (x >= 0)
! x
! -x
fail
?3
pass
otherwise
To cope with non-deterministic behaviour,
tests are not linear traces, but trees
© Ed Brinksma/Jan Tretmans
TT 2004, Test Derivation
fail
12
?-3
!4
?2
pass
?-2
pass
Test Execution Example

implementation
test
? x (x < 0)
otherwise
? x (x >= 0)
! -x
! x
!9
fail
?y
pass
otherwise
fail
© Ed Brinksma/Jan Tretmans
?3
TT 2004, Test Derivation
13
?-3
!4
?2
pass
?-2
pass
Validity of Test Generation
For every test t generated with algorithm we have:
 Soundness :
t will never fail with correct implementation
i ioco s
implies
i passes t
 Exhaustiveness :
each incorrect implementation can be detected
with a generated test t
i ioco s
© Ed Brinksma/Jan Tretmans
implies
 t : i fails t
TT 2004, Test Derivation
14
Validity of Test Generation
Theorem:
 Soundness :
i ioco s
implies
 t  der ( s ) . t ( obs ( t, i ) ) = pass
 Exhaustiveness :
 t  der ( s ) . t ( obs ( t, i ) ) = pass
© Ed Brinksma/Jan Tretmans
TT 2004, Test Derivation
15
implies
i ioco s
Soundness and Exhaustiveness
s

test
?dub
?dub
!dub

?coffee
!coffee

?tea
?coffee
?tea
fail
© Ed Brinksma/Jan Tretmans
fail


///
fail
pass
pass
///
fail
pass
Not sound
Still anymore
sound ! !
TT 2004, Test Derivation
16
Formal Testing with Transition
Systems
Test hypothesis :
s  LTS
ioco
der : LTS 
(TTS)
Ts  TTS
IUTIMPS . iIUT IOTS .
tTTS . exec(t,IUT) = obs(t,iIUT)
Soundness and exhaustivess proved:
iIOTS .
( tder(s) . t(obs(t,i)) = pass )

i ioco s
pass
iIUT
IUT IOTS
IMPS
obs
: TTS
exec
:
TESTS
IOTS 
IMPS


(traces)
(OBS)
© Ed Brinksma/Jan Tretmans
traces
TT 2004, Test Derivation
17
t:
(traces)
{fail,pass}
fail