Transcript Automated Model Based Testing From Theory via Tools to

Testing Techniques
Testing with
Finite State Machines
Ed Brinksma
course 2004
This Lecture : Overview
 Testing with formal methods:
 Generic framework
 Testing based on Labelled Transition Systems - ioco
 Testing based on Finite State Machines (FSM)
 Now: FSM
 State based testing
 H. Ural, Formal methods for test sequence generation,
Computer Communications, 15(5), 1992.
 Other literature:
D. Lee and M. Yannakakis,
Principles and methods of testing finite state machines - A survey.
The Proceedings of the IEEE 84, August 1996.
© Ed Brinksma/Jan Tretmans
TT 2004, FSM
2
State Machines
 Many systems can be specified / modelled as state machines
 State machines as the basis for testing :
 FSM : Finite State Machine
 black box
 specification based
 reactive systems :
 communication protocols
 control systems
 embedded systems
© Ed Brinksma/Jan Tretmans
TT 2004, FSM
3
State Machines




States
Transitions
Inputs ( “triggers”)
Outputs
state
input
output
new state
© Ed Brinksma/Jan Tretmans
TT 2004, FSM
4
Coffee Machine :
State Graph
stui? / -
koffie? / -
0
5
koffie? / -
stui? / -
dub? / dub? / stui!
koffie? / koffie!
10
dub? / dub!
© Ed Brinksma/Jan Tretmans
stui? / stui!
TT 2004, FSM
5
Coffee Machine :
State Table
 conventions (from Z):
 plain name: state
 name?:
input
 name!:
output
 Table gives new state and output as function of state and input
State
0
5
10
Input
stui?
5/-
10 / -
10 / stui!
dub?
10 /-
10 / stui!
10 / dub!
koffie?
0/-
5/-
0 / koffie!
© Ed Brinksma/Jan Tretmans
TT 2004, FSM
6
State Machine :
FSM Model
 FSM - Finite State Machine - or Mealy Machine is 5-tuple
 M = ( S, I, O, ,  )





S
I
O
: SxI  S
: SxI  O
finite set of states
finite set of inputs
finite set of outputs
transfer function
output function
 usually we also indicate an initial state
 Natural extension to sequences :
© Ed Brinksma/Jan Tretmans
TT 2004, FSM
 : S x I*  S
 : S x I*  O*
7
State Machine :
FSM Model
 FSM restrictions:
 deterministic:
 : S x I  S and  : S x I  O are functions
 completely specified:
 : S x I  S and  : S x I  O are complete functions
( empty output is allowed; sometimes implicit completeness )
 strongly connected:
from any state any other state can be reached,
or any state can be reached from the initial state
 reduced:
there are no equivalent states
© Ed Brinksma/Jan Tretmans
TT 2004, FSM
8
Coffee Machine FSM Model
stui? / -
koffie? / -
0
5
stui? / dub? / dub? / stui!
koffie? / koffie!
10
dub? / dub!
© Ed Brinksma/Jan Tretmans
stui? / stui!
TT 2004, FSM
9
koffie? / -
Testing with FSM
 Given:
 a specification FSM MS
 a ( black box ) implementation FSM MI




determine whether MI conforms to MS
i.e., MI behaves in accordance with MS
i.e., whether outputs of MI are the same as of MS
i.e., whether the reduced MI is equivalent to MS
 Possible errors:
 extra or missing states
 output fault
 transition fault
 to other state
 to new state
© Ed Brinksma/Jan Tretmans
TT 2004, FSM
10
FSM Testing
 Test with paths of the (specification) FSM
 Path = sequence of inputs with expected outputs
( cf. path testing as white-box technique)
 Infinitely many paths : how to select ?
 Different strategies :
 test every state : state coverage (of specification !)
 test every transition : transition coverage
 test output of every transition
 test output + resulting state of every transition
© Ed Brinksma/Jan Tretmans
TT 2004, FSM
11
FSM State Testing
 Make State Tour that covers every state
stui? / koffie? / -
0
stui? / -
5
koffie? / -
dub? / dub? / stui!
koffie? / koffie!
10
dub? / dub!
Test sequence :
© Ed Brinksma/Jan Tretmans
stui? / stui!
stui? dub? koffie?
TT 2004, FSM
12
FSM Transition Tour
 Make Transition Tour that covers every transition
stui? / koffie? / -
0
stui? / -
5
koffie? / -
dub? / dub? / stui!
koffie? / koffie!
10
dub? / dub!
stui? / stui!
Test sequence :
koffie? stui? koffie? stui? stui? dub? koffie? dub? koffie? stui? dub? koffie?
© Ed Brinksma/Jan Tretmans
TT 2004, FSM
13
FSM Transition Testing
 One big tour as test case not always desirable
( too long, too complex, difficult to analyse, not specific )
 Make test case for every transition separately:
S1
a? / x!
S2
 Test transition :
 Go to state S1
 Apply input a?
 Check output x!
 Verify state S2 ( optionally )
 Test purpose: “Test whether the system, when in state S1,
produces output x! on input a? and goes to state S2”
© Ed Brinksma/Jan Tretmans
TT 2004, FSM
14
FSM Transition Testing
 Go to state S1 :
 synchronizing sequence brings machine to particular state, say S0,
from any state ( but synchronizing sequence may not exist )
 or: use reset transition if available
 go from S0 to S1
( always possible because of determinism and completeness )
© Ed Brinksma/Jan Tretmans
TT 2004, FSM
15
FSM Transition Testing
synchronizing sequence :
dub? koffie?
stui?
stui? // -koffie?
koffie?
///--/-koffie?
koffie?
koffie? / -
00
55
stui?
stui? // --
koffie?
koffie? // --
dub?
dub? // -dub?
dub? // stui!
stui!
koffie?
koffie? // koffie!
koffie!
10
10
dub?
dub? // dub!
dub!
© Ed Brinksma/Jan Tretmans
stui?
stui? // stui!
stui!
TT 2004, FSM
16
FSM Transition Testing
synchronizing sequence :
dub? koffie?
stui? / koffie?
koffie?/ /- -
0
5
stui? / -
koffie? / -
dub? / dub? / stui!
koffie? / koffie!
10
dub? / dub!
stui? / stui!
To test dub? / stui! : go to state 5 by : dub? koffie? stui?
© Ed Brinksma/Jan Tretmans
TT 2004, FSM
17
FSM Transition Testing
•To test dub? / stui! :
•go to state 5 by : dub? koffie? stui?
•give input dub?
•check output stui!
•verify that machine is in state 10
stui? / -
koffie? / -
0
stui? / -
5
koffie? / -
dub? / dub? / stui!
koffie? / koffie!
10
© Ed Brinksma/Jan Tretmans
dub? / dub!
stui? / stui!
TT 2004, FSM
18
FSM Transition Testing
 State identification and verification :
 Apply sequence of inputs in the current state of the FSM
such that from the outputs we can
 identify that state where we started; or
 verify that we were in a particular start state
 Different kinds of sequences
 UIO sequences ( Unique Input Output sequence, SIOS)
 Distinguishing sequence ( DS )
 W - set
( characterizing set of sequences )
 UIOv
 SUIO
Single UIO
 MUIO
Multiple UIO
 Overlapping UIO
© Ed Brinksma/Jan Tretmans
TT 2004, FSM
19
FSM Transition Testing
 State verification :
 UIO sequences
 sequence x that distinguishes state s from all other states :
for all t  s :  ( s, x )   ( t, x )
 each state has its own UIO sequence
 UIO sequences may not exist
 Distinguishing sequence
 sequence x that produces different output for each state :
for all pairs t, s with t  s :  ( s, x )   ( t, x )
 a distinguishing sequence may not exist
 W - set of sequences
 set of sequences W which can distinguish any pair of states :
for all pairs t  s there is x  W :  ( s, x )   ( t, x )
 W - set always exists for reduced FSM
© Ed Brinksma/Jan Tretmans
TT 2004, FSM
20
FSM Transition Testing
UIO sequences
stui? / -
koffie? / -
0
stui? / -
5
koffie? / -
dub? / dub? / stui!
koffie? / koffie!
10
dub? / dub!
state 0 :
state 5 :
state 10 :
© Ed Brinksma/Jan Tretmans
stui? / stui!
stui? / - koffie? / dub? / stui!
koffie? / koffie!
TT 2004, FSM
21
FSM Transition Testing
DS sequence
stui? / koffie? / -
0
stui? / -
5
koffie? / -
dub? / dub? / stui!
koffie? / koffie!
10
dub? / dub!
DS sequence : dub?
© Ed Brinksma/Jan Tretmans
stui? / stui!
output state 0 :
output state 5 :
output
state
TT 2004,
FSM 1022:
stui!
dub!
FSM Transition Testing
go to state 5 : dub? koffie? stui?
give input dub? check output stui!
Apply UIO of state 10 : koffie? / koffie!
•To test dub? / stui! :
stui? / koffie? / -
0
stui? / -
5
koffie? / -
dub? / dub? / stui!
koffie? / koffie!
10
dub? / dub!
Test case :
stui? / stui!
dub? / * koffie? / * stui? / - dub? / stui! koffie? / koffie!
© Ed Brinksma/Jan Tretmans
TT 2004, FSM
23
FSM Transition Testing
stui? / -
koffie? / -
0
5
stui? / -
koffie? / -
dub? / dub? / stui!
koffie? / koffie!
10
dub? / dub!
stui? / stui!
- 9 transitions / test cases for coffee machine
- if end-state of one corresponds with start-state of next then concatenate
- different ways to optimize and remove overlapping / redundant parts
- there are (academic) tools to support this
© Ed Brinksma/Jan Tretmans
TT 2004, FSM
24
FSM Transition Testing
 Test transition :
 Go to state S1
 Apply input a?
 Check output x!
 Verify state S2
 Checks every output fault and transfer fault (to existing state)
 If we assume that
the number of states of the implementation machine MI
is less than or equal to
the number of states of the specification machine to MS.
then testing all transitions in this way
leads to equivalence of reduced machines,
i.e., complete conformance
© Ed Brinksma/Jan Tretmans
TT 2004, FSM
25
FSM:
variations on this theme
 there exists many variations on this theme:
 Moore machines:
output determined by state instead of transition
 Infinite state machines:
infinite number of states (e.g. state contains variable)
 Non-deterministic FSM:
transition relation instead of transition function
Labelled Transition Systems - ioco
 ...
© Ed Brinksma/Jan Tretmans
TT 2004, FSM
26
FSM: Moore machines
 Mealy Machine is 5-tuple: M = ( S, I, O, ,  )





S
I
O
: SxI  S
: SxI  O
finite set of states
finite set of inputs
finite set of outputs
transfer function
output function
 Moore Machine is 5-tuple: M = ( S, I, O, ,  )





S
I
O
: SxI  S
: S
 O
finite set of states
finite set of inputs
finite set of outputs
transfer function
output function, not dependent of input
 usually we add an initial state
© Ed Brinksma/Jan Tretmans
TT 2004, FSM
27
Coffee Machine:
Mealy model
stui? / -
koffie? / -
0
5
stui? / dub? / dub? / stui!
koffie? / koffie!
10
dub? / dub!
© Ed Brinksma/Jan Tretmans
stui? / stui!
TT 2004, FSM
28
koffie? / -
Coffee Machine:
Moore model
5/-
stui?
dub?
koffie?
K/koffie!
dub?
100/-
dub?
koffie?
koffie?
© Ed Brinksma/Jan Tretmans
105/5!
stui?
dub?
dub?
dub?
koffie?
stui?
stui?
stui?
0/-
stui?
koffie?
koffie?
TT 2004, FSM
29
1010/10!
Moore Coffee Machine :
State Table
State
0
5
100
105
1010
K
output
-
-
-
5!
10!
koffie!
stui?
5
100
105
105
105
5
dub?
100
105
1010
1010
1010
100
koffie?
0
1010
K
K
K
0
Input
 Compared to Mealy machine:
 more states
 simpler output function (not dependent on input)
 changes hardly anything for testing
© Ed Brinksma/Jan Tretmans
TT 2004, FSM
30
Infinite - State Machines
 States :
data structures / data bases
 Inputs ( “triggers”) :
operations on data bases
 Transitions :
new data base states
 Outputs :
results of queries
state
input
output
new state
© Ed Brinksma/Jan Tretmans
TT 2004, FSM
31
State Machine : FSM Model
 FSM - Finite State Machine - or Mealy Machine is 5-tuple
 M = ( S, I, O, ,  )





S
I
O
: SxI  S
: SxI  O
finite set of states
finite set of inputs
finite set of outputs
transfer function
output function
 Natural extension to sequences :
© Ed Brinksma/Jan Tretmans
 : S x I*  S
 : S x I*  O*
TT 2004, FSM
32
Infinite State Machine
Testing
 Not all transitions can be tested …..
 But principle remains the same : test transitions




Go to start state of transition
Apply input
Check output
Verify result state
 Use selection techniques to select transitions / start states
 equivalence partitioning
 boundary value analysis
 ……...
state
input
output
new state
© Ed Brinksma/Jan Tretmans
TT 2004, FSM
33
FSM Testing
vs. InfSM Testing
 Restrictions on FSM:
 deterministic
 completeness
 FSM has always alternation between input and output
 Difficult to specify interleaving in FSM
 FSM is not compositional
 FSM has “more intuitive” theory
 FSM test suite is complete
-- but only w.r.t. assumption on number of states
 FSM test theory has been around for a number of years
© Ed Brinksma/Jan Tretmans
TT 2004, FSM
34