Transcript Real-time Testing: From Practice to Theory

Verification Modelling of
Embedded systems
Ed Brinksma
Dept. of CS, University of Twente, NL
joint work with
Angelika Mader
Monterey Workshop 2003
Chicago
ES Verification Example:
storm surge barrier control
Area
flooded
in 1953
catastrophe
dimensions
Last
construction
of the
Delta Works
September, 2003
Monterey Workshop, Chicago
The control system
 no human intervention
human operation too unreliable
 responsible for closing & opening
online meteorological & hydrological data
 very low failure rates
event failure rate ~10-5/barrier event
 design & verification with FM
considered successful
September, 2003
Monterey Workshop, Chicago
Design validation
 data types & operations formally
specified in Z
 crucial control parts modelled in
Promela & model checked with Spin
 implementations were hand-coded
using Z specs
 implementations were tested
 no actual code was proved correct
September, 2003
Monterey Workshop, Chicago
Questions
 How to do practical verification of ES ?
 Is it methodologically sound?
 How should this affect research?
September, 2003
Monterey Workshop, Chicago
The Setting
 verification of ES designs is desirable
critical aspects are common:
safety-critical, high replication, costly, etc.
 verification needs formalization
operational model, logical theory, requirements
 formalization is problematic
 the (standard) combinatorial explosion
 incorporation of (physical) environment
September, 2003
Monterey Workshop, Chicago
Typical Situation
 verification model is constructed in an
ad hoc and opportunistic manner
the verification crisis:
 the success of verification is crucially
hacking
dependentmodel
on scarce
expertise
precedes
checkingmodel
 the relation
of model
the verification
to the actual design is opaque
September, 2003
Monterey Workshop, Chicago
What do we need?
Verification models should have/be
 limited complexity
must be open to computer-aided verification
 faithful
must capture relevant properties
 traceable
clear relation to actual design or system
September, 2003
Monterey Workshop, Chicago
Complexity Issues
 models must be sufficiently small
limited capacity verification tools
limited capacity verification management
 hybrid nature ES complicates models
mixed techniques, symbolic analysis
 tool capacity growth exceeds Moore’s law
better algorithms & data structures
September, 2003
Monterey Workshop, Chicago
Abstractions
Verification models are abstractions:
 inherent abstractions
mathematical modelling of physical aspects
 controlled abstractions
simplifications reducing complexity
September, 2003
Monterey Workshop, Chicago
Faithfulness
 Verification of erroneous models is useless
(or even worse).
 Models must obviously capture the relevant
system properties.
However:
verification models &
properties
must
be
 what are relevant
(formal)
properties?
!
these are often part validated
of the design
problem
 do our abstractions preserve them?
this can be difficult to show (begging the question)
September, 2003
Monterey Workshop, Chicago
Model validation
In addition to traceability verification
models can be validated by experimental
means:
1. simulation of the model
requires constructive modelling
2. analysis of verification results
in practice model validation and
verification are mixed
September, 2003
Monterey Workshop, Chicago
Separating the errors
 a verification run may fail due to
1. an error in the implementation
2. an error in the verification model
verification
3. an error
in the should
formalalways
property

include a systematic error
errors must be analysed
discussion (cf. physics)
to modify appropriate entity

requires rigourous protocol
for analysis & documentation
September, 2003
Monterey Workshop, Chicago
Software Model Extraction
 program code as model
 reduction by abstract interpretation
 data/predicate abstraction
 variable slicing
 model
does check
not workabstractions
non-programmable
 for
eliminate
false negatives
model parts
September, 2003
Monterey Workshop, Chicago
can be done
concurrently on
many abstractions
Verification Engineering
 Verification modelling as a design problem
 closely related but different from system design
 main design criterion: limited complexity & tool
support
 Systematic approach to model construction
 capture physical aspects
 reduce complexity
 formal and experimental justification
 Tool support for verification management
 model/property version management
 meta-level specification of verification campaigns
September, 2003
Monterey Workshop, Chicago
Systematic Verification Model
Design phase
system description
requirements
modelling
simulation
M, S
Modelling phase
abstract
Mi, i
model check
correct
error
interpret error
design
error
back to
design
phase
correct error
reverify models
no
all done?
yes
September, 2003
Monterey Workshop,verified
ChicagoM w.r.t. S
Verification phase
Xspin/Project - usage
Xspin/Project adds an extra Project-menu.
Verification data can be
saved into the repository.
“Sandbox” environment:
• Accessing PRCS
• Saving validation results
• Forcing version integrity
September, 2003
Monterey Workshop, Chicago
Challenges
 verification methodology (MoMS project)
 systematic model traceability
combining formal and non-formal aspects
 modelling & abstraction patterns
libraries, domain dependent solutions
 systematic model validation
 verification management tools
 documentation & version control
 verification integrity control
 verification campaign management
September, 2003
Monterey Workshop, Chicago
And finally …
 the company involved got very
enthusiastic about FM
 a 1-year technology transfer project
was carried out
 after 5 years they are still only using
the (model-driven) testing tools
September, 2003
Monterey Workshop, Chicago