Transcript USAGE MODEL BASED TESTING

Model-Based Black Box Testing
and its Standardization Layers
MaTeLo
THE SOLUTION TO BOOST
YOUR TEST EFFICIENCY
ALL4TEC
Since 1998
French: 45 experts
Turnover 2010: 4 M€
Expertise offer:
PROCESS IMPROVEMENT
SAFETY ENGINEERING
SYSTEM ENGINEERING
OPTIMIZED TESTING
SOFTWARE RELIABILITY
EMBEDDED
SYSTEMS
OUR COMPETENCES OFFER
Training
Tooling
Consulting
MaTeLo
EXAM
M Elopée
Safety Architect
Engineering
AGENDA
 THE TEST PROBLEM
 THE EXPECTED SOLUTION
 THE MODEL-BASED TESTING SOLUTION
 A PRAGMATIC INDUSTRIAL USE-CASE
 A MORE STANDARDIZED TOOLS INTEGRATION
THE TESTING PROBLEM
 Test Resources Lack
Testing
Effort
Test
Needs
System
Complexity
Test
Availability
Time
 Test Focus Deviation
Reliability
Usage
Intensity
Market
Expectation
Development
Phase
Reality
Customer
Validation
Normal
Usage
Product Life
HOW TO SOLVE THIS ISSUE?
Test the SUT usage
before product release
Use best in class
engineering process
Widely use
automatic generation
Use dedicated tools that decuple
engineering productivity
Use formalization
and traceability
MODEL-DRIVEN ENGINEERING
DESIGN
TEST
Functional
Requirements
Implement
Scenarize
Design
Model
Usage
Model
MaTeLo
Generate
Model Based Testing
Code
Generate
Test Cases
Compile
Derive
EXE
Test
Environment
Test Scripts
WHAT IS MaTeLo?
Formal Test
Specification
More
Productive
Coverage
Product Usage
Point of View
MaTeLo
Markov Test Logic
Fully
Measured
Process
Test Execution
Compatibility
Bidirectional
Requirements
Traceability
Systematic
Engineering
Productivity
MATELO SCREENSHOT
INTEGRATED TEST WORKFLOW
PLAN
ACT
MaTeLo
Usage Model Editor
Test Design into
Usage Models
MaTeLo
Test Campaign Analysis
MaTeLo
Testor
Confidence
And Coverage
Reporting
Automatic Test
Cases Generation
EXAM
Test Automation
DO
CHECK
System
Requirements
Usage
Model
Test
Cases
Requirements
Traceability
Model
Coverage
Test
Scripts
Adaptor
Reliability
Test Execution
Platform
System
Under
Test
Mean Time
To Failure
Intensity
of Test
MaTeLo EDITOR
Usage Model Design
MATELO USAGE MODELS
 Probability on every Alternatives
BEGINNING
 Association of Requirements
 Configuration of Test Operations
 Setup Transition as Test Step
TRANSITION
MACRO CHAIN
STATE
END
MODEL TRANSITION = TEST STEP
Select_Gear(5)
Accelerate(100%)
Check_Speed(185)
Requirements
Test Operations
Stimulations
Inputs Stimulation
Equivalence Classes
Timing
Test Oracle
Outputs = f(Inputs)
Verifications
Expected Outputs
Timing
MATELO PROFILE
 Profiles can be embedded to qualify the usage model
 Operational profiles
 Test profiles
Data distribution
87 (+/- 25)
347
0-100-150-200-500
Usage path probability
System
Requirements
Usage
Model
Test
Cases
Requirements
Traceability
Model
Coverage
Test
Scripts
Adaptor
Reliability
Test Execution
Platform
System
Under
Test
Mean Time
To Failure
Intensity
of Test
MaTeLo EDITOR
Requirements Management
REQUIREMENTS MANAGEMENT
IBM Rational Doors
MaTeLo Requirements Library
Import
MaTeLo Usage Model
Association
(N,M)
Requirement Update
New Baseline
Update
Quick
Model
Update
Requirements
Change
Analysis
Model
Impact
Highlighting
Available for
other ReqMgt tools
with import of
XML or CSV files
System
Requirements
Usage
Model
Test
Cases
Requirements
Traceability
Model
Coverage
Test
Scripts
Adaptor
Reliability
Test Execution
Platform
System
Under
Test
Mean Time
To Failure
Intensity
of Test
MaTeLo TESTOR
Test Cases Generation
MaTeLo TEST STRATEGY
Most probable
approach
Risk based
approach
Arcs coverage
approach
Start
Start
Start
Usage
approach
Start
Custom
Test profile
Usage
Test profile
Close
FREQUENCY
FOCUS
Close
Close
Close
CRITICALITY, COMPLEXITY
UPDATE FOCUS
REQUIREMENTS
COVERAGE
OPERATIONAL
COVERAGE
 DEFINE THE TEST STRATEGY, BY CHOOSING
 Test Algorithm
 Test Profile
 Part of model to test
Generate
MaTeLo TESTOR: HTML TEST PLAN
State
Requirement
Input
Test Function
Expected Result
Verdict
Example: MaTeLo FOR EXAM
Test Repository
Requirements
 SUT Interface
 Test Operations
 Stimulation
 Measurement
 Administration
 Sub Test Sequences
 Name
 UUID
 Description
Association
Automatic Test Cases
 Automatic call of Test Operation
 Automatic parameterisation
 Requirements association
 Test Case description generation
 Usage model respect
Association
Generate
System
Requirements
Usage
Model
Test
Cases
MaTeLo TCA
Test Campaign Analysis
Requirements
Traceability
Model
Coverage
Test
Scripts
Adaptor
Reliability
Test Execution
Platform
System
Under
Test
Mean Time
To Failure
Intensity
of Test
Test
New
Scripts
Package
Adaptor
Test Execution
Platform
System
Under
Test
Model
Coverage
Intensity
of Test
Reliability
System
Under
Test
Test Execution
Platform
Mean Time
To Failure
Intensity
of Test
Bugs
System
Under
Test
System
Under
Test
Intensity
of Test
Intensity
of Test
Reliability
Mean Time
To Failure
Adaptor
Adaptor
Reliability
Test
Scripts
Test
Scripts
Reliability
Test Execution
Test Execution Mean Time
Platform
Platform
To Failure
Test
Cases
Test
Cases
System
Under
Test
Reliability
Mean Time
To Failure Requirements
Requirements
Traceability Traceability
Intensity
of
Test
Model
Model
Coverage
Coverage
New Test
Cases
Model
Coverage
Generation
Test Execution
Platform
Adaptor
Adaptor
Test
Requirements Scripts
Traceability
Usage
Model
Test
Cases
Usage
Test
Model
Scripts
Test
Cases
Usage
Model
System
Under
Test
Test Execution
Platform
Adaptor
Test
Scripts
Test
Cases
Intensity
of Test
Mean Time
To Failure
Reliability
Model
Coverage
Requirements
Traceability
System
Requirements
Model
Coverage
Usage
Model
Requirements
Traceability
Target Quality Criteria
Test
Cases
Confidence Indicators
Usage
Model
System
Requirements
System
Requirements
System
System
RequirementsRequirements
Coverage
Reliability
Usage
Model
1.
2.
System
Requirements
TEST CAMPAIGN PROCESS
Metrics
Quality/Version
Requirements
Traceability
Mean Time
To Failure
New version
 Operational Confidence
 Needs Coverage
 Functional Coverage
System
Requirements
Usage
Model
From
Test
Cases
Requirements
Traceability
Model
Coverage
Adaptor
Reliability
System
Under
Test
To
Its Standard Adoption
Test
Scripts
Test Execution
Platform
Pragmatic Tools Integration
Mean Time
To Failure
Intensity
of Test
THEORETICAL MBT WORKFLOW
Automation
Test Framework
Atomic
Test
Action
Confidence
& Coverage
Reports
Test Scheduler
Test
Scripts
Usage
Model
Test
Cases
Verdicts
Test
Cases
Simulator
Functional
Requirements
Test
Execution
Log
System
Under
Test
A 1ST INDUSTRIAL TOOLS INTEGRATION
EXAM
Automation
Test Framework
Atomic
Test
Action
Confidence
& Coverage
Reports
Test Scheduler
UML
Python
Usage
Model
MaTeLo
Test
Scripts
Test
Cases
Verdicts
Test
Cases
dSpace
Simulator
Doors
Functional
Requirements
Vision
MicroNova
Carts
Test
Execution
Log
CAN
System
Under
Test
IOs
A 2nd INDUSTRIAL TOOLS INTEGRATION
EXAM
TestStand
Automation
Test Framework
Atomic
Test
Action
Confidence
& Coverage
Reports
Test Scheduler
UML
Python
Usage
Model
MaTeLo
Test
Cases
Test
Cases
Verdicts
Proprietary
dSpace
Opal-RT
Clemessy
Doors
Reqtify
Functional
Requirements
Test
Scripts
NI
Simulator
Vision
Diagnostic
MS Excel
Radio
MicroNova
Carts
Test
Execution
Log
CAN
System
Under
Test
Calibration
IOs
VARIOUS TOOLS INTEGRATION
EXAM
Automation
Test Framework
CANoe
Atomic
Test
Action
TestStand
PROVEtech
EXACT
Proprietary
Confidence
& Coverage
Reports
Test Scheduler
UML
TTCN-3
Python
CSV
Usage
Model
Test
Cases
MaTeLo
Java
dSpace
NI
Clemessy
HP QC
MS Excel
XML
C, C#
Proprietary
Opal-RT
Doors
Reqtify
Functional
CSV
Requirements
VB
Test
Scripts
Simulator
Test
Cases
Verdicts
MicroNova
Carts
Simulink
CANoe
Vision
CAN
Diagnostic
Flexray
Radio
System
Under
Test
Calibration
IOs
Temperature
Test
Execution
Log
VARIOUS TOOLS INTEGRATION
EXAM
Automation
Test Framework
CANoe
Atomic
Test
Action
TestStand
PROVEtech
EXACT
Proprietary
Confidence
& Coverage
Reports
Test Scheduler
UML
TTCN-3
Python
CSV
Usage
Model
Test
Cases
MaTeLo
Java
dSpace
NI
Clemessy
HP QC
MS Excel
XML
C, C#
Proprietary
Opal-RT
Doors
Reqtify
Functional
CSV
Requirements
VB
Test
Scripts
Simulator
Test
Cases
Verdicts
MicroNova
Carts
Simulink
CANoe
Vision
CAN
Diagnostic
Flexray
Radio
System
Under
Test
Calibration
IOs
Temperature
Test
Execution
Log
MBT AND ITS STANDARDS
EXAM
Automation
Test Framework
CANoe
Atomic
Test
Action
PROVEtech
Python
Proprietary
Test
Cases
MaTeLo
Proprietary
OTX
dSpace
Opal-RT
RIF
Clemessy
Doors
Reqtify
Functional
CSV
Requirements
XML
Test
Cases
Verdicts
Java
HIL API
NI
MicroNova
ODS
Carts
CANoe
XCP
CAN
SystemGDICalibration
DiagnosticODX
Under
Test
Flexray
IOs
Radio
ATX
C, C#ATX
Simulator
Simulink
FIBEX
Vision
Confidence
& Coverage
Reports
VB
Test
Scripts
CSV
ATX
Usage
Model
MS Excel
EXACT
Test Scheduler
UML
TTCN-3
ATX
HP QC
TestStand
Temperature
Test
Execution
Log
Items
Tools
Standard
Application
Feeling
Requirements
Doors
RIF, ReqIF
3/5
Test Specification Model
MaTeLo, Word
- (DSL)
-
Test Cases Specification
Doors, Word, EXAM
ATX, OTX
0/5 , 2/5
MaTeLo, Quality Center
ATX
0/5
EXAM, ECU TEST, TestStand, vb, python…
HIL API, TTCN3
1/5 , 2/5
Test Execution
-
-
-
•Physical Signal
NI, dSPACE, Vector…
HIL API
1/5
•Network Signal
CAN, LIN, FlexRay…
FIBEX
3/5
•ECU parameter
CANape, INCA
XCP
5/5
CANdela,
ODX, UDS
5/5
•Simulator Signal
Simulink, AMEsim…
HIL API
1/5
•Test Device Signal
Agilent, LeCroy…
GDI
3/5
NI, EXAM, dSPACE, ETAS…
ODS, ATML
2/5, 2/5
Quality Center, MaTeLo
ATX
0/5
BugZilla, Jira, Quality Center
ISSUE
?
Doors, Quality Center, MaTeLo
ATX, ReqIF
0/5 , 2/5
Quality Center, MaTeLo
ATX
0/5
Test Campaign, Management
Test Case Implementation
•Diagnostic
Test Case Execution Log, Trace
Test Case Verdict
Issue
Requirement Coverage
Confidence Indicator
QUESTION ?
www.all4tec.net
Sales contact
[email protected]
+33 6 80 88 40 59
Wiki & Forum & Documentation
www.all4tec.net
ABSTRACTION LAYERS
MaTeLo
Usage Scenario
Description
<< derive >>
Markov-Chain
Usage Model
ATX
Test Case
Specification
<< generate >>
Test Flow Control
Device Driver
UML Sequence
Diagram
HIL API
Python
Precompiled
Application
<< control >>ODX, FIBEX, XCP…
Third Party
Equipment
System Under Test
Hardware In the
Loop Test Bench
EXAM ABSTRACTION MANAGEMENT
Possible Usage Scenarios
Strategic Test Cases
Test Case 1
HIL Plateform
HIL API
dSpace
NovaSim
Carts
ASAM HIL API
Proprietary
…
ATX
Test Case 2
Test Case n
Available Test Configurations
Network
Functions
Test Tools
CAN_1
CAN_2
LIN_X
FlexRay_1
FlexRay_2
…
ODX, FIBEX, XCP…
Manual Gear
Auto Gear
Hand Free
ACC
StartStop
…
CANoe
CANape
INCA
MS Excel
Diag Tool
…
Env. Model
HIL API
Gasoline
Diesel
Turbo
Hybrid
Electric
…