Transcript Automated Model Based Testing From Theory via Tools to

Testing Techniques
Conformance Testing Methodology
and Framework
ISO IS-9646
Overview
 Conformance Testing Methodology and Framework
ISO IS-9646
Literature: “An Overview of OSI Conformance Testing”
© Ed Brinksma/Jan Tretmans
TT 2004, ISO Conformance
2
Overview: This Lecture
 Protocol conformance testing: why and what
 Standardization
 Overview Standard IS-9646:
Conformance Testing Methodology and Framework







Conformance: static and dynamic
Options, protocol implementation conformance statement
Test generation: test purposes, abstract-, executable tests
Test methods, test architectures,
Test classification and -structuring
Test notation
Test implementation and -execution
© Ed Brinksma/Jan Tretmans
TT 2004, ISO Conformance
3
Types of Testing
© Ed Brinksma/Jan Tretmans
TT 2004, ISO Conformance
4
Protocol Conformance Testing
distributed systems

exchange of information

rules for communication : protocol

standardized protocol specifications

conforming implementations

testing for protocol conformance
© Ed Brinksma/Jan Tretmans
TT 2004, ISO Conformance
5
Protocol Conformance Testing
International Standard ISO IS-9646 :
Conformance Testing Methodology and Framework

How to test protocol entity implementations
for conformance with respect to their specifications

Originally for OSI systems

Now more generally applied

Defines test language TTCN

Standard accepted by ITU-T, ETSI, ATM-Forum, ……
© Ed Brinksma/Jan Tretmans
TT 2004, ISO Conformance
6
Protocol Conformance Testing
 Starting point :
a correct and valid (standardized) protocol specification
 Original goal :
standardization of test suites in TTCN
 consistency between specification and test suite ?
 Conformance testing  interoperability testing
© Ed Brinksma/Jan Tretmans
TT 2004, ISO Conformance
7
The Conference Protocol
© Ed Brinksma/Jan Tretmans
TT 2004, ISO Conformance
8
The Conference Protocol
join
leave
send
receive
CPE
CPE
CPE
Conference Service
UDP Layer
© Ed Brinksma/Jan Tretmans
TT 2004, ISO Conformance
9
Conference Protocol
Test Architecture
Tester
TorX
A
UT-PCO = C-SAP
CPE
= IUT
U-SAP
B
C
LT-PCO
UDP Layer
© Ed Brinksma/Jan Tretmans
TT 2004, ISO Conformance
10
LT-PCO
Protocol Conformance Testing
vs. Interoperability Testing
(N)-ASPs
(N)-ASPs
(N)-service provider
(N)-Protocol
Entity
(N)-PDUs
protocol
(N)-Protocol
Entity
(N-1)-ASPs
(N-1)-ASPs
(N-1)-service provider
© Ed Brinksma/Jan Tretmans
TT 2004, ISO Conformance
11
Protocol Conformance Testing
vs. Interoperability Testing
 Protocol Conformance Testing:
 correct protocol (design and validation)
 conforming protocol entities (testing)

interoperability
 Interoperability testing
 test directly whether each pair of protocol entities can work
together (i.e. deliver (N)-service)
 Internet:
 interoperability testing
with (standard) reference entities
© Ed Brinksma/Jan Tretmans
TT 2004, ISO Conformance
12
Standard Conformance Testing
 Testing by:
 manufacturers
 users organizations
 telecom operators
 independent test laboratories
 Standard IS-9646 for
 certification of implementations
 accreditation of testing processes and test laboratories
 international acceptance of test results
© Ed Brinksma/Jan Tretmans
TT 2004, ISO Conformance
13
ISO IS - 9646
 Part 1 :
Introduction and general concepts
 Part 2 :
Abstract test suite specification
 Part 3 :
TTCN
Tree and Tabular Combined Notation
 Part 4 :
Execution of tests
 Part 5 :
Requirements on test laboratories and
administrative conformance assessment
process
© Ed Brinksma/Jan Tretmans
TT 2004, ISO Conformance
14
standard
protocol
specification
test
generation
implementation
process
standard
conformance
test suite
protocol
implementation
IUT
test
implementation
executable
test suite
test
execution
process of
conformance testing
© Ed Brinksma/Jan Tretmans
verdict
TT 2004, ISO Conformance
15
Conforming Implementation
 Specifications in natural language
 What is a conforming implementation ?
 “ … exhibits conformance
if it complies with the conformance requirements
of the applicable specification standard …”
 Satisfaction of conformance requirements
 Conformance requirements :
“the implementation shall do ….. “
“the implementation shall not do …..”
 Complication : options
© Ed Brinksma/Jan Tretmans
TT 2004, ISO Conformance
16
Conforming Implementation
 (P)ICS (Protocol) Implementation Conformance Statement :
 list of implemented options
 supplied by manufacturer
 (P)ICS proforma
 fill-in questionnaire with all possible options
 attachment to protocol standard
 to be completed by manufacturer to make PICS
 Restrictions on combinations of implemented options
 documented in standard in static conformance requirements
© Ed Brinksma/Jan Tretmans
TT 2004, ISO Conformance
17
Example :
Static Conformance Requirement
 Feature A is optional
 Feature B is optional
 But A may only be implemented
if B is also implemented
© Ed Brinksma/Jan Tretmans
TT 2004, ISO Conformance
18
Conforming Implementation
 Requirements on dynamic behaviour:
dynamic conformance requirements
 Messages :
 structure
 coding
 Ordering :
 sequence of sent and receive messages
 relation between information content of different messages
 Example:
 When in state Enable, upon receiving a Data-Request,
the implementation shall send a Data-PDU
or it shall enter the abort phase.
© Ed Brinksma/Jan Tretmans
TT 2004, ISO Conformance
19
Conforming Implementation
 A conforming implementation is one
which satisfies both static and dynamic conformance
requirements, consistent with
the capabilities stated in the PICS
 Conformance testing:
 checking static conformance by reviewing
 checking dynamic conformance requirements by testing
© Ed Brinksma/Jan Tretmans
TT 2004, ISO Conformance
20
Test Generation
dynamic conformance requirement

test purpose(s)

generic test case

abstract test case

executable test case

test case execution
© Ed Brinksma/Jan Tretmans
TT 2004, ISO Conformance
21
Test Generation
 Test purpose :
precise description (natural language)
of what is to be tested in order to decide about
satisfaction of a particular conformance requirement
 Example:
 When in state Enable, upon receiving a Data-Request,
the implementation shall send a Data-PDU
or it shall enter the abort phase.
 Check whether in state Enable, after receiving
a Data-Request, a Data-PDU is sent.
(but implementation may also enter the abort phase!)
© Ed Brinksma/Jan Tretmans
TT 2004, ISO Conformance
22
Test Generation
 Generic test case:
operationalization of test purpose;
description of actions to be performed
to reach test purpose
 Example:
 Check whether in state Enable, after receiving
a Data-Request, a Data-PDU is sent.
 Go to state Enable, send a Data-Request,
and observe a Data-PDU, and if not check that the
implementation starts entering the abort phase
© Ed Brinksma/Jan Tretmans
TT 2004, ISO Conformance
23
Test Generation
 Abstract test case:
take test method into account
 Test method ( test architecture, test environment ) :
abstract description of environment
in which testing will be performed,
in terms of PCO - Points of Control and Observation
 Example:
Do Connect-Request,
check Connect-Response (now in state Enable),
send Data-Request
observe Data-PDU
or observe Start-Abort
© Ed Brinksma/Jan Tretmans
TT 2004, ISO Conformance
24
Test Methods
Local test method
UT
PCO
test
coordination
(N)-ASPs
IUT
PCO
(N)-PDUs
(N-1)-ASPs
LT
© Ed Brinksma/Jan Tretmans
TT 2004, ISO Conformance
25
Test Methods
Distributed test method
test system
SUT
test
coordination
LT
UT
PCO
(N)-ASPs
(N)-PDUs
PCO
IUT
(N-1)-ASPs
service provider
© Ed Brinksma/Jan Tretmans
TT 2004, ISO Conformance
26
Test Notation
 Abstract test cases are written in standardized notation :
TTCN : Tree and Tabular Combined Notation
 Specification language for Abstract Test Cases
 Behaviour specified as tree of sequences of possible events
 Data specified in tables or in ASN.1
 ISO IS-9646 Part 3
 With:
input ? , output ! (from tester perspective!) ,
timers / time out, ordering of events,
default behaviour, alternative behaviours,
tree attachment (=subroutines),
parallel component testers,
verdicts: pass, fail, inconclusive
© Ed Brinksma/Jan Tretmans
TT 2004, ISO Conformance
27
Example : TTCN
Test Case Dynamic Behaviour
Test Case Name: Conn_Estab
Group:
transport/connection
Purpose:
Check connection establishment with remote initiative
behaviour
constraints
+preamble
LT ! T-PDU-connect-request
UT ? T-SP-connect-indication
UT ! T-SP-connect-response
LT ? T-PDU-connect-confirm
OTHERWISE
LT ? T-PDU-disconnect-request
OTHERWISE
© Ed Brinksma/Jan Tretmans
TT 2004, ISO Conformance
verdict
pass
fail
inconclusive
fail
28
Classification of Tests
 Basic interconnection tests
 Capability tests
 Behaviour tests
 Conformance resolution tests
© Ed Brinksma/Jan Tretmans
TT 2004, ISO Conformance
29
Structuring of Test Suites
Test suite
Test group
Test group
Test group
Test case
Test case
Test case
Test step
Test step
Test step
Test event
Test event
Test event
PCO ? X
© Ed Brinksma/Jan Tretmans
PCO ! A
TT 2004, ISO Conformance
30
Test Implementation
 Transform abstract test suite into executable test suite by
 compilation
 interpretation
 a test slave …..
 Select test cases from abstract test suite based on PICS
 Incorporate (P)IXIT (Protocol) Implementation eXtra Information for Testing :
 address information
 timer information
 specific parameter values
 …..
 PIXIT proforma supplied by test laboratory
© Ed Brinksma/Jan Tretmans
TT 2004, ISO Conformance
31
Test Execution
 Execute / interpret executable test suite
 Collect and analyse the results
based on verdicts pass, fail, inconclusive
of each test case execution
 Collect all results of
 static conformance review
 dynamic testing
into Protocol Conformance Test Report
© Ed Brinksma/Jan Tretmans
TT 2004, ISO Conformance
32
standard protocol
specification
implementation
process
dynamic
conformance req’s
test
purposes
static
conformance req’s
generic
test suite
PICS
proforma
PIXIT
proforma
(standardized)
abstract
test suite
protocol
implementation
IUT
Basic interconn.
capability
behaviour
PICS
PIXIT
test
selection
static
conformance
review
test
implementation
test
execution
analysis
of results
executable
test suite
verdict
test report
© Ed Brinksma/Jan Tretmans
certification
TT 2004, ISO Conformance
33
standardized
test methods
test
notation