TTCN, Past, Present and Future

O. Monkewich Chairman, ITU-T Joint Coordination Activity on Conformance and Interoperability Testing (JCA-CIT)

Overview

 Past - some pre-history, before TTCN-3  Present - new ITU initiative  Future - how can TTCN-3 and TTCN-3 experts help

In the Beginning

 

OSI

 Implementations from different suppliers to interconnect

October 10-13, 1983, Ottawa, Canada, ISO/TC97/SC16/WG1

  Dave Rayner of NPL, UK, appointed Rapporteur of OSI Conformance Testing Methodology and Framework First meeting achievements   Static Conformance – visual review Dynamic Conformance – in an instance of communication    Will need a test notation Test specifiers to be allowed their own notation The standard would provide some examples

Test methods, Verdicts, Multi layer, Single-layer, Embedded testing



February 13-15, 1984 GMD Darmstadt, Germany ISO/TC97/SC16/WG1 Conf. Group

  First discuss of requirements for test notation  ASPs, PDUs, Test Methods  Verdicts Must consider  single-layer testing  embedded single-layer testing  multi-layer testing

Neutral test notation or an FDT?



November 12-16, 1984, Ft. Lauderdale Florida, ISO/TC97/SC16/WG1 Conformance

  Anthony Wiles, Uppsala University, Sweden  Uppsala University developing tools for testing Conformance Group developing a neutral test notation but is also looking at FDTs  Plan to meet with the ISO FDT experts  Prepare testing examples for comparison with other potential notations

Tables, Trees and FDTs



February 3-13, 1985, Paris, France, ISO/TC97/SC16

    France - a table per test case UK proposed behaviour trees – textual, branching with indentations Both adopted – birth of TTCN Met with FDT groups - LOTOS, Estelle  Continue with TTCN but keep looking at FDTs 

July 7-12, 1985, Manchester, UK, ISO/TC97/SC16 Conformance Rapporteur Group

 Upper Testers, Test Coordination Procedures  Requirements on test notation:      Verdict assignment: Pass, Fail, Inconclusive Valid, Invalid and Inopportune PDUs timing, addressing sending, receiving PDUs Compare observed vs. required outcome

The “Bows-and-Arrows” notation



October 18-30, 1985, Paris, France, ISO/TC97/SC6 Conformance test suites

    3 X.25 test suites in ad hoc time sequence diagrams    DTE-DCE physical link – computer-modem DTE-DCE link access to network – computer-modem DTE-DTE packets over network – computer-computer Canada argues for using TTCN, labels the US notation “bows and arrows” notation Canada, please translate to TTCN or … Initiated effort to develop the University of Ottawa/Nortel “Workbench” tool for TTCN

Stabilize TTCN for X.25, MHS



April 21-29, 1986, ISO/TC97/SC21 jointly with CCITT Q.47/VII, San Diego (La Jolla), California

 Should ASN.1 be added for PDU description     Canada wants TTCN syntax to expressed in BNF TTCN is informal - sufficient to define it by example CCITT (now ITU-T) will develop a test suite for MHS, they prefer TTCN A version of TTCN was stabilized (San Diego version)

TTCN still informal, but, can we describe it in BNF



September 8-26, 1986, Egham, UK, ISO/TC97/SC21

       Canada argues need BNF for machine readability Keep San Diego (La Jolla ) version of TTCN for X.25 and MHS LOTOS gaining interest Criteria for test notation identified  Notations not evaluated by February 1987 will not be considered TTCN is still considered informal: it is for human readability, abstract test suite specifiers, not for test realizers PIXIT introduced (tongue-in-cheek acronym) but was adopted Test suite work not for the conformance group, shifted to protocol specifier groups

X.25 Test Suites will Allow Other Test Notations (in an Annex)



October 13-24, 1986, Tokyo, Japan ISO/JTC1/SC6 Conformance testing

  Definition of any new test notation will be added to the test suite annex Canada promises X.25 Packet Level test suite in TTCN to the next meeting

BNF for TTCN arrives

 

February 3-12, 1987, Nice, France, ISO/JCT1/SC21/WG1 Conformance Rapporteur

Canada tabled BNF description of TTCN, TTCN.MP to come  

April 27-May 1, 1987, Washington, DC, TTCN meeting

Construct to permit TTCN trees to execute in parallel introduced  

February 28-March 9, 1988, McLean, Virginia, USA, ISO/JCT1/SC21/WG1

TTCN should be separate part of DP9646 rather than an annex of DP9646 Part 2

TTCN.MP

  

May 23, 1988, Teddington, NPL, UK, TTCN Editing meeting July 14-22, 1988, Stockholm, Sweden, SO/JCT1/SC21/WG1 Conformance editing meeting (hosted by Telia Research/Teletest)

   Teddington/Stockholm TTCN becomes Part 3 of DP9646 (DP9646-3) TTCN made self-consistent, integrated with BNF, machine readable TTCN.MP

Could not agree on Operational Semantics

November 28-December 8, 1988, Sydney, Australia ISO/IEC JTC1/SC21/WG1 Conformance Group

   DP9646 Part 3 did not make it to DIS Operational Semantics substantially changed Stable “kernel” of TTCN could go to DIS after the next Stockholm editing meeting, July 18-29, 1989.

TTCN extensions

 

February 7-17, 1991, Phoenix, Arizona, ISO/JCT1/SC21/WG1 Conf. Group

 TTCN extensions    Rational for concurrency, TCs, PTCs and MTCs Coordination Messages, Coordination Points, Test Configurations Starting and stopping Test Components CREATE and TERMINATE

May 20-31, 1991, Arles, France, ISO/JCT1/SC21

 Editing TTCN Extensions   DEFAULT ACTIVATE, DEACTIVATE, RETURN, OTHERWISE Handling background activity (cannot be forced or ignored)  TTCN-1 was considered completed

TTCN-1 approved as ISO/IEC International Standard



Ottawa, February 18-19, 1992, ISO/IEC/JTC1/SC21/WP1

 TTCN-1 (TTCN edition 1) approved as ISO/IEC International Standards and published on 12 November 1992   Anthony Wiles hands over TTCN Editorship to Os Monkewich Further Extensions as well as the ongoing Extensions

Concurrency: Draft Amendments One and Two to TTCN-1:



RTP North Carolina, 19-27 October, 1992, ISO/IEC/JTC1/SC21/WP1 TTCN Editing Meeting

 Extensions and Further Extensions  TC, PTC, MTC  Test Suite Operations and control agreed  CREATE(List of PTCs) passed by value  RETURN for default trees only  ?DONE added  TERMINATE deleted

The Last TTCN-2 Meeting



BSI, Chiswick, London, UK, September, 1994, ISO/IEC/JTC1/SC21/WP1 TTCN Editing Meeting

  Cleanup, over 200 comments, defects reports on Extensions and Further Extensions Modular TTCN – Southampton, July 1994  Remaining work done by e-mail

TTCN Edition 2 (TTCN-2)

  

TTCN-1 vs. TTCN-2

TTCN-1 Non-concurrent:  Maximum of two PCOs   Single service provider Behaviour is expressed successive branch indentations only TTCN-2 Concurrent      Unlimited number of PCOs Multiple service providers Behaviour expressed by parallel execution of behaviour trees Modules for reuse of ATSs Uses 54 different tables

Years after the beginning

    

Started in 1983 TTCN-1 completed and published in 1992 TTCN-2 completed and published in 1998 ISO/IEC JTC1 stops work on OSI

  Defect Report processing more difficult ETSI corrected defects to produce TTCN-2++

ITU-T SG10 and SG7 (now SG17)

   Languages, SDL, MSCs, ASN.1 and TTCN Adopted TTCN-2++ in 2002 as Rec. X.292

2001- 2011, SG17 approved and continues to updated 11 TTCN-3 Recommendations based on ETSI standards

Since 2002, ITU-T has approved 11 TTCN-3 Recommendations

          

Z.161: TTCN-3 core language Z.162:TTCN-3 tabular presentation format (TFT) Z.163: TTCN-3 graphical presentation format (GFT) Z.164: TTCN-3 operational semantics Z.165: TTCN-3 runtime interface (TRI) Z.166: TTCN-3 control interface (TCI) Z.167: TTCN-3 mapping from ASN.1

Z.168: TTCN-3 mapping from CORBA DL Z.169: TTCN-3 mapping from XML data definition Z.170: TTCN-3 documentation comment specification Z.171: TTCN-3 and the use of C/C++

New Horizons



ITU-T WTSA-8 Resolution 76 approved

 TSB to carry out specific tasks with the view to introducing the use of ITU Mark Program 

ITU Mark Program

 permit manufacturers and service providers to add a visible ITU mark to their product if it

conforms

to ITU T Recommendations 

Assist the Developing Countries

  Capacity building Establishing of regional test laboratories

Interoperability – the ultimate objective

 Interoperability does not follow from conformance  Today’s ITU-T Recommendations not good enough for global interoperability  No explicit conformance or interoperability requirements  Interoperability is achieved by means of on-the-fly adjustments  Adjustments not made publicly available  Interoperability exclusive 

Developing Countries want both

 Conformance and Interoperability

Role of TTCN-3

 Conformance testing is required to   Qualify for the ITU Mark Be entered into the ITU Database  Testing quality  Must support the criteria for formal testing    Test laboratory accreditation Criteria for Global harmonization and recognition Product certification   Voluntary – wide acceptance in the marketplace Regulatory – meet requirements of national law

Can anything go wrong?

      

ITU Recommendations for the modern times

Packaging of conformance requirements     specifications from a standard implemented in different boxes no declaration of which requirements are in what box boxes may communicate over proprietary interfaces no standardized interfaces or access points for testing Boxes and network components from multiple vendors Streamed video utilizing a variety of "hidden" codecs Operators and service providers downloading their software onto user devices Plug-ins downloaded for use with the implementation Hidden proprietary actions as providers make changes to the user system

CIT Section for ITU Recommendations

 Proposal to include requirements for    Reporting any factors needed for achieving conformance and interoperability Defining the essential elements for end-to-end communication Information for network element management for multi-vendor mix of network component  JCA-CIT draft CIT Section and Guide  Use Guide to complete the CIT Section  Include Section in each Recommendation

Joint Coordination Activity-Conformance and Interoperability Testing (JCA-CIT)

 Membership     TSB staff and Study Group chairmen and vice-chairmen Representatives appointed by the Study Groups Experts form A.4/A.5 SDOs such as ETSI Meet 3 to 4 times per year   Combination of in body, via Webinar or GoToMeeting Focus    Recommendation ingredients for testing Assistance in the implementation Resolution 76   Roadmap for implementation ITU Mark, ITU Database, capacity building, test laboratories URL:

http://www.itu.int/en/ITU-T/jca/cit/Pages/default.aspx

New opportunities for experts

      

Training in Developing Countries

Conformance, compliance and interoperability Production of Requests For Proposal (RFPs) procurement of products Establishing new test laboratories Test laboratory accreditation Product certification On-site, hands-on training

New test laboratories

  Establishing regional test laboratories           Setting up a test laboratory from start Defining the requirements for the lab Mission statement Business plan laboratory operation cost recovery Funding land purchase Specification of the equipment needed for its operation Scope of what will be tested Plans for accreditation

Contribution to standards

    Conformance and interoperability Improving Recommendations for conformance and interoperability Development and standardization of test suites The testing process    Recognition criteria Assertion of compliance Credentials for admission to be listed by ITU

Conclusions

        15 years to develop early concepts used in TTCN-3 10 years of TTCN-3 New ITU and Regional initiatives creating new opportunities for TTCN-3 and its experts Testing experts are needed to  Develop and improve standards to allow testing   Help Developing countries gain technical capabilities Help build new test laboratories in regions Wide recognition of test results Accreditation and Certification ITU Database population ITU-Mark management