A conformance test suite for TTCN-3 tools Benjamin Zeiss, T
Download
Report
Transcript A conformance test suite for TTCN-3 tools Benjamin Zeiss, T
A conformance test suite for TTCN-3 tools
Benjamin Zeiss, T-Systems
Andras Kovacs, Broadbit
Bogdan Stanca-Kaposta, Testing Technologies
Agenda
Motivation
STF 409
Methodology
Discussion
Challenges
Conclusion
A conformance test suite for TTCN-3 tools
Slide 2
Motivation (1/2)
Many TTCN-3 compilers (~7 commercial, 2 internal)
High interest in the language
Complex standard: hundreds of pages
Estimation of ~5000 requirements
No way to check the compliance of TTCN-3 tools
TTCN-3 is based on ETSI standards
A conformance test suite for TTCN-3 tools is needed!
Quality of a test tool is critical
Errors in the test tool may influece test results.
A conformance test suite for TTCN-3 tools
Slide 3
Motivation (2/2)
Relevance:
For users:
Is my tool standards compliant?
Can I switch tools easily / vendor lock-in?
For tool vendors:
Can I improve the standard compliance of my tool?
Possibly later: compliance certification
For standard developers:
Where is need for clarification?
A conformance test suite for TTCN-3 tools
Slide 4
STF 409 (1/2)
Specialist Task Force initiated by ETSI MTS
Manpower: 3 persons w. 20 days each (60 days total)
Benjamin Zeiss (T-Systems/University of Göttingen)
Andras Kovacs (Broadbit)
Bogdan Stanca-Kaposta (Testing Technologies)
Dates: October 2010 – Early 2011
Work is based on a framework given by TTCN-3 STF.
Task:
Devel. of a conformance test suite for TTCN-3 tools.
Creation of a thorough documentation for the ATS.
A conformance test suite for TTCN-3 tools
Slide 5
STF 409 (2/2)
Three deliverables:
ETSI TS 102 950-1 V1.1.1 (2011-04) – ICS
ETSI TS 102 950-1 V1.1.1 (2011-04) – TSS & TP
ETSI TS 102 950-2 V1.1.1 (2011-04) – ATS & IXIT
Approved by MTS in April 2011.
Downloadable from the ETSI download area right now!
Time spent:
10 days each (30 work days)
Development of conformance tests.
10 days each + extra time (30+ work days):
Validation, fixing of problems, documentation, tool
development.
A conformance test suite for TTCN-3 tools
Slide 6
Methodology (1/7)
Five main steps:
1.
2.
3.
4.
5.
Create clause priority list. Discuss with MTS.
Develop conformance tests.
Create documentation framework.
Generate documentation content.
Validate and fix.
Step 1 (creation of clause priority list):
Not all clauses can be covered. Selection has to be
made.
How to prioritize?
Vendor feedback + most basic functionality.
There is no perfect selection!
A conformance test suite for TTCN-3 tools
Slide 7
Methodology (2/7)
Step 2 (development of conformance tests):
The ATS is not the test automation, but the test input!
In most cases: 1 module = 1 conformance test.
Two steps:
1.
2.
Specification of the test purpose: @purpose + @verdict
Implementation of the test in TTCN-3.
Conformance verdict:
Expected tool output is specified in the document tags (@verdict).
Purpose:
Also specified in the document tags (@purpose).
Test automation is not provided by the ATS!
Can be scripted easily by extracting the document tags.
Example Java source code can be provided!
Test suite structure: organized according to clauses.
A conformance test suite for TTCN-3 tools
Slide 8
Methodology (3/7)
Step 2 (development of conformance tests) cont.:
Expected output for verdict determination:
@verdict pass accept/reject [expectedoutput]
Examples:
@verdict pass reject
@verdict pass accept, ttcn3verdict:pass
@verdict pass accept, noexecution
Purpose specification:
@purpose documentreference, description
Documentreference: part:clause
Examples:
@purpose 1:5, Ensure that when the IUT loads a module
containing some definitions before the module declaration
then the module is rejected.
A conformance test suite for TTCN-3 tools
Slide 9
Methodology (4/7)
Step 2 (development of conformance tests) cont.:
Complete TTCN-3 test input example:
/***************************************************
** @purpose 1:5.2.2, Ensure that the IUT correctly
handles the uniqueness of
variable names in its scope.
** @verdict pass accept, ttcn3verdict:pass
***************************************************/
module Sem_050202_Uniqueness_001 {
import from Sem_050202_Uniqueness_001_import {
const all;
}
type component GeneralComp {}
function f_funcScope() {
var boolean repeatedIdentifier := true;
if(repeatedIdentifier==true) { setverdict(pass);
}
}
testcase TC_Sem_050202_Uniqueness_001()
runs on GeneralComp { f_funcScope(); }
control { execute(TC_Sem_050202_Uniqueness_001()); }
}
module Sem_050202_Uniqueness_001_import {
// repeated indentifier
// from imported module
// is allowed
const integer repeatedIdentifier := 0;
}
A conformance test suite for TTCN-3 tools
Slide 10
Methodology (5/7)
Step 3 (creation of documentation framework):
Already provided by STF393 (Proforma).
Split single document into three documents.
Step 4 (generate documentation content):
Only 60 days manpower, little time for documentation.
Working on a single source saves time.
Solution:
Write test purposes and expected tool outputs directly into
the test inputs.
Generate tables for the documentation from the ATS:
Test purpose tables.
Test suite structure.
Much easier to maintain!
Documentation generation tools developed as part of
STF409.
A conformance test suite for TTCN-3 tools
Slide 11
Methodology (6/7)
Step 5 (validation and fixing):
Validation and fixing took more than half of the
development time!
STF internal validation with 4 tools:
TTworkbench: compile-time and execution.
TestCast: compile-time and execution.
IBM: some initial compile-time validation.
MTP: as reports are sent in.
Lots of feedback from Elvior and MTP!
A conformance test suite for TTCN-3 tools
Slide 12
Methodology (7/7)
Executing the ATS:
TTCN-3
File
TTCN-3
File
TTCN-3 File
ATS
Sem_060101_TopLevel_001.ttcn
NegSem_060303_component_types_001.ttcn
…
Sem_060101_TopLevel_001.ttcn
** @verdict
pass accept, ttcn3verdict:pass
Test input
TTCN-3 Tool (test subject)
Expected
output
Tool output
Rejection as invalid
Execution with results
(e.g., pass/inconc/fail)
Rejection to execute
Tool output evaluation
pass
fail
A conformance test suite for TTCN-3 tools
Slide 13
Discussion
TTCN-3
Conformance ATS
TTCN-3 Tool
pass
Meaning?
Compliance does not imply safety from a vendor lock-in!
Testing is never complete!
With v1.1.1 of the ATS, about 32% of the core standard clauses
are covered with at least one test.
Ideally, we should have many tests for each clause.
Compliance means less likelihood for vendor lock-in.
With better coverage and higher number of test cases, we
improve the quality of this statement.
Compliance reduces the likelihood for major tool errors
Tests range from simple arithmetic operations to
communication operations and verdict handling.
ATS checks syntax and semantics.
ATS does not check the correctness of every single tool
message, but of the overall output.
A conformance test suite for TTCN-3 tools
Slide 14
Challenges (1/2)
Various standard clarifications were necessary:
9 CRs for TTCN-3 core languages have been written.
Unclarified test cases have been excluded from the delivery
("on-hold").
Tools sometimes handle the standard differently.
Tests need to be expressed as tool independent as possible.
Time management:
Proper validation and fixing is extremely time consuming.
More than 10 voluntary days have been actually spent.
Responsibilities:
Everyone mixes up STF393 and STF409 responsibilities
(proforma).
A conformance test suite for TTCN-3 tools
Slide 15
Challenges (2/2)
Version numbers:
To which core language standard version does the
conformance test suite relate to?
How to maintain compatibility across different language
versions?
Against which version should the conformance test
suite be developed in the future?
Target quality:
The target quality of the test suite should be fixed right
in the beginning (Terms of Reference).
A conformance test suite for TTCN-3 tools
Slide 16
Conclusion
The conformance test suite is available now!
ATS Metrics:
Total number of conformance tests: 733
Number of positive syntactic conformance tests: 110
Number of negative syntactic conformance tests: 34
Number of positive semantic conformance tests: 335
Number of negative semantic conformance tests: 254
Total number of clauses in the TTCN-3 standard: 342
Total number of clauses (at least partially) covered: 110
About 140 pages of test purposes.
Free to use for tool developers. Use it to improve your tool!
Bug reports: http://t-ort.etsi.org
A conformance test suite for TTCN-3 tools
Slide 17
Special thanks:
Tool validation and comments
Elvior, MTP, Conformiq, Ericsson, ISPRAS
Jens Grabowski / University of Göttingen
Questions? Feedback?
A conformance test suite for TTCN-3 tools
Slide 18