Transcript Presentation: What is this thing called conformance?

Conformance
Lynne S. Rosenthal
Mary Brady
National Institute of Standards and Technology
[email protected]
[email protected]
Outline

Terminology
 Testing Methodologies
 Benefits of Conformance Testing
 Test Suites for XML Technologies
– XML, DOM, XSLT,
– XSL FO, XML Schema
Conformance
clause
Conformance Testing
Test suites
Test tools
Certification
Validation
Certificates
Procedures, reporting
Branding
Unscrambling the Terminology
Conformance
Specification
Conformance Clause
Conformance Testing
test suite, test tools
Validation
process, reporting
Certification
certificate, brand
Terminology - Conformance

CONFORMANCE - the fulfillment of a product,
process or service of specified requirements
(ISO Guide 2)
– These requirements are specified in a standard or
specification as part of a conformance clause or in
the body of the specification

CONFORMANCE CLAUSE - a section of a
specification that states all the requirements or
criteria that must be satisfied to claim
conformance
Terminology - Conformance Testing

CONFORMANCE TESTING – a way to verify
implementations of a specification to determine
whether or not deviations from the specifications
exist (through the use of test suites)
– tests capture the technical requirements within the specification
– testing is bound in scope by the specification
– internal structure of product-under-test not accessible (black box
testing)

Necessary, but not sufficient, for interoperability
Note: referred to as conformity assessment in ISO Guide 2
Testing Methodology

Falsification Testing
– find errors by means of experimentation
– show presence of errors not their absence
– prove non-conformance; can never prove conformance

Tests Based on:
– required functionality
– logical errors by misunderstanding requirements
– errors from boundary conditions and divergence
– common programming errors
Input
Output
Specification
Requirements
Implementation
Conformance Tests
Test programs
Terminology - Testing Process

VALIDATION - process necessary to perform
conformance testing in accordance with a prescribed
procedure and official test suite
– ensures that testing can be repeatable and reproducible
– ensures that conclusions are consistent with facts presented
in the evaluation

CERTIFICATION - acknowledgement that a
validation was completed and the criteria established
by CIO for issuing certificates (brands) was met.
Testing Ingredients
Certification
qualified bodies to do the testing and certification
control board - advisory and arbiter
Validation
Process - policy and procedures for testing
Conformance Testing
Test assertions
Test suite
(test software, test scripts, test criteria)
Standard
Conformance clause
Benefits of Conformance Testing

Increased probability that products are
implemented correctly
– Contains required functionality
– Behaves as expected
– Performs functions in a known manner

Increased likelihood of portability and
interoperability
– Portability - move software or applications among different
systems
– Interoperability – ability of 2+ systems to exchange and use
information
Benefits of Conformance Testing

For Developers of Specifications
– Feedback loop

For Software Developers
– Help to improve quality - identify areas in which
they conform or not conform

For Buyers
– Increases buyer’s confidence in a product and its
ability to meet their needs
– Substantiate seller’s claims
– Not locked into purchasing from a single vendor
cont.
Components of a Test Suite
Test Harness
Test Cases
Test Scenario
Test Assertions, Axioms,
Semantic Requirements
Specification
OASIS XML Conformance

XML Conformance Committee
– Develop tests to improve the quality of XML
processors
– Provide a forum to discuss issues related to
interoperability
– Provide input to W3C regarding errors/ambiguities

Deliverables
– Test Suite containing over 2000 tests, Second Edition
– Associated test report
– Built using XML/XSLT
XML Test Suite

Types of XML Tests
– Binary
• Validating processors must check validity constraints.
• Non-validating processors do not have to read external
entities. Accordingly, they may not report errors which
would have been detected had those entities been read.
– Output Tests
• Requirements on XML processors to ensure that they
pass the necessary information on to applications.
XML Test Matrix
Non-validating
External Entities
Ignored
External Entities
Read
Validating
Valid Documents
Accept
Accept
Accept
Invalid Documents
Accept
Accept
Accept
Not-wf Documents
Reject
Reject
Reject
WF Errors tied to
External Entity
Accept (varies)
Reject
Reject
Documents with
Optional Errors
(not specified)
(not specified)
(not specified)
XML Test Suite
Testcases.dtd
Conformance.xml
• Test Profile
• NIST / OASIS
• Test URI
• SUN Microsystems
• Unique ID
• James Clark
• Spec Section
• FUJI Xerox
• Type
• IBM
• Entities
• Description
Test Case DTD
XML Test Description Files
Test Report
•XSL Stylesheets
•IE5, HTML renditions
XML.COM
• Java application
• Tests XML processors
• Dave Brownell
Outputs
<TESTCASES PROFILE=“SUN Microsystems XML Tests”>
<TEST URI=“sun/invalid/e101.xml” ID=“e101” SECTIONS=“3 [39]” TYPE=“invalid” ENTITIES=“none”>
Tests the Element Valid VC by including an undeclared child element.</TEST>
</TESTCASES>
XML Test Suite Results, v1


XML.COM Conformance Update – Dave Brownell
Validating Parsers
– Tested JAXP 1.0, IBM Xerces/Java 1.0.3, Layered Validator
(distributed with Aelfred 2), MSXML3 Technology Preview, Oracle
V2.0.2.7
– Pass Rate 82% - 99%

Non-validating Parsers
– Tested all of the above plus XP 0.5
– Pass rate 86% - 99%

Microsoft Update – July 2000 MSXML 3.0 Beta – Chris Lovett
– Non-validating: DOM: 95%, SAX: 97%
– Validating: 97%
NIST DOM Test Suite
Test Harness – automatically runs available tests and presents user with
a color-coded table of results; gives access to spec, assertions, and source code
Test Cases
800 ECMAScript
fundamental, extended, HTML
200 Java – fundamental,
extended
Test Scenario
Load appropriate XML or HTML file, exercise DOM calls
Test Assertions – Fundamental, Extended,
HTML Interfaces
DOM L1 Specification
DOM Test Suite

Test Results:
– Run tests against IE5, Mozilla (some fundamental
interfaces), Xerces, JAXP, and Oracle; Pass rate: 82 – 98%
– Implementations differ greatly in the way that parsers handle
white-space and entity expansion issues – differences in
DOM tree

DOM Level 2
– Working with W3C to identify resources
– Modified Java tests to be consistent with SUN contribution
– Will update NIST tests to L2 status
XSLT Test Suite





OASIS Technical Committee, chaired by G.
Ken Holman
Member contributions include Lotus, NIST,
Sun
Expands on XML Test Description file
Dave Marston, XSLT/XPATH Conformance
Wed, 4pm, Delaware Room
Public Information Meeting, Wed, 8pm, Suite
8228
XSL FO Test Suite







W3C WG to test Candidate Recommendation
Develop test assertions for all basic formatting
objects and properties
Develop test DTD and associated harness
Work with vendors to categorize available tests
Research translating available CSS and XSLT
tests
Develop tests for additional spec components
Expected March 2001
XML Schema Test Suite
Test Control
• Test Types
• # schemas
• # instances
Constraints
• Structure
• Limits
• Special info
Test Generator
Schema
Tests
Summary

Interesting URL’s
– http://www.nist.gov/xml/
– http://www.w3.org/
– http://www.oasis-open.org/

Questions???