Transcript International Users Group Testing Quality Assurance

UCA International Users Group
The Quality Assurance Program
prepared by the
UCA® International Users Group
Testing Subcommittee
UPDATED: 28 July 2008
Jack Robinson: UCA International Users Group Secretary
Bruce Muschlitz: Chair Testing Subcommittee
Marco C. Janssen: editor of the QAP
International Users Group Mission
• Advancing interoperability for the utility enterprise by
providing a global forum in which stakeholders can work
cooperatively as members of a common organization to
support international standards:
•
•
•
•
Educational Activities
Improvement Process for International Standards
Testing and Certification
Promotional Activities
International Users Group Charter
• Not-for-Profit International Consortium
• Utilities and Vendors working together in support of
international standards
• Over 109 Members From 26 Countries:
• IEC TC 57 Members (Editors and Conveners of IEC 61850, CIM
and OpenAMI/DR)
• Vendors of Control and Substation Automation Products
• Utility Buyers of 61850, CIM and OpenAMI/DR Products
• Testers and Certification Providers
• Consultants, Universities, Research Facilities
Broad Technical Focus
• Utility enterprises are complex and cross many
functional boundaries
• Promoting interoperability in the utility enterprise requires
a similarly broad focus
• UCA IUG uniquely offers members an organization to
promote standardization across several functional areas
UCA IUG – Technical Focus
• Communications
– IEC 61850 – Substation Automation
– IEC 61850 based extensions for wind, hydro, distributed
generation, etc.
• Application Integration
– Common Information Model (CIM) IEC 61970 and IEC 61968
– Generic Interface Definition (GID) IEC 61970
• Metering and Demand Response
– OpenAMI – Information/data models, reference designs and
interoperability guidelines based on IntelliGrid and IEC standards.
ONE MEMBERSHIP
The Users Group and
Testing
• The Users Group strongly supports testing
• Focus for testing is now on devices and systems
developed in conformance with IEC Standards
• Users Group works closely with the IEC for problem
resolution and tracking (e.g. WG 10 members are all in the
IEC 61850 Technical Issues Users Group Subcommittee)
• Testing procedures, requirements and guidelines:
– Quality Assurance of performed activities
– Traceability of results
Testing TSC Org Chart
Testing
Subcommittee
IEC 61850
Test Procedures
WG
CIM
Test Procedures
WG
OpenDR
Test Procedures
WG
Testing SC
Testing
Subcommittee
•
•
•
•
•
•
•
•
•
•
AEP
EPRI
EnerNex
KCA
Triangle/
MicroWorks
SISCO
CIM
CIM
DR
DR
A. Clark
J.Hughes
B.Muschlitz
J.Robinson
G.Schimmel
H.Falk
Member
Member
Member
Member
Co-Chair
Co-Chair
Secretary
Committee / Working Group
Editors/ Representatives
• Testing Quality Assurance Program Editor: Marco Janssen
(UTInnovation)
• CIM Document Editors
• Tester Accreditation Task Force (unique to each area)
• IEC 61850 Document Editors:
– Tester System Accreditation:
Willem
Strabbing (KEMA)
– Server Product Test Procedures:
Richard Schimmel (KEMA)
– Client Product Test Procedures: Richard Schimmel (KEMA)
Testing Web Site Support
• Testing Subcommittee / WG Areas (Meeting Agenda’s /
Minutes, Draft Documents, Issues Discussion, Action
Items, Tasks, Events)
• Member Only Testing Documents (Corporate Access)
– Quality Assurance Program and Addendums
– Product Test Procedures
– Tester Accreditation Procedures
• Public Documents
– QAP Overview/ Testing White Papers
– Tester Certification (IEC 61850)
– Testing Completed (IEC 61850) (Posted Certificates)
The Users Group and
Quality Assurance for Testing
• A Quality Assurance Program (QAP) for IEC Product
Testing and Test System Accreditation and Recognition is a
major benefit to the members and is expected to save time
and money for participants
• The QAP allows for the continuous improvement of test
centers, test systems and the international standards
• The Users Group QAP identifies the means to quickly
resolve IEC specification problems and, through IEC
coordination, provides the vehicle for later update of IEC
standards following IEC charter rules
Concepts of the QAP
• Feedback is essential
– To keep all informed
– To make sure standards are correct and well understood
– To quickly resolve problems uncovered during testing
• Process is decentralized
– All activities relating to testing are performed individually by parties
that may be other than the UCA® International Users Group
– UCA® International Users Group plays a major role in coordinating
and managing information dissemination
• The process provides value for all who participate, and a
strategy to bring all stakeholders together (users, vendors,
testers, consultants).
QAP - The Development Process
IEC TC57
Working
Groups
Specifications
Product
Development
Products
Test Cases
Test
System
Development
Test Systems
Test
Labs
Tested Products
Projects
Success
QAP - The Quality Assurance Process
Vendor
Difficult to Coordinate
Manufacturer
System
Integrator
All Parties Need Feedback
IEC TC57
Working
Groups
Projects
(Users)
QAP
Test
System
Developers
Test
Labs
Realistic Coordination
Standards Improvement
Technical Issues (TISSUES)
Initiator emails
Could be a short
term fix
TISSUES
Selected and distributed to the
responsible member of Group of
Experts (GoE)
1st Proposal
Resp. GoE member will create
1st proposal & distribute to GoE
Final Proposal
After discussion a final proposal
is distributed to TISSUES group
for vote.
100% approval
“green” TISSUES
“yellow” TISSUES
IEC TC57 WG10
The Contents of the QAP
• Roles of:
– Standards organization
– Product development
– Test cases, test system development, test systems and test
laboratories
– Projects
• Rules, requirements and procedures for exchange of
information between them
– Input from and to:
• Vendors, manufacturers and system integrators for product
development
• Test laboratories
• Vendors, manufacturers and system integrators for projects
• Users for projects
The contents of the QAP
(cont'd)
• QAP Maintenance
• Problem reporting and handling
– Standard problem report
– Formal procedure for initial evaluation, proposals for resolution and
resolution and formal reporting of solutions and determine which
developers are involved
– Short term recommendation to user(s) and vendor(s) on how to
proceed
– Process to upgrade the documents and/or test systems associated
with the problem
• Mechanism for identification, registration and publication of
interoperability problems and the resolutions within various
revisions/errata of IEC standards as well as specific
revisions of the test systems and lab certified devices
The contents of the QAP
(cont'd)
• Organization and infrastructure
– Users Group Executive and Technical Committees
– Audits
• Authority to organize and/or perform audits at testing organizations
participating in the UCA® International Users Group QAP
• To verify the set up, implementation and use of the required quality
system.
• Periodically or upon request from a UCA® International Users Group
member with notice and mutually agreeable time schedule and
agenda.
– Confidentiality of documents and information
• In general testing results owned and controlled by testers and vendors
• Results reported back to UCA® International Users Group and are
considered public information for dissemination to Members
• Field problems carry the level of confidentiality as designated by the
reporting company
The contents of the QAP
(cont'd)
• Organization and infrastructure
– Information sharing
• Amongst UCA® International Users Group Members in accordance
with the confidentiality guidelines
• Primary sharing vehicle is the Users Group web site under access and
password control
• Public areas available to any one who visits the web site *
• Confidential areas open to Users Group Members
• Private Forum open to committee or working group members
* = www.ucaiug.mobi
The contents of the QAP
(cont'd)
• Procedures, requirements and guidelines for the realization
of tests including definitions for:
planning of tests
test definitions and set up
formal and informal testing
the use of test tools
re-testing
self testing
procurement steps
test architecture
/ system configuration
test architecture for conformance
what is to be conformance tested?
test configurations
conformance test procedures
/ steps
/ results
the conformance test process
standard test procedure groups
IEC 61850 conformance blocks
test case list
test report
test procedures
control of testing organizations
and systems
The contents of the QAP
(cont'd)
• Test Procedures
– Detailed test procedures for IEC 61850 Device Testing
• Used by testing organizations
• Maintained by the UCA® International Users Group
• Contains Abstract Test Cases, based on IEC 61850 Part 10
– Certification done by test centers, users and test vendors
(No direct certification of devices by Users Group)
– Any qualified facility may issue a “Product Certificate”
– Users Group maintains certified devices list on the web site
– Users / vendors that submitted the device for test own the detailed
test report / results
The contents of the QAP
(cont'd)
• Control of Conformance and Interoperability
– Starts with accreditation and recognition of testing organizations
– No minimum conditions IEC 61850 protocol functionality to be
supported in order to get a certificate
– Certificate remains valid as long as the owner follows the Users
Group guidelines specified in the QAP
– Users Group maintains record of all tested and all tested and
certified products and systems including the references to the
documents and their respective versions that the tests performed
were based upon
– Arbitrator role for the Testing Committee
The contents of the QAP
(cont'd)
• Resolution of Problems resulting from Testing
– Testing must be documented, repeatable, and results comparable
across tested equipment / applications
– QAP provides process for problem tracking, analysis, and
resolution of tests, standards, and implementation agreements
– Users Group assists testing organizations in
•
•
•
•
•
Maintaining problem reports
Resolving specification conflicts
Modifying the standards as problems are uncovered
Maintaining lists of certified products
Ensuring the testing facilities keep up their qualifications per approved
test procedures
The contents of the QAP
(cont'd)
• Resolution of Problems resulting from Testing
– Access to QAP and problem information, along with key technical
detail for users, product developers and integrators
– Status evaluation of own devices against a ‘non-aligned’ technical
resource that presents an unbiased interpretation of the problems
and possible resolutions
– Participation in the problem feedback process by Approved Testers
– Information on problems uncovered in the field by vendors who
receive device certificates from approved testers
– Vendors and testers shall include in their users and quality
assurance manuals, a problem identification, resolution and
reporting process that includes the UCA® International Users Group
feedback step
The contents of the QAP
(cont'd)
• Improvement Process
– In accordance with the ISO 9000 series of standards, the UCA®
International Users Group strives for continuous process
improvement
– The community including members and non-members of the UCA®
International Users Group are invited to use the mechanisms for
providing feedback
– In accordance with the procedures laid down in the QAP the UCA®
International Users Group will facilitate improvements.
Other Available Testing
Documents
• Test Center Accreditation and Recognition Procedure for
IEC 61850 Device Testing (Test System Accreditation)
• Conformance test for IEC 61850 Servers
(Device Test Procedures)
• Testing of CIM Products (Draft by the CIM group)
• CIM Compliance Document/ Conformance Testing
Requirements
• Conformance test for IEC 61850 Clients
Test System Accreditation
• Describes the procedure to qualify test centers
• Conditions that test centers should comply to in order to
certify IEC 61850 based products and attach an Users
Group label to the tested product
• To Date: There are 3 Users Group approved Test Centers
for IEC 61850 Products: KEMA, Nari-Relays and ABB
• Tested device vendors (receiving certificates) participate in
the Users Group Quality Assurance Program
Device Test Procedures
• Conformance test procedures and results in general
and in detail (IEC 61850 Example)
Testing Milestones
• AEP/UG Testing Workshop (May 05: Demo of Testing
Capability, Tools, and Rollout of the QAP)
• QAP Version 2.6 (March 2007)
– Update for General Quality Assurance Program
– Addendum for IEC 61850 (June 2008)
– Addendum for CIM and OpenAMI/DR (in preparation)
• Tester Certification IEC 61850
– KEMA
– ABB (Recognized)
– Nari-Relays (Recognized)
• Device Testing IEC 61850 Server Procedures Rev 2.2
(October 2007)
• Testing Completed (IEC 61850) for 12 Vendors and 79
Products (posted on Web Site)
Testing Current Plans
• QAP Version 2.X
– Addendum for CIM
– Addendum for OpenAMI/DR
• Tester Certification IEC 6185
• Device Testing IEC 61850 Server Procedures Use
Rev 2.2 (after October 2008)
• Device Testing IEC 61850 Client Rev 1.0 (by early
2008)
• Future Documents: Product Mark, Implementation
Agreements, Testing Specifications, Test
Procedures for CIM and OpenAMI/DR
Bottom-Line Benefits
• Test and certification activities that lower costs for users
• Provide input to international standards activities at no
additional cost and no travel
• Members only access to:
–
–
–
–
Resolution of technical issues related to standards
Industry experts via a global help desk
Technical documents and best practices from existing users
On-line discussion forums
• Discounts on paid seminars and exhibitions
Conclusions
• The available documents provide comprehensive means to
assure the quality of the testing process
• Testing will be well documented, repeatable, and results
shall be comparable across tested equipment/applications.
• The QAP provides an appropriate process through which
to perform problem tracking, analysis, and resolution of
tests, standards, and implementation agreements
• The UCA® International Users Group provides a vehicle for
deployment and testing of high-quality products based on
open standards for the Electric Utility Industry
To Join The UCA International Users
Group
• Go To: www.ucaiug.mobi or E-mail
[email protected] or call (919) 847-2241
• For Public Documents and Information:
www.ucaiug.mobi or E-mail
[email protected] or
[email protected]