Transcript NIEM IEPD Quick Start

Accelerated NIEM IEPD
Development
Process, Procedures and
Techniques
1
NIEM and IEPD Overview
• NIEM – National Information Exchange Model approach
(http://www.niem.gov)
• Provides Naming and Design Rules (NDR) and process for
schema development
• Provides set of core component schema (NIEM core) and schema
for domains – family services, census, environment, trade,
hazmat, nuclear, justice, immigration, et al (15 domains).
• IEPD – Information Exchange Package Documentation
• Describes formal exchange that conforms to NIEM
• Provides schema, example XML, supporting artifacts
• Re-uses NIEM components
• Defines domain specific components
2
IEPD “Fish Hook” lifecycle
1
5
2
4
3
3
Agenda
• Developing NIEM IEPDs
• Current accepted practice – pitfalls and challenges
• How to do this faster, simpler, more reliably?
• Accelerated process overview
• Blueprints and Dictionaries
• Constructing your exchange and blueprints
• Leveraging re-use – NIEM, LEXS, UCORE dictionaries
• Creating your own domain dictionary from XSD or UML
• Generating IEPD artifacts
• NDR evaluation, Exchange schema, NIEM mapping, “want
list” + subset, XML instances, business rules documentation
• Summary
4
Developing IEPDs
Current Practice – Pitfalls and Challenges
How to do this faster, simpler, more reliably?
Accelerated Process Overview
5
Current Practice – Conceptual
Business
Modeling
Charter
Exchange
Model
Mapping
Artifact
XML
Schemas
Artifacts
Project Inception
Exchange
Content
Modeling
Mapping
Schema Building
Packaging
Business
Model
Modeling/
Diagraming
Tools
WorkGroup
Local
Requirements
6
Searching
Tools
Tools
Subset
Schema
Tool
XML Editor
Tools
Current Practice - Mechanics
JIEM &
Ref Model
Business
Model
Modeling
Tool
Other
IEPDs
Search
Tool
Forms
Mapping
Tool
Subschema
Generator
Standards
Mapping
Schema
Building
Test via
Validating
Packaging
& Posting
XML
Editor
All IEP
Artifacts
Exchange
Inception
Data
Modeling
Flows/
Use Case
Model
UML
Domain
Model
NIEM
Mapping
Document
Schemas
Sample
XML
Instance
Support
Docs
Exchange
Model
Business
Rules
Wantlists
Extension
Schemas
Sample
Style Sheet
Metadata
Constraint
Schemas
7
IEP Dev
Tool
IEPD
Posting
Current Practice – Team Matrix
Skillset
Qualifications
Experience
Resource
NIEM practitioner / project lead
- Proficient in NIEM
methodology and IEPD
development requirements
-- Proficient in complex XSD
syntax writing. Familiar with
developer support tooling and
constraints
-- Familiar with project
requirements and business
applications and also developing
XML based exchanges
-- Use of UML diagramming and
models. Information modelling
-- Knowledge of NIEM tools
available from www.NIEM.gov
website.
-- Creating test environments,
working with XML test cases,
test data generation
-- Writing documentation and
spreadsheets
Prior NIEM project work
1 FTE
One to two years actively
writing XSD schema
1 FTE
One year or more in
application area
1 FTE
Prior UML based modelling
1 FTE
NIEM tools training and
XML
1 FTE
Data analysis and XML
content creation
1 FTE
Technical writer
1 FTE
W3C XSD schema expert
Domain business analyst
UML/ data modelling
practitioner
NIEM tooling user
XML testing and development
Documentation resources
8
IEPD - Pitfalls and Challenges
• Significant amount of manual labor needed to develop all the
•
•
•
•
•
•
9
NIEM IEPD artifacts and XML related end products
Multi-discipline team and supporting cast of NIEM / XML savvy
developers needed
Disconnect between the software delivery teams’ schedule and
process and the NIEM IEPD development team and process;
production system not matching what the IEPD says it does
Alignment to existing domain Enterprise Data Model (EDM)
Varying quality of hand checked results and no consistency of
technical approach to schema development techniques and reuse of NIEM
Process not repeatable and predictable
Scalability - differing production XML details across teams, often
incompatible across implementations
IEPD Level of Effort Estimates
Component
Tasks
Timings
Constraints
Collect exchange needs
Model information needs
Weeks
Spiral analysis
Perform XSD schema development
with NIEM alignment
XSD syntax writing
Weeks
Documentation of each element
Excel spreadsheet
Weeks
Document NIEM dictionary mapping
(want list)
Excel spreadsheet
2 to 5
days
Complex with steep
learning curve and
limited practitioners.
Manual preparation
and review
Manual preparation
and review
Create test cases and examples
Sets of XML instances
Weeks
Perform interoperability testing
Build test environments
Weeks
Manual hand editing
of XML from XSD
Test harnesses vary
Create IEPD document
Word documentation
Weeks
Manual preparation
Currently 800+ hour process for 300+ node exchange
10
Improving the Process
• Resolving the issues and challenges
• Ensuring consistent results that can be easily reviewed
• Leverage existing dictionary work and repositories of
components that the enterprise already has
• Reduce the learning curve and need for specialized
skills
• Business analysts not excluded from design, review and
implementation by technical barriers
• Customizable and configurable so can adapt to
changing requirements
11
Faster, Simpler, Predictable
• Tooling automates much of the manual tasks; ensures predictable
•
•
•
•
•
•
•
•
•
12
quality of results
Reduce need for specialized technical knowledge of NIEM and XML
Provide consistent approach that leverages NIEM techniques with
built-in NIEM smarts and knowledge
Tooling checks for common pitfalls, applies NDR checks
Allow business analyst to complete much of the design work and
crosscheck application details
Leverage reuse of domain component dictionaries and blueprints
Lockstep development to IEPD artifacts and their delivery
Accelerate development tasks (test cases, testing, schema writing)
Produce result that are neutral to developer tooling platforms
Process repeatable and replicatable when requirements / versions
change
Using Dictionaries & Blueprints
• Dictionaries provide reference sets of components to be used in
exchanges; three possible sources:
• NIEM domain dictionaries imported from existing NIEM schema
• Domain dictionary built from an Enterprise Data Model schema
• Reverse engineered out from existing exchange schema
• Blueprint
• Is the outline of the structure components to be used in an
exchange schema
• Can import components from one or more domain dictionary
collections
• Sketches out the desired information exchange with re-use of
existing exchange component structures, plus any local additions /
extensions
• Expander tool reads the blueprint, references the dictionary, and
constructs the complete exchange schema
13
Accelerated Process Overview
Business
Modelling
Dictionary Generation
Project Inception
Import EDM Import XSD
Naming /
NDR /
spell
check
Generate
Dictionary
WorkGroup
Model Exchange
Blueprints
Search tools
Dictionary
Dictionary
Exchange
Content
Modelling
Expand, Review, Refine
NDR
Evalu
ation
Complete
d
Exchange
Structures
Expand and
Evaluate Results
Review / Tailor
Exchange
Structures
Test
Cases
Completed IEPD artifacts
XML
XSD
schemas examples
Test examples and rules
Validation
Engine
Wayfarer
SSGT
Excel
dictionaries
Generate IEPD
Artifacts and
Test them
Package IEPD
14
Charter
Rules
Report
Subset
“Wantlist” Schema
Mapping
Spreadsheet
Rendering
Stylesheet
Blueprints and Dictionaries
Constructing your exchange and blueprints
Leveraging re-use – dictionaries for NIEM, LEXS/UCORE
Creating your own domain dictionary from XSD or UML
15
Blueprint Approach Overview
NIEM
dictionaries
formatted as XML
Enterprise
Data Model
EDM
Ele
Def
DDL
Structure Outline
Blueprint
2
Components Definition (XML)
1
Import and refactor
for use with NIEM
Target
3
Pick
Components
applications
4
Expand
Structure
IEPD
generator
tools (CAM)
Exchange Structure
Local domain
dictionary
formatted as XML
Exchange
Blueprint
Designer
5
7
Def
Dictionary
Repository
IEPD
Package
LEGEND
Automated
Manual
16
Exchange
Components
Build
User Interface
Ele
6
Blueprint Development Tools
Excel
Domain dictionary
1
SSGT
Blueprint
Designer
NIEM dictionary
2
Insert
Dictionary
Parent
Components
17
3
Searc
h
Tools
Component Definitions
Component Definitions
Expander
Tool
4
5
Completed
Exchange
Template
Blueprint Expander Example
COMPLETED EXCHANGE TEMPLATE
2
Expander
Tool
1
Dictionary
Lookups
Exchange
components
outline
Structure
Details
Expanded
3
EXCHANGE BLUEPRINT
Blueprint
follows
model
18
Console Log detail
Exchange Template Editor
Exchange
Structure items
2
1
Rules for
each item
3
4
19
Can add new
custom
domain Items
Domain
Components
Namespaces
Generating IEPD artifacts
NDR evaluation, NIEM mapping, “want list” + subset,
Exchange Schema, XML instances, business rules
documentation
20
IEPD Generation Steps
EXCHANGE TEMPLATE
Structure
Toolkit for
IEPD
artifacts
generation
Rules
1.
1
Run template Evaluation Report
2.
2
Compare to NIEM dictionary
- create mapping spreadsheet
- create wantlist.xml
- NIEM online SSGT uploading +
subset schema generation
Documentation
Suite of menu options
and tools in desktop
CAM toolkit editor
(each run custom xslt
scripts on the exchange
CAM template xml)
CAM Template
21
3
Generate business rules report
4
3.
Generate exchange XSD schema
5
4.
Generate XML test instance(s)
NDR Evaluation Report
• Provides scoring, alerts, warnings and potential
issues including:
• Naming and Definition Conventions
• Naming / Restriction consistency checks
• Interoperability Enablers/Inhibitors checks
• Rules integrity and duplicates
• Statistics on exchange size
22
Evaluation Report (NDR) example
Part of an example
report for LEXS
getDataItemRequest
template displayed in
toolkit HTML viewer
23
Compare to NIEM dictionary
• References NIEM dictionary of names and properties
•
•
•
•
•
•
24
currently in NIEM 2.1 or 2.0 releases
Matches on physical names
Reports mapping details
Compatible with Microsoft Excel
Report can be used to do spell checking (in Excel)
Generates wantlist.xml file
Wantlist can be uploaded to SSGT and then subset
schema generated
(http://niem.gtri.gatech.edu/niemtools/ssgt/SSGTOptions.iepd )
NIEM cross-reference spreadsheet
Formatted view
in Microsoft
Excel of import
of crossreference
report details
(from
generated XML
file)
25
NIEM online SSGT and wantlist.xml
Options menu
and Load
Wantlist option
with resulting
load of subset
details
26
Documentation and Schema
• Documentation:
• Create HTML report of exchange schema details and
associated content and business rules
• Report layout and content designed to be reviewed by
business analysts
• Schema:
• Generate XSD schema for exchange
• Customizable IEPD folder layout management by
namespace for extension, subset and exchange schema
components
• Writes XSD schema in syntax that is clear, simple and
compatible with deployment tooling environments
27
Business Rules Documentation
Part of the example rules for LEXS
getDataItemRequest template
displayed in toolkit HTML viewer
28
Export Exchange to XSD Schema
Export Template to
Exchange XSD:
Completed
Exchange
Structure
Set
Exchange
Options
Complete set
of NIEM
exchange
schemas
generated
29
Exchange Schema Generated
Each namespace file is import for
those specific type definitions
Set of XSD files with filename
and namespace suffix
Reviewing XSD results in
a schema editor tool
30
Examples Generation
• Support for software development testing process
• Designed to allow creation of concrete realistic
•
•
•
•
31
examples not just random value based
Hinting system allows insertion of actual test system
values into XML examples
Can create both valid and invalid examples to support
unit testing of application software
Exclude capability allows generator to create examples
that contain only a portion of the entire exchange
Control over random seed value used allows regeneration of identical test cases
XML example generation wizard
View of CAM toolkit
with LEXS
getDataItemRequest
and dialogue for XML
test example
generator tool
32
Running validation rules tests
• Built-in validation engine allows testing of XML
•
•
•
•
33
instances against actual exchange rules.
Critical to ensure that the exchange validates actual live
production example scenarios correctly
Allows deployed solution to match IEPD exchange
details
Errors can be reviewed interactively in exchange visual
interface
Post-processing of validation results allows unit
regression tests to be created with reporting of errors,
warnings and information level notes
Run Exchange Template
Pick XML test case to validate
Run validation
Review results in visual editor
34
Formal IEPD Packaging Details
IEPD Artefact
Description
Exchange Files
Subset Schema (SSGT)
Subset of the full NIEM schema—a compressed directory of schemas to distinguish from other schema sets.
Wantlist
User requirements -distinguishes user-required data components from components they depend on for conformance. Generated by
and up-loaded to the Schema Subset Generation Tool (SSGT). This is an open specification and the SSGT is not required to
create a wantlist, though it is easier.
Exchange Schema
Base document schema that defines the XML root element and is generally named after the IEPD itself. Also known as the
document schema, reference schema, or root schema.
Constraint Schema
Constraints for separate constraint validation path—a compressed directory of schemas to distinguish from other schema sets.
Extension Schema
Specification for extended components—separate local name-spaces of components not contained in NIEM.
Sample XML Instance
Example instance(s) – may reference optional stylesheet.
Stylesheet
Example stylesheet for display of instance(s).
Documentation
Master Documentation
The Master Document is the main document for which all of the context and details around the exchange are explained. This
document includes, the overview, as well as details surrounding the exchange, business drivers and requirements
Exchange model
Exchange model in standard open format (xmi, vsd, zargo) and standard open graphic (jpg, pdf, etc.) preferably a Unified Modeling
Language (UML) model.
Business rules
Business rules in one of the following formats: (1) plain or structured English, (2) written into master documentation, (3)
Schematron or other formal business rule language, or (4) generated by a development tool.
Mapping to NIEM
Mapping of domain components, tagged with constraints (i.e., cardinality, etc.) to NIEM components; preferably the Component
Mapping Tool (CMT).
Extended components
Components created because they were not in NIEM—may be part of mapping spreadsheet and include structure and definitions of
new components. Prefer CMT.
Change log
Record of cumulative changes from previous IEPD versions. The initial IEPD simple records its creation date.
Catalog
Catalog XML file
A machine-readable list of artifacts in the IEPD in an open, portable and browser-displayable format (simple manual edit).
Metadata XML file
All metadata of owner and domain to be registered with the IEPD (simple manual edit).
35
Summary
Dictionary driven exchanges
Blueprint enabled reuse
Automated IEPD generation
Alignment to NDR Principles and Rules
Testing and validation support
36
Review
• Top Down development
• Reference dictionary components
• Create exchange blueprint
• Run Expander tool
• Refine desired structure in visual editor
• Automated IEPD generation
• Schemas, XML, documentation, mapping to NIEM
• Test generated example XML with rules validation
• Dictionary driven reuse
• Enterprise Data Model and NIEM/LEXS components
• NDR Principles and Rules
• Best practices for interoperability and schema techniques
37
Reference Materials
References and Links
38
Links and References
• DOWNLOADS • CAM Toolkit download
• https://sourceforge.net/projects/camprocessor
• SUPPORTING NIEM MATERIALS • NIEM Naming and Design Rules (NDR) 1.3
• http://www.niem.gov/pdf/NIEM-NDR-1-3.pdf
• NIEM Practical Implementer’s Course (online)
• http://www.niem.gov/elearning.php
• NIEM Course Modules (17 free PDFs)
• http://www.niem.gov/course.php
• Techniques for Building and Extending NIEM XML Components
• http://www.niem.gov/Techniques_for_Building_and_Exten
ding_NIEM.pdf
39