Transcript Enterprise Service Bus Subtitle Date: x/x/2008

Enterprise Service Bus
Lowering the cost of integration
Date: 9/1/2009
M
D
Metadata Solutions
Dan McCreary
President
Dan McCreary & Associates
[email protected]
(952) 931-9198
Agenda/Objective
• What is an ESB?
• What is enterprise messaging?
• Calculating the ROI on enterprise service bus
M
D
Copyright 2008 Dan McCreary & Associates
2
Why Build an ESB?
Consistency
Precision
Agility
• Three primary business benefits
– Precision
– Consistency
– Agility
M
D
Copyright 2008 Dan McCreary & Associates
3
Precision
Precision
• Requirements reference Business
Capabilities and are defined in business
approved terms and data elements.
• Business terms and data elements are
stored in the metadata registry
• Metrics and business rules reference
data elements that are in the registry
• Shared processes and services use
data elements that are in the registry
M
D
Copyright 2008 Dan McCreary & Associates
4
Consistency
Consistency
• There is a common compensation processing
model and each system/product maps into that
common model.
• The processing model references Thrivent
Business Capabilities, business terms, data
elements, rules and metrics that are in the
registry.
• Everyone (architects, analysts, developers,
testers) use the same definitions for business
terms, data elements, rules and metrics – all
stored in the registry.
M
D
Copyright 2008 Dan McCreary & Associates
5
Agility
Agility
M
D
• A single change to elements in the
registry allows all models to be
updated
• A single change to models can more
seamlessly be reflected in updates to
applications and services
• Changing and testing the models,
then transforming the models into
executable code allows faster
response to change, and higher
quality of delivered changes.
Copyright 2008 Dan McCreary & Associates
6
Quality Metrics for an ESB
Components
Interfaces
Flexibility
Orchestration
Rules
ATAM
Output
Standardized
Calculations
Consistency
Standardized
Information
Products
Plan Components
Maintainability
Rules and Policies
Participants
Utility
M
D
Copyright 2008 Dan McCreary & Associates
7
The Challenge
A1
A2
AN
A3
A7
A4
A6
A5
• N applications that need to
communicate with each
other
• 1st application needs N-1
connections
• 2nd application needs N-2
connections
• 100th system needs 99 new
adapters created
• O(N*N)
• 100 systems = O(10,000
interfaces)
M
D
Copyright 2008 Dan McCreary & Associates
8
The Solution
A1
A2
AN
EAI
Bus
A7
A3
• Build a central EAI
information bus that
has “adapters” to each
of the applications
• O(N) adapters
A4
A6
A5
M
D
Copyright 2008 Dan McCreary & Associates
9
The Challenge
Application
Adapter
Messaging
Protocol
EAI Bus
• How to define an
Adapter for each
system so that other
applications can
“subscribe” to its data
and process services
• Make the transport
reliable
M
D
Copyright 2008 Dan McCreary & Associates
10
Incoming!
Has this web
thing gone
away yet?
M
D
Copyright 2008 Dan McCreary & Associates
11
Publish/Subscribe Securely
Publisher 1
Publisher 2
Publisher 3
Enterprise Service Bus
Subscriber 1
Subscriber 2
Metadata
Registry
Subscriber 3
M
D
Copyright 2008 Dan McCreary & Associates
12
3 EAI Architectures
Presentation
Presentation
Presentation
Method
Method
Method
Database
Data
Database
M
D
Copyright 2008 Dan McCreary & Associates
13
The Heart of the Enterprise
The Metadata Registry
A metadata registry is a central location in an organization where
metadata definitions are stored and maintained in a controlled
method.
M
D
Copyright 2008 Danhttp://en.wikipedia.org/wiki/Metadata_registry
McCreary & Associates
14
Quality
• How do you measure it?
– Quality Metrics
• How can you put in “benchmarks” to make
sure it is improving?
M
D
Copyright 2008 Dan McCreary & Associates
15
Reuse Rule #1
If you don’t know about it
…you can’t reuse it
• If we have any asset, how does a person know it exists?
• What are the corporate repositories of reusable assts?
• How easy is it to use and be notified if an asset changes?
M
D
Copyright 2008 Dan McCreary & Associates
16
Metadata Publishing Workflow
Funnel
Metadata Harvesters
•
•
•
•
M
D
Glossary
Of Terms
Initial
Draft
Under
Review
Approved for
Publication
Develop a simple workflow system for publishing data elements
Include harvesting areas of simple glossary-of-terms found in documentation,
web sites and by using metadata “scrapers” to inventory all columns in
relational database systems
Get stakeholder teams to “accept” a data elements, review them and take on
the data stewardship role for these data elements
Commit to change-control only after data elements are marked “approved for
publication” by over 50% of the stewardship team
Copyright 2008 Dan McCreary & Associates
17
The Garden Metaphor
• Beautiful gardens don’t just grow by themselves
• They need to be designed, cultivated and
maintained
• Getting out the weeds is a key part of all beautiful
gardens (removing duplicate elements)
M
D
Copyright 2008 Dan McCreary & Associates
18
Data Stewardship
I Love Precise
Data Definitions
• An organization needs to believe that data and the management of
enterprise data is a strategic goal of the enterprise
• At it core the ongoing data stewardship process in not a function of IT,
it is an enterprise value instilled in each business unit
• We need to find data steward champions in each business unit
• IT can facilitate the maintenance of data but the business unit needs to
be empowered
M
D
Copyright 2008 Dan McCreary & Associates
19
Federated Search
• Search many sources using a single tool
M
D
Copyright 2008 Dan McCreary & Associates
20
Search Results
Internal Data Standards
External Data Standards
M
D
Copyright 2008 Dan McCreary & Associates
21
Rules
Data
Can they be separated?
Process
Rules
M
D
Copyright 2008 Dan McCreary & Associates
22
Rules and Data
Rules should always be expressed in terms
that are consistent with corporate metadata
standards
–Tae Sondreal, Business Rules Expert
But how do we show this in our POC?
M
D
Copyright 2008 Dan McCreary & Associates
23
Assume a Future SOA ESB
Adapter
Adapter
Adapter
Adapter
Enterprise Service Bus
Adapter
Adapter
Adapter
Adapter
• Each system has adapters that send and receive data in
standardized XML formats governed by an enterprise metadata
registry
M
D
Copyright 2008 Dan McCreary & Associates
24
What are SOA-Centric Rules?
Business
Event
XML
File
Exception
Exception
Exception
Rule
Rule
Rule
Happy Path
• Rule are used by any business process that
executes conditional logic on a fragment of XML
data on the Enterprise Service Bus
M
D
Copyright 2008 Dan McCreary & Associates
25
Data Type Rules
A rule that only looks at a leaf on a and
XML tree
• Example:
– Is -1.O3 a valid number?
– Is “2008-28-29” a valid date?
•
•
scope
•
•
•
•
•
M See w3c XML Schema datatypes
D
•
Rules do not depend on any
resources outside of that node
Executes quickly using streaming
memory
“Leaf-level rules”
Not applied to branches
Use w3c XML Schema datatype
standards
Most code tables under 1,000 values
Patterns validation using regular
expressions
Enterprise SimpleTypes are stored in
the metadata registry with the data
element definition
Copyright 2008 Dan McCreary & Associates
26
Structural Rules
XML
Schema.xsd
M
D
• Rules that express the structure
of a document
• What branches are valid?
• What branches are required and
how many repeating values
(cardinality)?
• Can be expressed by XML
Schema constraints
• Output is a valid/invalid
Boolean value for the entire
document
• Executed in a single pass even
for gigabyte files
• No need to keep the entire
document in working memory
Copyright 2008 Dan McCreary & Associates
27
Transformation Rules
A
B
C=A+B
M
D
C
• Rules that change an XML
document either by adding
or changing structure
• Rules split up documents or
combine documents
• Expressed as XQuery or
XSLT
• Stored in transformation
libraries that promote reuse
• Classified by the Enterprise
Integration Patterns (Content
Enrichment, Content Filter
etc.)
Copyright 2008 Dan McCreary & Associates
28
Conditional Rules
A
B
C
If C does not equal A + B then report
an error.
Rule
registry
Metadata
registry
M
D
• Rules that can be take
every node in the tree into
account
• Expressed as a Boolean
(true/false) outcome
• Used to redirect a
document in a workflow
• Expressed as XPath
expressions using
Schematron rules file and
compiled into an XSLT file
• XForms application checks
for valid data element
definitions
Copyright 2008 Dan McCreary & Associates
29
Service Rules
Web
Service
database
M
D
• Rules that require
sending a subset of
the XML data to an
external web service
• Depends on an
enterprise resource
on a remote server
• Required
consideration of
unreliable networks
and services
Copyright 2008 Dan McCreary & Associates
30
Classification of Rules in the SOA Realm
• Simple Business Rules
– Data type rules (XML Schema data types)
– Structural rules (XML Schemas, ER relationships)
– Done quickly using streaming technology (SAX)
• Medium Business Rules
– Transformation and Conditional rules
– Stated as XSLT or if-then-else expressions
– Assume and in-memory model (DOM)
• Service Rules
– Any check that requires data not easily stored in a simple XML
Schema or a rules files
– Address validation
– CIF lookup
– Legacy services require web service adapters
M
D
Copyright 2008 Dan McCreary & Associates
31
Rules in the Registry
Metadata
Registry
Property Data
Elements
SimpleTypes
Wantlists
XML Schema
Version Registry
Service
Registry
Subset
Generation
• We have datatype, structural and services in the registry but we do not
yet have a registry of transformation and conditional rules
M
D
Copyright 2008 Dan McCreary & Associates
32
Rule Registry
Rule
Registry
contains many
Rule
Set
Controlled by
Group
contains many
Rule
contains many
And (Or)
Expression
assertions
reference many
Data
Element
report
literal
•
•
•
•
M •
registered in the
Metadata
Registry
The Rule Registry would contain rule sets
Each rule set would be controlled by a group
Rule sets contain many Rules
Each Rule contains many And (Or) expressions that are either of type assertion or
report
Expressions my only reference registered data elements or literal values
D
Copyright 2008 Dan McCreary & Associates
33
Scope of Proof-of-Concept
• We can do an architectural proof-of-concept
today as long as we don’t have to deal with
network reliability issues right away
• Network reliability issues need to be
considered for production systems
• Can be simulated with “wrapper” web
services
M
D
Copyright 2008 Dan McCreary & Associates
34
Rule as a Service
Adapter
Adapter
Adapter
Adapter
Enterprise Service Bus
xml
Adapter
•
•
•
•
M
xml
Rule Engine
Adapter
•
Rule Engines can be implemented as services that hangs off of the ESB
They have XML/REST interfaces
The take XML data in and send XML data out
They can live within the eXist XQuery engine or DB2 v9 or MS-SQL 2005 or
Oracle 11g
XML Schemas define their inputs and outputs
D
Copyright 2008 Dan McCreary & Associates
35
Distributed Computing Issues
• What if a remote web service is down?
• Request will time out after timeout interval
(by default – 15 seconds)
• Ideally we would have a “watcher” that
would watch over these calls and keep
trying until the remote service returns
• How can this be added to the XQuery
environment?
M
D
Copyright 2008 Dan McCreary & Associates
36
Reliable Web Services Issues
• When you call any service on a remote
computer it gets transparently put in a local
message queue
• The XQuery waits till the result returns
• People are notified when messages are not
returned with an acceptable timeframe
• Example WS-Reliable Messaging
M
D
Copyright 2008 Dan McCreary & Associates
37
What is Orchestration?
• The ability for any web service to call other
web services
Call Web Service Y
Call Web Service X
Web
Service
Call Web Service X
Request
Response
M
D
Also known as a “composite”
Web service
Copyright 2008 Dan McCreary & Associates
38
XQuery
• XQuery is inherently a choreography
language
x.xq
xquery version "1.0";
let $y := doc(‘http://www.example1.com/service-y.xq’)/my-xpath-into-y
let $z := doc(‘http://www.example2.com/service-z.xq’)/my-xpath-into-z
return
<results>
{$y}
{$z}
</results>
M
D
Copyright 2008 Dan McCreary & Associates
39
Strategy
• Continue to build an infrastructure that is
consistent with the metadata registry
– Selection list are context sensitive
• Use simple XML-centric standards (like
XPath)
– Rules expressed in terms of metadata
• Leverage “as-you-type” user interfaces (see
suggest demo)
M
D
Copyright 2008 Dan McCreary & Associates
40
Thank You!
Please contact me for more information:
•
•
•
•
•
•
Native XML Databases
Metadata Management
Metadata Registries
Service Oriented Architectures
Business Intelligence and Data Warehouse
Semantic Web
Dan McCreary, President
Dan McCreary & Associates
Metadata Strategy Development
[email protected]
(952) 931-9198
M
D
Copyright 2008 Dan McCreary & Associates
41