Business Unit Empowerment Through Declarative Systems

Download Report

Transcript Business Unit Empowerment Through Declarative Systems 

Declarative Programming and
the Business Unit
Strategies for Custom Application Cost
Reduction
Dan McCreary
Dan McCreary & Associates
September 11, 2007
M
D
Metadata Solutions
DISCLAIMER
• The following views are my own and do
not necessarily reflect the views of the
Minnesota Department of Revenue or the
State of Minnesota
M
D
2
Presentation Abstract
M
Title: Declarative Programming and the Business Unit
Description: For the last few years software development has seen a
dramatic growth in the use of XML in all aspects of information
systems. What began as a convenient way to store structured
documents and server configuration files has morphed into complete
systems that are built using only XML and XQuery.
The amount of procedural code written in new systems is dropping
dramatically. This has had a large impact on the ability of business
units to create and maintain their own requirements and business logic
without the need for an extensive software development staff.
This presentation discusses how XML has allowed business units to
precisely capture the "what" (sometime graphically) without the need to
be concerned with the "how" systems are implemented. We will
present a full example of a rich-client web application that has been
defined and built using only a pure "declarative" style of model-driven
software development. These technologies include CSS, XForms, XML
Schemas, XML Transforms, XQuery and business rules. We will also
discuss how pushing this power out to the business unit requires new
data stewardship and business semantics skills to be emphasized.
D
3
Audience Survey
• How many people here…
– Work for organizations that have web sites?
– Are currently creating web applications?
– Are familiar with XML syntax?
– Have heard of the term “declarative”?
– Are moving toward service orientation?
– Have an enterprise service bus?
– Have a metadata registry?
– Regularly do ROI analysis on emerging
technologies?
M
D
4
Incoming!
Has this web
thing gone
away yet?
M
D
5
How did we get here?
• From punch cards…
to tables…
to trees…
to cubes and graphs…
M
D
6
In the Beginning…
There were punch cards…
…and there were 80 columns…
and the widths were fixed…
and batches of cards were called "decks“…
and they were loaded into tables in “core” memory…
M
D
7
And Codd and Date said…
• Let there be relationships…
Person
person_id
first_name
last_name
birth_date
Role
person_id
project_id
role_nm
time_pct
Project
project_id
project_nm
description
status
and we had the relational databases…
and we had SQL…
and we had joins…
and we had ER diagrams…
and we had PowerBuilder…
and all seemed right with the world…until
M
D
8
Sir Berners-Lee Said…
• Let there be HTML…
<html>
<head>
<title>Dan’s Homepage</title>
</head>
<body>
<h1>Welcome!</title>
<p>Welcome to my web site!</p>
<a href="about.htm">About Me</a>
</body>
</html>
and we had URIs…
and we had HTTP…
and we had we links and resources…
and we had trees (not tables)…
and we had search…until
M
D
9
Customers Said..
• We wanted to access our tables…on the
web…in HTML!
html
?
head
body
h1 p
a
and there were objects…
and there was Java and JSP … and there was ASP.Net
and there were models, views, controllers…
and there was conversion from forms to objects to tables
and tables to objects to forms…
and there were rooms and rooms of programmers…
M
D
10
The Translation “Pain Chain”
Name:
Street:
City:
Zip:
Web Forms
Objects
RDBMS
• From web forms to objects…to SQL inserts…to
selects…to objects and back to web forms
– Many format translations…
– Impedance mismatch
M
D
11
And there Was Chaos…
HTML Forms
html
Objects
JSP/ASP
head
body
h1 p
Hibernate
a
Serialization
100+ Frameworks
OR Mapping
JavaScript
Struts
XML
JDBC
SQL
.Net
AJAX
M
D
Sessions
Stored Procedures
And the number of languages you need to
learn and the headcount of the IT department
went up…and up
RDBMS
12
And the W3C Said…
• Let their be simplicity…what if we…allowed you
to put XML in the browser…and database
html
head
Database
body
model
input
Person
label
FirstName
LastName
Projects
Roles
Project
Database Vendors that Support XQuery
• IBM DB2 Version 9
“PureXML”
• Microsoft SQL
Server 2005
• Oracle 10g Release
2 Enterprise Edition
• eXist (open source)
• + 50 others…
M
D
14
Evolution Metaphors
• Specialization of Languages
• Generalization of Languages
M
D
15
Evolution: Specialization
• Darwin’s Galapagos Finches
• Beaks are highly adapted to different food sources
• Finches adapted to specific ecological "niches“ over
millions of years of isolated evolution
• Similar to domain-specific declarative languages
M
D
See Wikipedia "Darwin's Finches"16
Evolution: Generalization
• Generalization: The Raccoon
– The world has a higher population of raccoons today
due to their ability to quickly adapt to changing urban
environments
– Similar to highly adaptive procedural languages
M
D
17
Computer Science Abstractions
Higher abstractions
Declarative
<xf:input>
<xf:label>
Object
Class MyClass(
Method MyMethod
Structured
Fortran
Assembly
Function(A, B)
FOR I = 1 TO 10 DO
1010001010
M
D
time
18
The Software Development Process
Requirements
Test
(QA Staff)
(BA)
Declarative Languages
Build
Design
(Programmer)
(Architect)
• Requirements are about “What”
• Design and Build is about “How”
M
D
19
A Declarative “System” Is…
• A software development system, tailored to a specific domain
(such as web applications), used to capture precise business
requirements within the context of a problem domain (the
implicit context)
• Declarative systems do not specify how requirements are
implemented to build working systems. Declarative systems
only define the requirements
• Declarative systems document requirements in specialized
vocabularies and can be used to generate entire working
systems including user interfaces, persistence and test data
• Declarative systems specifically omit some assumed
requirements (such as system availability, performance,
reliability, security etc)
A Declarative system is a set of "little languages" with precise
semantics that fit together like a puzzle to solve a problem
M
D
20
HTML, CSS and SQL
<?xml version="1.0" encoding="UTF-8"?>
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>Declarative Systems</title>
</head>
<body>
<h1>Introduction to Declarative Systems</h1>
<p class="author">Dan McCreary</p>
<p class="date">April 2007</p>
</body>
</html>
/* global CSS used by all web pages */
body {
font-family: Arial, Helvetica, sans-serif;
font-size: 75%;
margin: 0;
padding: 0;
width: 1000px;
}
h1 {
color: blue;
padding: 0 15px;
}
HTML
CSS
xquery version "1.0";
<html><body>
<h1>Old Expensive Books</h1>
<ul>{
for $book in doc("books.xml")//book
order by $book/title
return
<li>
{$book /title},
{$book}/author},
{$book}/price},
{$book}/pubyear}
</li>
}</ul></body></html>
XQuery
http://www.w3.org/TR/REC-html32#body
• HTML, CSS, XQuery and SQL are
declarative languages using these
definitions
• The semantics or “meaning” of
each tag in the file is determined by
an external organization
• The syntax does not have to be
XML
M
D
-- Old expensive books
SELECT title, author, price, pubyear
FROM books
WHERE
price > 100 AND
pubyear BEFORE ‘1960’
ORDER BY title
SQL
21
XML Schema Sample
• Screen capture of Altova XML Spy
• 30 minutes to learn graphical notation
M
D
See: http://www.altova.com/products/xmlspy/graphical_xml_schema_editor.html
22
XML Schema (XMLSpy Diagram)
Solid line
means required
Data Element
Sequence
of items
Definition
Dashed line
means optional
Cardinality
Graphical notation that most non-programmers can learn in 20 minutes.
M
D
23
Use Case: Electronic Forms
• User fills out a webbased form
• Typical requirements
may include listing of
data to be gathered,
data types, validation
codes and data
repetition patterns
• Examples:
– HTML Forms
– XForms
– InfoPath™
M
D
24
Forms Players
<html>
<head>
<xf:model>
<xf:/model>
<head>
<body>
<xf:input>
</xf:input>
</body>
</html>
XForms
Extension
Mobile Client
Netfront Mobile
Forms Server
My-XForm.xhtml
Rendered
Form
XForms
“Players”
M
D
See Wikipedia “XForms”25
What Is Declarativeness for A Context?
Schema Drawing Tool
XML Schema File (.xsd)
XForms
xf|input:required
{font-weight: bold;}
xf|input:required .xf-value
{background-color:#fff6af;}
*:required::after
{font-weight:bold;
font-size:1.5em;
content: "*"; color: red; }
XForms CSS tags
• Efficiency at capturing
the testable business
requirements in a
semantically precise
and concise manner
• Example: Is a field
optional? Will
validation of the data
fail if the field is
missing?
• Ability of each data
element to have precise
meaning over time and
within organizations
M
D
26
Declarative Spectrum
General Purpose
Narrow Purpose
Less Abstract
More Abstract
Apache Ant
C#
Assembly
Language
C
C++
Java
JavaScript
perl
Python
Groovy
Ruby
XSL
XQuery
XUL
CSS
HTML
XML Schema
XForms
XQuery Update
XPath
SQL
• For any given context different “languages” have
different levels of “declarativeness”
• General purpose languages are less abstract but can
solve a wide variety of problems
• Declarative languages have a more narrow purpose
and target a specific problem like styling a web page or
selecting data
M
D
27
Middle-Tiers Remain Procedural
declarativeness
user
database
Presentation/style
(client tier)
•
•
•
•
M
D
Business Logic
(middle tier)
Data definition, insert,
selection and update
(persistence tier)
When the interface is consistent, declarative languages flourish
Middle tiers tend to have the most variation
Wikipedia lists over 200 web application frameworks
CMSMatrix.org lists over 700 content management systems
28
See: http://en.wikipedia.org/wiki/List_of_web_application_frameworks and http://www.cmsmatrix.org
The Application Development Puzzle
Semantics
Constraints
Presentation
Workflow
Query
Update
Forms
Publish
Build
Transform
M
D
29
Filling In Each Piece
With A Declarative Language
Metadata
Registry
BPEL
XMLSchema
CSS
HTML
XQuery
XQuery Update
XForms
Cocoon
Apache
Ant
XSL
M
D
30
Semantically Precise Vocabularies
M
Standard
Domain
Atom/RSS
Syndication/Notification
ACORD
Insurance Industry Forms Data
FpML
Financial Products Markup Language
FixML
Banking (Financial Information eXchange)
HL7
Healthcare (now using XML)
MISMO
Mortgage Industry
NIEM
National Information Exchange Model (registry)
OTA
OpenTravel Alliance (Travel industry)
SIF
School Interoperability Framework
GJXDM
US Department of Justice (registry)
XBRL
XML Business Reporting Language
See: http://en.wikipedia.org/wiki/Category:XML-based_standards
D
31
Metadata Shopping Tools
Phone
Address
FirstName
• You don’t need to know about
100,000 SKUs to purchase 10
items from a grocery store
• Sub-schema generation tools give
you exactly what you need and
nothing more
M
D
See http://niem.gtri.gatech.edu/iepd-ssgt/SSGT-SearchSubmit.do32
M
D
33
If You Use Industry Standards…
You Could Be Almost Done…
• If you use industry standards…
– and these standards publish their documents in XML
Schema format…
– and these standards have been transformed from XML
Schema to XForms…
– and you use native XML databases to store and
XQuery to report on the data…
• …then sample applications have been created
and do not require additional procedural code
– just change the constraints in the XML Schema and
rerun the transforms
See: http://www.exist-db.org
M
D
See also: http://www.alphaworks.ibm.com/tech/purexml 34
Accessibility
• Architecture and
Strategy
(prevent unnecessary
procedural code)
IT
Architects
Procedural
Programmers
Business Analysts
SMEs and GUI Tools Users
M
D
• Extend declarative
vocabularies and
provide web service
“glue”
• Precisely specify
business requirements
• Requires data stewardship
training
Lower costs by moving routine logic maintenance to lower levels in the pyramid
35
Java Libraries
10,000 class and 100,000+ methods available
…but which ones are relevant to your business problem?
M
D
36
“Less is More”
• XForms 1.1 has
only 21 XML
elements
• Much of the
presentation of
XForms is
deferred to CSS
• Event
management is
deferred to the
XML Events
• XML binding is
deferred to the
XBL standard
M
D
Mies van der Rohe
Reconstruction of the
German Pavilion in Barcelona
See Wikipedia “Minimalism”
37
XForms is Only 21 Elements
1. Bind
2. Case
3. Input
4. Instance
5. Group
6. Help
7. Hint
8. Label
9. Load
10. Output
M
D
11. Message
12. Model
13. Repeat
14. Secret
15. Select
16. Select1
17. Switch
18. Submission
19. Submit
20. Textarea
21. Trigger
Color coding limited vocabularies can increase the speed of pattern matching.
Look for advanced text editors to provide custom element coloring.38
The New Semantics of "Nutshell"
• The 1.4 release of Java 2
Standard edition increases
the size of the platform by
50%, to 2,757 classes in
135 packages
• 1.5 and 1.6 add additional
classes
992 pages
nutshell: something of small size, amount, or scope
in a nutshell : in a very brief statement
M
D
39
Procedural Programming is Not All “Poison”
• It would be a mistake to tell all
your procedural programmers that
the programs they are creating are
fundamentally evil
• The relevant questions are:
- How closely does it fit the problem
domain?
- Can BAs, SMEs and other nonprogrammers maintain the business
rules?
- What are the chances that others will
be able to maintain it in future years?
- How good are the development tools
for your system?
M
D
40
If You Give a Kid a Hammer…
…the whole world becomes a nail
• People solve problems using
familiar tools
• People develop specific
Cognitive Styles* based on
training and experience
• What are we teaching the next
generation of developers?
M
* Source: Shoshana Zuboff: In the Age of the Smart Machine (1988)
D
41
Semantics
• The science of meaning
• What you mean when you say “cat”
• How do you associate meaning with
symbols (verbal, physical, textual)
• How do we know if we both mean the
same thing when we use a word?
• What if a word has multiple meanings?
Reference: WordNet
M
D
42
Semantic Triangle
concept
“cat”
referent
symbol
• Symbols can only link to referents through concepts
• You can not link directly from a symbol to a referent
M
Wikipedia: Semiotic triangle
D
43
Communication
“cat”
• Domestic feline
• House pet
• Has fur
• Has whiskers
• Sometimes has fleas
• Chases mice
• Domestic feline
• House pet
• Has fur
• Has whiskers
• Sometimes has fleas
• Chases mice
• Communication involves exchanging symbols that
describe common attributes
• A one-to-one match of attributes that describe a common
symbol match implies a high precision match
M
D
44
Same symbol – different meaning
“customer”
• First name
• Last name
• Home address
• E-mail
• Cell phone number
• Gender
• Company name
• Home office address
• Branch office address
• CEO name
• Web site
• Industry code
• Little match of common attributes
• Low precision semantic match
M
D
45
High and Low Precision
precise standard
precise usage
best
precise standard
vague usage
better
vague standard
vague usage
Not-so-good
• The ideal is to have a precise standard and to use the
data elements exactly as they were intended
M
D
46
Semantic Precision
• Semantic mappings are relative in time and
between groups of people (organization)
• Semantic variability over time
– Something that has precise meaning to you today
may not have the same precise meaning a year from
now
– Our memories are imperfect and change over time
• Semantic variability across organizations
(project and organization)
– A “customer” to one organization may denote a
person but to another organization it may denote a
company
M
D
47
Semantics of an XML Data Element
<code>47</code>
• A developer puts an XML data element in an xml file
• The tag has some meaning and the data within the tag
has some meaning when it was created by the developer
• What is the probability
– That the developer will know the same meaning of the code 47
one year later
– That another project that opens file will understand and be able
to use the meaning of the tag
– Vague standards often trigger vague interpretations of the
meaning of data
M
D
48
The Heart of the Enterprise
The Metadata Registry
A metadata registry is a central location in an organization where
metadata definitions are stored and maintained in a controlled
method.
M
http://en.wikipedia.org/wiki/Metadata_registry
D
49
Semantic Precision in Space and Time
space: (projects, organizations)
Large
Semantic
Footprint
(long lifetime
systems)
world
enterprise
dept.
team
person
M
Small Semantic
Footprint
(rapid prototype)
weeks
D
time
months
years
10+ years
50
Use Procedural “Glue”
XForms XML
(REST)
Procedural
“Glue”
XQuery Update
(JDBC)
• XForms uses a REST interface to send XML to a server
• Some XML databases (DB2 v9) sill use JDBC interfaces to insert
XML documents
• Use procedural glue to build custom interfaces between systems
with incompatible interfaces
M
D
51
How Quickly Can We Create New
Declarative Languages?
• Configuration files are really just small
languages
• Configuration files are easy to parse,
validate data using XML tools and build
custom forms to use
• Easy to teach non-programmers to
graphically build XML Schemas to validate
XML files
– Answer – About a week
M
D
52
How Quickly Can We Create
Consensus?
• Example: XForms standard
– Work started in 2001
– XForms 2.0 is still a two to three years away
• How can we accelerate this consensus
building process?
What factors impact the rate that species evolve?
What impact does life span have on a species?
How are design ideas exchanged between species?
M
D
53
Solution: Wikis and Collaboration
• How long does it take to build consensus on the
semantics of a new data element?
• How many people might use this declarative
language?
• The larger the stakeholder group, the longer it takes
M
D
54
“In the Can” vs. “On The Wire”
• Make my data insert
and update quickly
• Extract, Transform
and Load (ETL)
• SQL Skills
Name:
• Create shared
meaning between
systems
• Register the meaning
in a metadata registry
• XML Skills
Street:
City:
Zip:
Enterprise Service Bus
Metadata
M
D
55
Strategic Consequences
• Empowerment of business units to build
and maintain their own applications with
much less involvement of IT staff
• Dramatically reduces the need for
“middleware” IT staff using traditional
procedural languages like Java and C#
M
D
56
Strategic Impact
• Consider this:
– 50% of Applications are simple CRUD Applications
– Create, Read, Update, Delete
• Complex business logic can be stored in an
XML-based rules engine such as Schematron
• How will this transform an organization?
• How will it allow business units to build and
maintain IT applications?
• How will it allow the business to be responsive to
customer needs?
M
D
57
Change Where the Line is Drawn
Requirements
Requirements
BAs
SME
Developers
vs.
Graphical Requirements and Specifications
SME/BA
M
IT Staff
• Shorten the “distance” between the business
unit and the IT staff
D
58
Many Small vs. One Large
CSS
XForms
XSLT
XPath
XQuery
XML Schema
Requirements
Business Rules
It is easier to build a GUI on a small, precise
language than a single large language
M
D
59
Data Stewardship/Governence
• Importance of the business units role in
defining data elements semantics “on-thewire”
• How to write and maintain enterprise
definitions
– Concise
– Precise
– Distinct
– Non-circular
– Unencumbered
M
D
60
Relative
Code Base
Parker Projection
100%
Time
M
D
Source: Jason Parker, Minnesota Department of Revenue, November 2006
61
Recommendations
1.
2.
3.
4.
5.
6.
7.
8.
9.
Use semantically-precise declarative systems over procedural
systems when the business problem is in the right context
Aggressively promote data stewardships within business units
Use controlled vocabularies and metadata shopping tools to allow
non-programmers to select data elements from lists
Separate semantics (leaf elements) from constraints (branches)
Leverage the pattern matching portion of the right brain using
graphical development tools and element-based syntax coloring
editors
Use procedural code to build interfaces between incompatible
systems
Isolate procedural by providing XML/REST service wrappers code
behind semantically precise web services
Don’t expect your procedural programmers to jump on the declarative
bandwagon without some training
Use social networking tools (voting, wikis) to rank and find
appropriate data elements
M
D
62
Next Steps For IT Strategists
1. Read some articles on the semantic web
(Scientific American)
2. Read a book on business semantics
3. Convince yourself it works:
Google for “XForms Examples”
Google for “XQuery Example”
4. Create a pilot project
Pick a "friendly" business unit
Find a medium-sized problem
5. Train staff with XPath-centric tools
6. Roll out to the enterprise in stages
M
D
63
XForms Wikibook
http://en.wikibooks.org/
wiki/Xforms
Over 75 working
XForms example
programs
M
D
64
Devx Article
• Semantics and the Evolution of
Specialized Languages
http://www.devx.com/semantic/Article/34805
M
D
65
Thank You!
Dan McCreary
Dan McCreary & Associates
Enterprise Data Architecture Consulting
[email protected]
http://www.danmccreary.com
Cell: (612) 986-1552
Home: (952) 931-9198
M
D
66