Transcript Slide 1

HPTS Workshop, October 7-10, 2007, Asilomar, CA
Lessons for High Performance Services
or
Towards Transaction Processing Engineering
Dr. Alfred Z. Spector
[email protected]
Views herein are my own, and do not represent any other organization.
October 10, 2007
Towards Transaction Processing Engineering
Towards Transaction Processing Engineering
Abstract
Transaction processing as a field of engineering and science has a 50-year long record: Thus, due
consideration of the engineering lessons of systems from Saber on (and on and on) and the academic
work, say, on atomicity or replication, should now be yielding high quality, non-speculative design
principles. Perhaps some elements would be marked as understood and codified; some as explored, but
not yet fully understood; and some as emerging and still unexplored. But, there would at least be the
beginnings of an unambiguous, engineering backbone to the field.
But, the reality is that we have many too many unpooled experiences and too many proposed techniques,
structures, and algorithms on which no critical consensus has been reached. For example, I have my own
opinions on the value of our teams' work on distributed transaction management (done 10-20 years ago at
Carnegie Mellon and Transarc), but I've yet to write (or see) a broad critical analysis that attempts to put the
relevant topics conclusively to bed. There are many more examples like this.
My motivation for desiring to begin this classification is a feeling that our field (and computer science as a
whole) does not sufficiently and clearly document what we know. We are not good at reducing large
aspects of software/systems project to a more traditional type of engineering. Due to the breadth of my
recent management experiences, I recognize I may lack qualifications to contribute details to this effort,
but indeed no one can see the whole elephant. Perhaps, the right approach is to come up with shared
publication mechanisms (and needed structural elements) that can leverage the great successes of social
networking technologies to harness the collective wisdom of the community.
I will present this critique, illustrate it with categorized examples (some of which may generate debate), and
propose some conceptual structures that could lead to a better codification of the science and engineering
principles of transaction processing.
HPTS 2007
Page 2
Towards Transaction Processing Engineering
Outline
1. Our field is old, grand, and of the essence
2. Anecdotal problems
3. Deep Issues in TP
4. From “Towards a Software Science of Design1”
5. A Step for TP: The codification of what we
know
6. Conclusions
1 National
Science Foundation Science of Design PI Mtg., Arlington, VA, 3/07, Towards a Software Science of Design
HPTS 2007
Page 3
Towards Transaction Processing Engineering
The Grand Ole Field

Transaction Processing, or “service-oriented computing”, is ~50 years old:
• Highly successful through many waves:
•

• The few and the big (1st decade)
• Growth of the many (2nd two decades)
• Distributed systems (4rd decade)
• Web support, universal access, & massive utilities (5th decade)
Powered by:
• Practical engineering by computer industry and users
• Substantial researcher interest
• A great and balanced community (!)
A Strong Future:
• So much more than 6 billion/(ThinkTimeInSecs) TPS
• Goal: To minimize overhead, support optimization of all systems, and provide
universal aggregation and access of all knowledge

We envision highly interconnected systems making life better
• However, with substantial design and engineering challenges
HPTS 2007
Page 4
Towards Transaction Processing Engineering
Our Inventory of Technology Supporting the Agenda
25%
20%
Software
15%
Computer Hardware
10%
Telecommunications
5%
0%
1970
1975
1980
1985
1990
1995
2000
2004
2005
Source: Table 2.1, Current-Cost Net Stock of Private, Commercial Fixed
Assets, Equipment and Software, National Bureau of Economic Analysis
• I/T (particularly S/W) a Growing Share of U.S. Commercial Capital Stock1
• Note: Global I/T expenditure per year is roughly $1.5 x 10^12
• My analysis: Growth in I/T value is probably much steeper; why?
• In terms of value, I/T has grown much faster due to price deflation
• Software depreciation/expensing is demonstrably fast
1and
by implication, the World’s.
HPTS 2007
Page 5
Towards Transaction Processing Engineering
We Should and We Do Know a Lot, but…
 We make huge, reliable systems with great brilliance
 We install small systems to run stores with minimal fuss
 Our gurus are indeed gurus
 But,
•
•
•
•
•
•
HPTS 2007
Why are the technical leaders so overwhelming key to major, new
systems design and implementation?
Why does (or can) the industry develop many system with minimal
advances, but significant incompatibilities?
That is, why are there so many different ways to do the same thing?
Why do we regularly re-invent the wheel?
Why is it virtually impossible to get the details right?
Where is the regularity in system design?
Why are achieving reuse, failure analysis, security, etc. either so
difficult or requiring of genius?
Page 6
Towards Transaction Processing Engineering
Do We Truly Understand the Deep Issues in TP (1)?

The Data Abstraction Layer:
• Relational, XML, Text w/arbitrary annotation, Chunks, Posix
• …

The Service Runtime:
•
•
•
•

Apache and load balancing
Heavier-weight transaction a la J2EE, or Microsoft .Net
Stored procedures
…
Security and Privacy Architecture:
• Confidentiality
• Access control: data layer, service layer, …

Availability Architecture:
• Replication techniques

Problem determination:
• Architecture and techniques

Service Architecture with embedded services
• Balancing encapsulation and transparency
HPTS 2007
Page 7
Towards Transaction Processing Engineering
Deep Issues in TP (2)

Reuse
• How to support and reduce reinvention: e.g., O.O., vs. Open Source, vs…
• Huge opportunities for engineering benefits across society

The limits to atomicity
• Where, when, and how much?

Disaster recovery
• Mirroring vs. event transmission/logging

Systems management
• Lower cost management





Design for evolution
Risk analysis
Performance Analysis
Software Engineering/Project management
And many more…
HPTS 2007
Page 8
Towards Transaction Processing Engineering
Problems in TP Design are part of the Software
Design Problem (from 2004 talk I gave at MIT1)
 Score high on many
bad metrics:
•
•
•
•
•
Amount of code
# of dependencies
# of programmatic
interfaces
# of layers
Administrative interface
size & configuration
options
• Non-uniformity
• Non-orthogonality
• Defects
• Documentation
• # of programmers
•
involved
Added: dynamic change
More generally, we have a software design problem
1From
The Conudrum of Systems, MIT, 2004, http://www.csail.mit.edu/events/DLStalks/dlsspector03.html
HPTS 2007
Page 9
Towards Transaction Processing Engineering
Stepping Back: Design, Broadly1

The process of design refers to the application of synthetic and analytic
processes to plan and create new objects.
My perspective – great commonality in design across many human endeavors

•
•
•
•
•
•
•
•

Protocols
(Medical) Differential Diagnoses
Bridge/Building Design2
Proofs
Plays, Music, Novels
Urban Areas
Political Systems
Complex Systems
While design is central, it is not today a holistic discipline, studied in the same
ways as the classical humanities or even science/engineering. Should it be?
Cross-fertilization to be expected:

• AASHTO specifications for requirements of highway bridges
• Design workbenches
• Project design

Software design is often a part of broader designs so design is often
inextricably linked
1 National
2 Bridge
Science Foundation Science of Design PI Mtg., Arlington, VA, 3/07, Towards a Software Science of Design
Design and Construction, CACM 29(4): 267-283, 4/86, A. Z. Spector and D.K. Gifford
HPTS 2007
Page 10
Towards Transaction Processing Engineering
Difficulty of Software Design









Diversity of objectives and scale
Quantity of artifacts
Growing scale even for individual artificats
Post facto, dynamic assembly
Malleability
Longevity
Absence of manufacturing cost
Changing development possibilities
Difficulty of verification
HPTS 2007
Page 11
Towards Transaction Processing Engineering
A First Step in TP: The Codification of What We Know

Social Networking approaches to knowledge capture, organization, and
refinement have been more successful than many thought:
• Wikipedia is an obvious example
• Linux is another
• Other domain specific social networking sites

Grand Question:
• Could we devise a going-in structure for our field that organized requirements,
•
•
•
•
HPTS 2007
algorithms, etc…along various dimensions (Recursive application of technology)?
Could this be debated and refined over time?
Could we then provide answers
• Facts and Proofs
• Designs
• Software templates
• Code
Some tagging for elements: some would be marked as understood and codified; some
as explored, but not yet fully understood; and some as emerging and still unexplored
Could we use social networking technologies based on our systems and using our
community to put Transaction Processing Engineering on a firm foundation?
Page 12
Towards Transaction Processing Engineering
Do We Need A TP Knowledge Repository (TPKR)?


Yes, here’s why:
As a field, we don’t get to the bottom of things very well:
• Industry has challenges:
•
•

• The good (or bad) old thing
• The new, new thing
Academe has challenges:
• Oft., lack of detailed knowledge on big systems
• The new, new thing (or LPU)
• The time to really answer hard problems
The whole field is not really disciplined towards achieving design principles
TPKR could motivate:
• More discourse and information exchange
• Challenges and concomitant debate on important design issues
• A publication vehicle, more in keeping with the times than the Journal
• Contributions to the compendium of definitive information would be judged highly
• Perhaps, more so than individual journal/conference articles


The social phenomena might unleash a lot of energy
I note that Jim Gray was very interested in alternative forms of publication
(e.g., as compared to traditional journals), in my last conversation with him
HPTS 2007
Page 13
Towards Transaction Processing Engineering
Towards a Software-wide Science of Design
A Software Science of Design would facilitate software production across the
lifecycle by helping to understand, codify and integrate currently understood
approaches to design, but also by creating better approaches and tools.




A broad, integrative mission
Key foci:
• Cross-fertilization with other design disciplines
• Holistic approach
• Scientific analysis (not, “here’s a new approach, it’s real cool ”)
• Education and standardization of approaches
• Interdisciplinary work
Elements probably not included
• Domain-specific abstractions and their implementations
• Elements directly covered in other areas of the field:
• Programming Language Design
• Verification
• Etc.
• These are admittedly blurry lines
Note:
• I don’t think that S/W design, broadly, will admit to as much standardization as exists in, say,
road design
• Nonetheless, there can still be families of approaches, parameterized (or meta-), approaches &
tools…
HPTS 2007
Page 14
Towards Transaction Processing Engineering
Elements of a Research Agenda














Establishing Convincing Truths
Terminology and reference models
Metrics
Modularity and reuse
Workbenches/platforms
Development methodologies
Automated assembly
Design for change
Design for security
Complexity
Education
Team culture
Impact of accounting and taxation
Impact of liability, intellectual property, and anti-trust law
HPTS 2007
Page 15
Towards Transaction Processing Engineering
Observations and Conclusions







The state of software design is not sound. (1) We don’t know enough (2) we rarely
codify (in the broadest sense) what we know (3) we don’t consistently use or educate
what we codify.
Design is everywhere. While particularly important in software, there should be a
cross-discipline design community looking for synergies in innovation, education, and
practice.
Design in software is multi-faceted. It applies to all aspects of the lifecycle. A Science
of Design should be broad enough to address all aspects.
Design is of growing importance. Related not only to amount of software, more
serious applications of software, greater diversity of it’s development across time and
space, and post-facto, dynamic assembly, with a greater base of components, design
(but not programming) is oft. needed! Implications are manifold.
The research agenda has great opportunity. There are plenty of topics for a broad
research agenda. Work should proceed towards categorization and completeness:
tops down and bottoms up.
Interdisciplinary research is of particular important. The design of software is not
purely a problem of computer science.
Establishment of objectives. We should establish quantifiable objectives.
 And we could do a big thing in our subfield: TPKR
HPTS 2007
Page 16
Towards Transaction Processing Engineering
Thank you!
HPTS 2007
Page 17