Transcript ConferenceCode_SessionCode

Event-Driven Applications: Where they Apply
and How they are Built
Gartner Application Integration and
Web Services Summit
K. Mani Chandy
California Institute of Technology
[email protected]
Special thanks to Roy Schulte and David Luckham.
Notes accompany this presentation. Please select Notes Page view.
Mani Chandy, Caltech
EDA Business Value
Respond to events – threats and opportunities.
Mani Chandy, Caltech
Take Away 1: EDA Characteristics
1.
Monitors and correlates events outside and
inside the enterprise.
2.
No communication IMPLIES reality matches
model.
Mani Chandy, Caltech
Take Away 2: You’re ready for EDA now
3.
Your company is already event driven.
4.
You already have the components of the
architecture in your enterprise stack, and it’s
complementary to SOA.
Key Question for you:
Do the incremental benefits of IT for EDA exceed
its incremental costs?
Mani Chandy, Caltech
Organisms Sense and Respond: EDA
1)
Streams of data: sight, sound,
touch, smell, taste
2)
Central nervous system detects
the rare event that is a threat or
an opportunity
3)
Organism responds appropriately
to threat or opportunity
4)
Too many false positives, a false
negative, or too many delayed or
inappropriate responses results in
death
Mani Chandy, Caltech
Group Sense and Respond: EDA

Three enterprises: lions, hyenas, zebras
Critical conditions: fuse information from inside and
outside the enterprise

Mani Chandy, Caltech
External Awareness: Questions for you

Does your enterprise monitor its competitors?
Government agencies?

Do people in your enterprise correlate
information from multiple sources? e.g.,
correlate flood at a supplier’s factory with
deadlines for critical customers.
Mani Chandy, Caltech
Responding to Unexpected Situations
Question for you:

A fire has just occurred in a factory that is
going to effect customers severely.

Which of two scenarios represents your
enterprise?
1.
2.
The CEO doesn’t expect VP Mfg to say
anything unless the CEO asks.
The CEO expects VP Mfg to tell the CEO.
Mani Chandy, Caltech
No Communication => Reality = Model
1.
CEO has an expectation – a model – of the
CFO:
 No communication implies reality matches the model.
2.
Space shuttle director has a model for the
engineer responsible for shuttle foam insulation.
 No communication implies reality matches the model.
Mani Chandy, Caltech
Enterprise-Wide Situational Awareness
Military situational awareness
Corporate situational awareness
London
Edmonton
Corporate
VP, risk
Denver
NY, NY
Scheduler cockpit
Risk management
cockpit
Trader
cockpit
Houston
Sydney
Risk manager
Houston
Mani Chandy, Caltech
Take-Away Points
1.
EDA: Global situational awareness
2.
EDA: Respond when reality deviates from model.

Your enterprise is:
3.
Already event-directed.
4.
Has the software components including SOA

Question for you: incremental costs vs. benefits?
4:05 PM
Mani Chandy, Caltech
What is EDA?

System that manages and executes rules of the
form:
WHEN reality deviates significantly from
expectations
THEN initiate appropriate response.
Mani Chandy, Caltech
EDA Characteristics
Respond
Sense
Detect events across
extended environment in
real-time
Invoke distributed services
in real-time
Analyze
Aggregate events across multiple
sources
Mani Chandy, Caltech
EDA: Software Characteristics
1.
Asynchronous coupling.
 The timing, place, and characteristics of threats and
opportunities are not determined by you.
 So sensors are responsible for pushing information;
responders are not responsible for pulling it.
2.
Defensive programming to the extreme.
 Data may be unstructured and inaccurate. Protocols
may be unspecified.
Mani Chandy, Caltech
Defensive Programming to the Extreme
Airline
A
Monitoring
Airline
B
• One airline can make few assumptions about another airline.
• EDA should be very robust; or it is very brittle
• The robustness comes at a price
• EDA is at the limit of coupling “looseness”
Mani Chandy, Caltech
Defensive Programming to the Extreme
Division
Monitoring
Division
A
B
• One division can make few assumptions about another
division.
• EDA should be very robust
• The robustness comes at a price
• EDA is at the limit of coupling “looseness”
Mani Chandy, Caltech
EDA Structure: Sense, Analyze, Respond
Mani Chandy, Caltech
Defensive Programs: Sensors & Responders
SENSORS
PROGRAM
OUTWARD-FACING
COMPONENTS
EXTREMELY DEFENSIVELY
RESPONDERS
Mani Chandy, Caltech
Less Defensive: Event Processing Network
PROGRAM
INWARD-FACING
COMPONENTS
LESS
DEFENSIVELY
Mani Chandy, Caltech
Compare EDA Requirements with SOA

SOA: Components are collaborators
– Accounting “client” calls a “sales pipeline expectation”
method on a sales “service” which returns with a
report

SOA: Time of interaction determined by client

SOA: Service protocols and schemas are well
defined.

SOA: Units obtain global situational awareness
by invoking multiple services.
Mani Chandy, Caltech
EDA: Software Characteristics - Review
1.
Asynchronous coupling.
 The timing, place, and characteristics of threats and
opportunities are not determined by you.
 So sensors are responsible for pushing information;
responders are not responsible for pulling it.
2.
Defensive programming to the extreme.
 Data may be unstructured and inaccurate. Protocols
may be unspecified.
3.
Sense – Analyze – Respond.
4.
When-Then rules.
Mani Chandy, Caltech
An Event-Driven Architecture
Hostile
Web sites
Screen
Scrape
News
feeds
News
Handler
Stock
tickers
Ticker
Handler
Scheduler
Applications
Application
Handler
Electronic
Markets
Market
Interaction
DB
Database
Interaction
Web sites
Web
Service
Alert
Engine
End Users
Complex
Event
Patterns
E
Approximate
Matching
S
B
Text
Analysis
Parametric
Analysis
Time series
Analysis
When-Then Rule Mgmt
System Configuration
Monitoring
Mani Chandy, Caltech
Take-Away 2 (repeated)

Your enterprise already has most of the
components of the architecture:
– EAI tools; Messaging, ESB or event distributors;
Databases; Alerts engines; Web Services; Rules
engines


EDA is compatible with and complements SOA.
Your enterprise is already event-directed.
Your key question: Do the incremental benefits
exceed the incremental costs?
Mani Chandy, Caltech
Inevitable Down-Sides of EDA

Effort required to specify and tune rules.

Errors:
1.
False positives: Response to non-event
Drowning in Events: “turn that darn thing off!”
2.
False negatives: Non-response to event
Why are these down-sides inevitable?
Mani Chandy, Caltech
Inevitable EDA Down-Sides: Errors

When-then rules are imperfectly specified.

When clauses evaluated incorrectly or late.

Changing rules to reduce false positives
increases false negatives (and vice-versa).
Mani Chandy, Caltech
Inevitable EDA Down-Sides: Drown in Events

Most responses alert people.

Perception: false positives cost much less than
false negatives.

Too many false positives: “so, turn that thing off.”
Correct evaluation should be based on total costs
of all the false positives and negatives over time.
Mani Chandy, Caltech
Inevitable Down-Sides of EDA: No Rules

People are busy.

Learning tools for specifying rules takes time.

So, rules aren’t specified, or are specified in a
cursory fashion.

Destroys business value of EDA.
Mani Chandy, Caltech
Rule Development Solutions
1.
Have IT (or some central org.) specify rules.
 Doesn’t work.
2.
Work with business users to specify rule
templates; individuals fill in templates.
3.
Have role-based rule repositories.
Mani Chandy, Caltech
Is Anything Missing in your Software Stack?

1.
2.
Event Process Agents
Machine can “learn” the critical condition from
positive and negative examples
Users can specify critical conditions
– SQL-like queries
– Fuzzy matches
– Statistical operators
– Regular expressions
– CEP
Mani Chandy, Caltech
Estimating Performance Requirements





Delay from occurrence of condition to initiation of
response: Minutes? Sub-seconds?
Number of data sources: Tens, Hundreds?
Numbers of rule templates: Tens, Hundreds?
Numbers of users?
Numbers of rules?
Observation: Many enterprises overshoot: they
estimate greater performance requirements than
they need.
Mani Chandy, Caltech
Yes Use Your Existing Software Stack If

Delay from occurrence of condition to initiation of
response: Tens of seconds

Number of data sources: Ten,

Numbers of rule templates: Ten,

Numbers of users? 100s

Numbers of rules? 1000s
This is the more common case.
Mani Chandy, Caltech
Speed of EDA uptake

1.
Enterprises are event-driven. So, why is EDA
uptake slow?
Simple event processing is widely used.
 Enterprise service buses and EAI tools allow when
clauses on single documents (events).
2.
Multiple event streams are correlated in some
applications
But these apps are not seen as part of an EDA
paradigm.
Mani Chandy, Caltech
Speed of EDA uptake

Correlation across multiple event streams in:
1.
Financial trading; IT Infrastructure
management; Plant control; Defense

Why these spaces?
1.
Small designated groups responsible for
responding to critical events.
2.
Clear additional value
3.
Performance
Mani Chandy, Caltech
Should you build EDA?

Your company is already event driven; you have
many of the software components. Question:
Do incremental benefits exceed incremental costs?
Mani Chandy, Caltech
Should you build EDA? Do you have:
1.
Applications:
 Apps that sense and respond to the environment and
that benefit from automation?
 Evolving middleware that can benefit from
asynchronous coupling?
2.
Responsibility in a single group or LOB?
3.
Performance requirements not met?
4.
Small numbers of data sources and rule
templates satisfying large numbers of users?
Mani Chandy, Caltech
Should you build EDA?

Are costs of false positives and negatives
smaller than benefits of EDA?

Do you have a senior manager in a LOB who
will make all the hard business work happen?
(Building effective EDA is 95% effort by business
people and only 5% effort in technology.)
Mani Chandy, Caltech
Future Talks

How not to build EDA. Lessons from the
trenches!

Steps and gotchas in building EDA.
Mani Chandy, Caltech
Take-Away Points and THANKS!
1.
2.
EDA: Global situational awareness
EDA: Unit “knows” reality matches model until
the unit gets a message.
4.
Your enterprise:
Is already event-directed.
Has the software components including SOA

Your Question: incremental costs vs. benefits?

3.
Mani Chandy, Caltech
Getting Business Users to Specify Rules is Hard

Different roles have different sets of rules. Who
specifies the rules?

How many rule templates? Tens, hundreds?

How many rules? Hundreds, thousands?

How are rules specified? Language, visual UI,
positive and negative examples? Fill in templates?

Who can turn a rule off?

What if rules are at cross-purposes?
Mani Chandy, Caltech