Transcript Short Presentation Title

Systemübergreifende Prozessüberwachung
mit Event Stream Processing
Thomas Dücker, SAP Schweiz
Juni, 2013
Agenda
Herausforderung und Trends
Übersicht Operational Process Intelligence
 Übersicht
 Beispiele
 Szenario
Mehrwert für Sie
Weitere Informationen
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
2
Event Stream Processing
Capture
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
Disseminate & Decide
Response & Action
3
Traditional Approaches to Event Stream Processing
Database Application
Custom Application
“Black-box”
Application
Application
or
Dashboard
High-Speed
Data
Queries /
Results
High-Speed
Data
Alerts /
Actions
• DB application (OLTP)
— Pull not push (i.e. not event driven) or
— Requires triggers – performance degradation / maintenance nightmare
— latency in seconds
• Custom application (C/C++ / Java)
— Specialized, high development / maintenance cost
— Slow to change, unresponsive to the business
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
4
The SAP Approach – Sybase ESP
Input Streams
PUBLISH
Visual / SQL
Authoring
Output Streams
SUBSCRIBE
Market Data
Orders, Trades
Console
Input
Reporting Tools
Dashboards
?
Trading Systems
Message Bus
Reference Data
SAP HANA / Sybase IQ
• Combines advantages of customized application with ease of visual data flow diagramming or SQL
• Incoming data is processed as it arrives, according to the model rules and operations
• Publish streaming results to other models, apps, message bus, dashboard, etc.
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
5
CEP Product Convergence
Coral8 v5.6.5
Sybase CEP
R4
Aleri 3.2
Sept 2011
Sybase ESP
5.0
Sept 2012
Sybase ESP
5.1
Dec 2012
Sybase ESP
5.1 SP1
Optional
(maintenance only)
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
6
Advantages of Sybase ESP Approach
Analyze Events as they occur
• Continuous Insight
• Respond Immediately
Rapid Application Development
• Reduce/eliminate dependence on specialist
programming skills for process
• Cut implementation/deployment time
• Broad out of the box connectivity
Non-intrusive Deployment
• Event-driven integration with existing systems
• Unify existing disparate data models
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
7
Event Stream Processing Scenarios
Capital Markets
React to incoming ticks to determine trading
strategies, monitor the success of the strategies
by calculating PnL in real-time; monitor risk in
real-time.
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
Retail
Provide clickstream analysis for online customer
assistance / alternatives, use web scraping to
react to competitor price changes
8
Event Stream Processing Scenarios
Energy
Monitor and optimize energy usage using Smart
Metering to ensure QoS
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
Utilities / Manufacturing
Service Status Monitoring and Process Control
using SCADA / OSISoft
9
Event Stream Processing Scenarios
Telecoms
Monitor events for QoS, use DPI to detect
fraudulent use
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
Police / Homeland Security
ANPR event correlation for traffic monitoring,
security
10
ESP Architektur
ESP Project Architecture
Authoring Tools
Operational Console
Log
Store
C+C
Command & Control
(XML-RPC)
Studio
Security (PAM, SSL, RSA, Kerberos)
Publish
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
Data Streams
Adapters
Mem
Store
I/O Gateway
Stream Processor
+ Multi-threaded
+ 64 bit
- Suse 11 / Redhat 6 /
Solaris 10 (Sparc / i86)
Win x64
+ Low latency
+ Optional persistence
Security
Security
Adapters
Data Streams
I/O Gateway
Data Streams
Subscribe
Data Streams
Log
Store
12
Out of the Box Connectivity
In Process
•
•
•
•
JMS, MQ, TIBCO
ATOM (RSS)
TCP Sockets
Databases via JDBC /
ODBC
• Files
• MS Excel RT
• SMTP (out)
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
External
• Market Data
(Reuters, Bloomberg, IDC,
ACTIV, NYSE Tech)
• FIX
• OpenAdaptor Java
Framework
• Replication Server
• IQ / HANA
PubSub SDK
• C/C++
• Java
• .NET
13
ESP Cluster Architecture I
Manager
Nodes Cache
Manager
Manager
Manager
Manager
Controller
Project Server
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
Controller
Project Server
Controller
Project Server
14
ESP Cluster Architecture II
•
Manager Node
− Deploys project(s) to server(s)
− Maintains heartbeat with project(s) to detect failure
− Manages failover
− In multi node, managers are run in a cache so can failover to manage other
controller nodes
•
Controller Node
− Basically acts as a launch daemon for Manager to start project(s) in a Container
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
15
ESP Cluster Architecture III
•
Single node cluster –
− Single manager/controller node runs multiple projects on a single machine
− Detects when a project fails and tries to restart it
− This is the default for project development
•
Multi node cluster
− Multiple equi-peer managers and controllers run multiple projects on multiple machines
− Used to scale out projects across multiple machines
− Allows for server failure – promotes failover recovery and data redundancy
− Allows for Highly Available “active:active” deployment of projects within a Data Centre
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
16
ESP Installed Clusters
•
Three clusters created by install script: -
•
Studio Cluster (single node) on 9786 : –
− Used for Studio Development
− Can run multiple projects
− Located in $ESP_HOME/studio/clustercfg
•
User-defined Cluster (single node) on 19011
− Located in $ESP_HOME/cluster/nodes/node1
•
Example multi-node Cluster on 19011 - 19014
− Located in $ESP_HOME/cluster/examples
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
17
ESP Key Concepts
Authoring Options – Eclipse Plugin
Visual Dataflow Authoring
CCL Authoring
•
Analyst-level skills (Excel, VBA)
•
Rapid programming
•
Easy to understand complex models
•
Easy to use language (CCL – derived from SQL)
•
No need to learn language syntax
•
Modular, project based approach
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
19
Visual Dataflow Authoring
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
20
CCL Authoring
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
21
Procedural Logic - SPLASH
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
22
ESP Modularity
IMPORT 'CreateCompleteChannel.ccl';
---------CREATE MODULE CreateCompleteChannel
IN MessageIn, CacheIn
OUT MessageOut
BEGIN
// Module Code
END;
LOAD MODULE CreateCompleteChannel AS Channel_A
IN MessageIn = MessageA, CacheIn = CacheA
OUT MessageOut = MessageOutA;
LOAD MODULE CreateCompleteChannel AS Channel_B
IN MessageIn = MessageB, CacheIn = CacheB
OUT MessageOut = MessageOutB;
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
23
Project Code, Compiled Code & Project Resources
•
Project Source Code .CCL (Continuous Computation Language)
− Used for schema, stream, window, method definitions compiled into
•
Project Executable .CCX (Continuous Computation eXecutable)
•
Project Resource File .CCR (Continuous Computation Resource)
− An xml file used for in process adaptor, cluster, bindings, failover, HA definitions –
allows for easy migration between DEV, TEST and PROD
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
24
Constructing ESP Projects
•
Build model
•
Connect to data feeds using publish adaptor
•
Connect to output devices via subscribe adaptor
Tick data feed
ESP Project Container
Trades
DB Replicated Data
Console
Positions
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
25
Stateful Events
•
Events are retained as records in memory with a primary key so that further events may modify the
state of the record - either update or delete it.
•
A stateful event must have PK attribute(s) set.
•
A stateful event MUST have an associated operation code (opcode), one of INSERT, UPDATE, UPSERT,
DELETE. If no opcode is provided, ESP defaults it to UPSERT.
•
When an event occurs it applies the opcode through the data flow model depending on the opcode.
An insert in a source stream may become an update to an aggregate or join stream.
•
The full set of unique (insert) events to be retained MUST fit in memory.
•
Records in memory can be removed either by setting a retention window, or by publishing a DELETE
event for that key.
•
In ESP, stateful events are implemented using a WINDOW
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
26
Stateless Events
•
An event is retained only while it is being processed. When an event moves from one stream to
another it is removed from the first stream so it cannot be updated.
•
All events are treated as insert.
•
A stateless event ignores any PK attribute(s) and always outputs an insert
•
In ESP stateless events are implemented with a STREAM
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
27
ESP Elements
ESP implements stateful and stateless events using 3 element types
•
WINDOW : used for stateful events. A WINDOW maintains state for a user defined period
•
STREAM : used for stateless events. A STREAM does not maintain or react to state
•
DELTA : used to process stateful events, but without maintaining state
•
All can be INPUT, LOCAL, OUTPUT
•
INPUT subscribe to input adaptors i.e. events are published to an INPUT. They are the entry points
for events to enter an ESP project.
•
LOCAL perform processing logic. They can NOT be published or subscribed to
•
OUTPUT are derived elements which perform processing logic which CAN be subscribed to
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
28
Windows (stateful)
•
WINDOWS must have a PK set
•
Can have an optional retention period set
•
Maintain state in memory or persisted to disk. i.e. State can be recovered in the event of system
failure
•
Can publish to STREAM, WINDOW or DELTA
•
Only a WINDOW can be used for aggregation
•
At lease one side of a join MUST be a WINDOW (unless using SPLASH). The side which maintains
state depends on the type of the join.
•
Use WINDOW when developing models as they help debugging
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
29
Streams (stateless)
•
Stream
•
Cannot be persisted so any data is lost in the event of system failure
•
Can NOT publish to DELTA stream
•
Can publish to a WINDOW, but WINDOW must aggregate the input
•
Delta
•
Can only be LOCAL, OUTPUT
•
A stateless element (a STREAM) that understands and processes state (a WINDOW)
•
Can only use a WINDOW as a source
•
Must have a PK set
•
As it does not store events it can NOT be used for aggregation.
•
Cannot be persisted so any data is lost in the event of system failure
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
30
Example: Simple Position Maintenance
Trades Stream
Symbol*
Time*
Price
Shares
Aggregate
AverageTrades Window
Symbol*
Last Price
Weighted Avg Price
Last Time
Positions By Book Stream
Position Window
BookId*
Symbol*
Current Position
Average Position
Join
Total Book Window
Aggregate
BookId*
Current Position
Average Position
BookId*
Symbol*
Shares Held
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
31
Example: Simple Position Maintenance
Trades Stream
Symbol*
Time*
Price
Shares
MSFT
12:05:23
32.45
200
IBM
12:05:24
57.87
5000
CSCO
12:06:01
20.04
100
MSFT
12:06:02
36.55
900
Aggregate
MSFT
32.45
36.55
33.01
34.00
12:05:23
12:06:02
Book1
MSFT
97350
109650
99030
102000
AveragePrices Window
IBM
57.87
56.90
12:05:24
Book1
IBM
40509
40509
3983039830
Symbol*
Last Price
Weighted Avg Price
Last Time
CSCO
20.04
17.00
12:06:01
Book2
MSFT
32450
36550
3301034000
Individual Positions Stream
BookId*
Symbol*
Current Position
Average Position
Join
Position Window
BookId*
Symbol*
Shares Held
Book1
MSFT
3000
Book1
IBM
700
Book2
MSFT
1000
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
Book Positions Window
Aggregate
BookId*
Current Position
Average Position
Book1
150159
139539
141830
138860
Book2
36550
32450
34000
33010
32
ESP Solution Architektur
Event Processing Solution Architecture
•
Events don’t happen in isolation
• Most Event Stream Scenarios need access to historic data
• Pattern Detection
• Risk Calculation
• Optimisation algorithm
• To ensure sub millisecond response times use in memory e.g. HANA
•
Many Event Stream Scenarios need access to master data
• Data written directly to OLTP system
• Updates to reference, semi-static data
• For sub second updates without polling use log trawler e.g. Replication Server
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
34
Real-time ANPR Alerting Framework
OCR Radars
SQL
Event Stream Processor
Output
Adapter
Input
Adapter
Monitoring
Business Inteligence
CEP Engine
SQL
HANA
In-database Algorithms
BO Predictive Analysis
Predictive models
(Ad-hoc analysis)
Analytical data
Historic data
Reference data
SQL
BusinessObjects DataService
External
Databases
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
35
Real-time Retailing Framework
Detect
Event
Validate
Event
HANA
ESP
GUI
Is competitor
price<95% of our
price?
Manage
Exception
Straight Through
Processing
Run Optimization
Algorithm
Check
Results
Output
Is optimum price<95% of
current our price?
Phase 1:
Phase 2:
Phase 3:
Phase 4:
Event Detection & Validation
Processing
Validation & Exception
Output
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
36
Integration with Hana I
•
Generic input / output ODBC adapter
• Data can enter via a source stream
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
37
Integration with Hana II
•
Can be called from a FlexStream using SPLASH for event driven query: -
ON TRADE {
if (TRADE.Price <> optimalPrice) {
[ string s; | ] rec;
vector (typeof(rec) v := new vector(typeof(rec));
getData(v,'HanaDSN','CALL DPTL.sp_price_opt (?, ?, ?, ?)', TRADE.Id, TRADE.Datetime, TRADE.Sector, TRADE.Price);
if (not (isnull(v))) {
newrec := v[0];
}
// rest of method
}
}
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
38
Real-time Risk Framework
Trading
Trading
Systems
Systems
Continual Mark t o Market
Transformation of
granular data
•
•
•
CEP
Real-time Aggregation
of Risk and P&L
Cleansing
Normalisation
Enrichment
Bus
Trading
Market
Data
Systems
Analytics:
• Heatmaps
• Drill-downs
• Reports
Replication Server
Trading
Risk
Systems
Sensitivities
•
ASE
•
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
HANA
Reference Data
•
Scenario Shifts
VAR calculations
Showing:
• Concentration limits
• Clustering
• Outliers
• Etc…
By:
• Instrument
• Industry
• Country
• Debt Rating
• Counterparty
• VaR Time Horizon
• Etc…
39
Oil & Gas - Energy Efficiency Reporting Framework
Oracle
RA#1
RS
(Replication
Server)
HANA
ESP
(Event Stream Processor)
RS
HANA
ODBC
SOI2
Oracle
RA#2
SOI2
server
SCADA
SCADA
SCADA
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
Replication flow
Event flow
40
Besten Dank
Kontaktinformation:
Thomas Dücker
SAP Schweiz
[email protected]
© 2013 SAP AG or an SAP affiliate company. All rights reserved.