Transcript Document

University of Toronto
Composite Subscriptions in
Content-based Pub/Sub Systems
Guoli Li
Agenda






2
Motivation
PADRES System
Composite Subscriptions
Case Study: Workflow Management
Evaluations
Conclusions and Future Work
Middleware 2005 Grenoble France
2015/7/17
Motivation



A large number of atomic events
Lack of event correlation
Potential Applications:
–
–
–
3
Workflow processing and business process
execution
Network management
…
Middleware 2005 Grenoble France
2015/7/17
The PADRES System


PADRES: Publish/Subscribe Applied to
Distributed REsource Scheduling
Features:
–
Predicate-based subscription language


–
Rule-based publication/subscription matching


–
4
(attribute, op, value) pair
Extended to supporting composite subscriptions
Publications are mapped to facts
Subscriptions are mapped to rules
Advertisement-based routing
Middleware 2005 Grenoble France
2015/7/17
Content-based Routing

Advertising
Advertisement
Distributed Overlay
Broker Network
Broker
…
Broker
…
Broker
5
Broker
PADRES
Broker
Middleware 2005 Grenoble France
2015/7/17
Content-based Routing

Subscribing
Subscription
Distributed Overlay
Broker Network
Broker
…
Broker
…
Broker
6
Broker
PADRES
Broker
Middleware 2005 Grenoble France
2015/7/17
Content-based Routing

Publishing
Publication
Distributed Overlay
Broker Network
Broker
…
Broker
…
Broker
7
Broker
PADRES
Broker
Middleware 2005 Grenoble France
2015/7/17
Several Concepts

Message
–
–
–

Event
–
–

Atomic event: publication in pub/sub literature
Composite event: a pattern of events
Subscriptions
–
–
8
Advertisement
Subscription
Publication
Atomic subscription: atomic event
Composite subscription: composite event
Middleware 2005 Grenoble France
2015/7/17
Composite Subscriptions

Composite subscription consists of primitive
subscriptions
–
–

Provide a higher level view for subscribers
&
Save network bandwidth
Subscription language features
–
–
–
Operators: AND (&), OR (||)
Variables: ($x)
Advanced event patterns
||
S1
S5
&
||
S2
S3
S4
CS={{S1 OR S2} AND{S3 OR S4} AND S5}
9
Middleware 2005 Grenoble France
2015/7/17
Composite Subscriptions

Parallelization pattern:
s1 & s2
e.g. {rule ((job (appl = $X) (name = A) (state = succ)) &
 Alternation
s1 ||=s2
(job pattern:
(appl = $X) (name
B) (state = succ)))
=> (forward a notification to proper destination)}
10

Sequence pattern:
(s1;[timespan:ts]s2)[within:wi]

Repetition pattern: Repetition(S, n, attr, v)
Middleware 2005 Grenoble France
2015/7/17
Composite Subscriptions

Parallelization pattern:
s1 & s2
e.g. {rule ((job …. (name = A) (time = $Y)) &
 Alternation
s1 || >s2
(job ….pattern:
(name = B) (time
$Y+ts) (time < $Y+wi)))
=> (forward a notification to proper destination)}
e.g. {rule ((job …. (name = A) (attr = $Z)) &
(job …. (name = A) (attr = $Z+v)) &
 Sequence pattern: … (s1;[timespan:ts]
& s2)[within:wi]
(job …. (name = A) (attr = $Z+(n-1)v)) )
=> (forward a notification to proper destination)}

11
Repetition pattern: Repetition(S, n, attr, v)
Middleware 2005 Grenoble France
2015/7/17
Composite Subscription Routing
P
Publishers
S
Subscribers
CS={{S1 AND S2} ANDS3}
&
CS’={S1 AND S2}
||
S1
P1
B1
Distributed Overlay
Broker Network
S1
B3
S2
B5
CS’
S3
S2
CS
S
B4
S3
B2
B6
P2
P3
12
Middleware 2005 Grenoble France
2015/7/17
Composite Event Detection
P
Publishers
S
Subscribers
CS={{S1 AND S2} ANDS3}
AND
CS’={S1 AND S2}
AND
P1
P1
S1
S1
B1
Distributed Overlay
Broker Network
P12
B3
P2
P2
B2
CS’
B5
CS
CS
S2
S
B4
P123
B6
S2
S3
P3
13
S3
Middleware 2005 Grenoble France
P3
2015/7/17
Advantages



Rule-based approach naturally enables the
composite subscription
Event detection results can be shared
Network traffic is reduced
–
–
14
Composite subscription is forwarded into the
network as far as possible before it is split
Composite events are detected close to the data
source and not widely disseminated
Middleware 2005 Grenoble France
2015/7/17
Case Study: Workflow Management


Decentralized Workflow Management based
on Pub/Sub
Functions
–
–
–
–
15
Workflow transform
Workflow deployment
Workflow execution
Workflow control and monitoring
Middleware 2005 Grenoble France
2015/7/17
Architecture
Workflow Management and Business Activity Monitoring
start
Deploy
Control
halt
resume
add
remove
Redirect
7
6
4
Visualize
Update
Business
Events
3
Monitor
...
Workflow and Business Process Execution
Business Process
Events
Content-based Routing
Business Process
Execution Events
Clients (publisher/subscriber)
Content-based Router
Computers
Computers
Laptops
Server
Computers
Switch
Server Farm
16
Server Database
Network and
System Events
Workstation
Switch
Database
Server
Computing, Storage, and Networking Resources
Middleware 2005 Grenoble France
Laptops
Event Management
Framework
2015/7/17
Workflow Descriptions




17
Workflows are described using
XML-based languages, such as
BPEL
trigger
Activities defined in a workflow
have dependencies among each
other
A workflow instance is generated
by a trigger
Failures can be handled in the
workflow description
Middleware 2005 Grenoble France
A
E
F
B
C
D
2015/7/17
Workflow Transform

The first job(s) subscribe to trigger messages
e.g. Job A subscribes to trigger publication:
{[class,eq,trigger],[workflow,eq,payroll], [instanceID,=,$x]}

Job dependencies are implied by subscriptions
e.g. Job E subscribes to job A failed:
{[class,eq,job_info],[workflow,eq,payroll], [nstanceID,=,$x], [job,eq,A], [status,eq,fail]}
e.g. Job D subscribes to B and C finished, which is a composite subscription

18
Advertisements are generated from a workflow
specification file
Middleware 2005 Grenoble France
2015/7/17
Workflow Deployment
BPEL
A
B
C
sub/advs
sub/advs
sub/advs
sub/advs
JobCD
Job
Job
Job AB
Deployer
D
Distributed Overlay
Broker Network
B1
sub/advs
Job A
Job A
Agent
B3
B4
B2
B6
sub/advs
Job B
19
B5
sub/advs
Job D
Job D
Agent
Job B
Agent
sub/advs
Job C
Middleware 2005 Grenoble France
Job C
Agent
2015/7/17
Workflow Execution (Success)
BPEL
A
B
Workflow
Manager
C
trigger
D
END
B1
Distributed Overlay
Broker Network
B3 A
A
Job A
Agent A
sub/advs
Job A
20
B5BC
B4
B2
Job D
Agent
sub/advs
Job D
B6
Job BB
Agent
sub/advs
Job B
Middleware 2005 Grenoble France
Job C
C
Agent
sub/advs
Job C
2015/7/17
Workflow Execution (Failure)
BPEL
A
E
F
B
Workflow
Manager
C
D
sub/advs
Job D
B1
Distributed Overlay
Broker Network
B3
Job A
Agent A
sub/advs
Job A
21
trigger
B5
B4
Job D
Agent
END
B2
B6
E
Job B
sub/advs Job E
sub/advs Agent
Agent
Job E
Job B
Middleware 2005 Grenoble France
Job C
Agent sub/advs
Job C
Job F
Agent
sub/advs
Job F
2015/7/17
Workflow Monitoring


Monitor is a Pub/Sub client
Monitor/control workflow execution status
–
–



22
Workflow level
Activity level
Support dynamic workflow modification
Visualize topology of broker overlay network
Visualize message routing
Middleware 2005 Grenoble France
2015/7/17
Evaluations

Environment
–
–

Workload description
–
–
–

Predicates based on 20 attributes with random operators
Values are uniformly distributed in the value range
Two workflows with different number of jobs
Metrics
–
–
–
23
Intel Xeon 3GHz, 1GB RAM
JDK 1.4.2
Matching time
Routing delay per composite subscription
Network traffic
Middleware 2005 Grenoble France
2015/7/17
Publication Matching Time
Routing Time (ms)(log scale)
1000
100
10
1
0.1
Naive Matching Algorithm
Predicate Counting
PADRES Broker with Random Data
PADRES Broker with Zipf Data
0.01
0.001
0
50,000
100,000
150,000
200,000
Number of Subscriptions
24
Middleware 2005 Grenoble France
2015/7/17
Composite Event Detection
100
Number of Matched Pubs (Uniformly Distr.)
Number of Matching Publications
450
Number of Matched Pubs (Zipf Distr.)
400
Matching Time (Uniformly Distr.)
350
Matching Time (Zipf Distr.)
10
300
250
200
1
150
100
50
25
10
0
20
0
30
0
40
0
50
0
60
0
70
0
80
0
90
0
10
00
11
00
12
00
13
00
14
00
15
00
0.1
10
0
Composite Subscription Matching Time(ms)
500
Number of Composite Subscriptions
Middleware 2005 Grenoble France
2015/7/17
Routing Delay
14
Routing Delay (ms)a
12
10
8
6
4
2
0
1
2
3
4
5
6
Number of Atomic Subscriptions per Composite Subscription
26
Middleware 2005 Grenoble France
2015/7/17
Number of Notifications
8,000
Number of Notifications
7,000
Without Composite Subscription
6,000
Composite Subscription
5,000
4,000
3,000
2,000
1,000
0
0
27
10,000
20,000
30,000
40,000
50,000
Number of Publications
Middleware 2005 Grenoble France
2015/7/17
Workflow Traffic
3500
3000
2500
Network Traffic 2000
(KB)
1500
1000
500
0
28
Workflow A (8 jobs)
Workflow B (24 jobs)
W/t Composite Sub.
1582
3264
Composite Sub.
1084
2026
Middleware 2005 Grenoble France
2015/7/17
Related Work



29
P.R. Pietzuch, B. Shand, and J. Bacon. Composite event detection as a
generic middleware extension. IEEE Network Magazine, Special Issue
on Middleware Technologies for Future Communication Networks, Jan
2004
SIENA
REBECA
Middleware 2005 Grenoble France
2015/7/17
Conclusions




An expressive subscription language
A rule-based approach of message matching
and composite event detection
Case study: a decentralized workflow
management system based on PADRES
Future Works
–
–
30
Support BPEL4WS
QoS based workflow execution
Middleware 2005 Grenoble France
2015/7/17
Questions?
31
Middleware 2005 Grenoble France
2015/7/17