Transcript Cognitive Publish/Subscribe for Heterogeneous Clouds Šarūnas Girdzijauskas, Swedish Institute of Computer Science (SICS) [email protected] Joint work with: Fatemeh Rahimian (SICS)

Cognitive Publish/Subscribe for
Heterogeneous Clouds
Šarūnas Girdzijauskas,
Swedish Institute of Computer Science (SICS)
[email protected]
Joint work with:
Fatemeh Rahimian (SICS)
Future Clouds?
• Based on decentralized
architecture
• Abundance of networked
collection of connected devices
forming micro-clouds
• Decentralized Publish/Subscribe
service
–
–
–
–
–
Content distribution
IP TV Streaming
Online gaming
Collaborative editing
Etc..
• Adapting to the topology and
network dynamics of microclouds
• Adapting to different usage
patterns
Šarūnas Girdzijauskas, Cloud Futures, Redmond,
April 2010
Pub/Sub Systems: Our Focus
• Scalable pub/sub service
– Very large number of nodes
– Very large number of “topics”
– Heterogeneous environments
• Arbitrary geographical distribution
• Arbitrary subscription and dissemination patterns
– Central solutions will not scale
Šarūnas Girdzijauskas, Cloud Futures, Redmond,
April 2010
Pub/Sub Systems: Our Focus (2)

Tradeoffs:





Node degree
Number of uninterested (relay) nodes involved
Dissemination delay
Dissemination cost
Cognitive pub/sub:



Fixed node degree
Account for the underlying topology (bandwidth & cost)
Minimize the number of relay nodes by exploiting user
subscription correlation & event publication rates
Šarūnas Girdzijauskas, Cloud Futures, Redmond,
April 2010
Conceptual Architecture
Pub/sub Dissemination
structures
Cognitive
Overlay
Physical
Network
Šarūnas Girdzijauskas, Cloud Futures, Redmond,
April 2010
Gossip based pub/sub
• Gossip (epidemic) overlays
– A lot of research (e.g., Cyclon, T-man)
– Lightweight, scalable and robust mechanism
– Cyclic/Periodic, pair-wise interaction between
peers (bounded amount of information)
Šarūnas Girdzijauskas, Cloud Futures, Redmond,
April 2010
Towards Cognitive Structure
• Gossiping enables us to find and cluster peers with
similar interests connected by cheap and fast links
– A node starts with a local fixed size view in a
random network
– Performs a bidirectional exchange of the view with
a random node  2 views
– Keeps the only the preferred (ranking function)
nodes in the view  1 view
– Repeat
Šarūnas Girdzijauskas, Cloud Futures, Redmond,
April 2010
Building Cognitive Structure
Gossiping
• Making clusters by utilizing ranking
function which prefers neighbors
with similar interests
• Peer interest similarity metric
– Node subscriptions s1, s2 ⊆ T
– sim(s1, s2) =|s1∩ s2|/|s1∪ s2|
• Weighted by link cost (bandwidth
and $)
• Weighted by Topic publication rates
$
– Number of neighbors is limited!
• Decided locally on each peer
Šarūnas Girdzijauskas, Cloud Futures, Redmond,
April 2010
Problem: How to publish?
• Clustering peers of similar
interests into bandwidth
and cost effective clusters
– Clusters might (will) be
disjoint
– Event publishing requires
connected components for
each topic
Šarūnas Girdzijauskas, Cloud Futures, Redmond,
April 2010
Inter-Cluster Connectivity
Navigable Small-World Network
14
29
12
24
20
7
34
32
16
2
31
15
22
28
1
27
23
4
19
35
26
30
18
• Structure is added:
Navigable Small-World
topology
– Purely by using gossiping
21
8
3
10
9
6
5
13
17
11
Šarūnas Girdzijauskas, Cloud Futures, Redmond,
April 2010
Building Navigable Structure
• Every peer decides on
random ID
• Updating ranking function
for choice of neighbors:
– Ring Link(s)
– Long-Range link (SmallWorld style) for
polylogarithmic routing
performance
2
5
8
1
4
3
7
6
10
Šarūnas Girdzijauskas, Cloud Futures, Redmond,
April 2010
9
Inter-Cluster Connectivity
Navigable Small-World Network
14
29
12
24
20
7
34
32
16
2
31
21
8
15
22
3
28
1
27
26
30
23
4
19
35
10
9
6
5
13
17
11
18
• Structure is added
(Navigable Small-World
made by gossiping)
– Ring Links
– Long-Range (finger) link(s)
– Clustering (friend) links
• Clusters are connected by
greedy routes
– Rendezvous node for each
topic
– All links are used!
• All topics become
connected
– For publishing “flood the
topic”, or
– Choose a rendezvous node
to publish
Šarūnas Girdzijauskas, Cloud Futures, Redmond,
April 2010
Ongoing work
•
•
•
•
Synthetic data sets for user subscription correlation
Twitter data set
Skype churn data
Our experiments show:
– Up to 10 fold reduction of relay traffic as compared to
existing approaches (e.g., Scribe, Bayeux)
Šarūnas Girdzijauskas, Cloud Futures, Redmond,
April 2010
Gossip based pub/sub (recap)
• Large scale pub/sub for heterogeneous environments
• Dissemination structures are self-organizing
– Forming clusters of similar nodes
– Converging into least expensive dissemination paths on
the underlying physical network
– Continuously adapting to the environment conditions
– Fast convergence, robustness to churn and failures.
Šarūnas Girdzijauskas, Cloud Futures, Redmond,
April 2010
Thank you!
Questions?
Šarūnas Girdzijauskas, Cloud
Futures, Redmond, April 2010