Transcript Performance of a possible Grid Message Infrastructure Edinburgh e-Science Centre December 9 2002 PTLIU Laboratory for Community Grids Geoffrey Fox, Shrideep Pallickara Computer Science, Informatics,

Performance of a possible Grid
Message Infrastructure
Edinburgh e-Science Centre December 9 2002
PTLIU Laboratory for Community Grids
Geoffrey Fox, Shrideep Pallickara
Computer Science, Informatics, Physics
Indiana University, Bloomington IN 47404
http://grids.ucs.indiana.edu/ptliupages
[email protected]
11/6/2015
uri="http://www.naradabrokering.org"
email="[email protected]"
1
Some Basic Observations/Goals



Grids manage and share asynchronous resources in a
rather centralized fashion
Peer-to-peer networks are “just like” Grids with
different implementations of services like registration
and look-up
Web Services interact with messages and Everything
will be a Web Service
• Microsoft Office will be a WS?

Computers are fast and getting faster. One can afford
many strategies that used to be unrealistic
• Software/Application-level message processing/routing
sensitive to performance information
• Uniform messaging framework for Middle-tier, “High
performance layer”, Multimedia (audio-video) etc.
11/6/2015
uri="http://www.naradabrokering.org" email="[email protected]"
2



Deduction?
Grids or P2P Networks consist of a sea of message-based Services
Services inject and extract messages whose transport and
manipulation is support by a logically distinct “MessageGrid” of
brokers/routers supplying “Grid Message Layer”
The message brokers support adaptive routing, filtering,
workflow
• They have publish/subscribe queued connection semantics
• They separate logical and actual transport
• They form a federated XML database and support
asynchronous collaboration
• They process real-time messages in about a millisecond and
support synchronous collaboration

NaradaBrokering is a prototype of this idea
11/6/2015
uri="http://www.naradabrokering.org" email="[email protected]"
3
Peers
Database
Database
Service Facing
Web Service Interfaces
Event/
Message
Brokers
Event/
Message
Brokers
Event/
Message
Brokers
Peer to Peer Grid
Peers
User Facing
Peer to Peer Grid
of
Resources supported by
Web Service Interfaces
internal “MessageGrid” of message or event brokers
Events are “just” Time-stamped messages
11/6/2015
uri="http://www.naradabrokering.org" email="[email protected]"
4
Shared Input Port (Replicated WS) Collaboration
Collaboration as a WS
Set up Session with XGSP
R
U
Web
F
Servic
I
I
e
O
O
F
WS
Viewer
WS
Display
Master
U
Web
F
Servic
I
I
e
O
O
F
Event
(Message)
Service
R
R
U
Web
F
Servic
I
I
e
O
O
F
11/6/2015
WS
Viewer
WS
Display
Other
Participants
WS
Viewer
uri="http://www.naradabrokering.org" email="[email protected]"
WS
Display
5
NaradaBrokering

Based on a network of cooperating broker nodes
• Cluster based architecture allows system to scale to arbitrary
size


Originally designed to provide uniform software
multicast to support real-time collaboration linked to
publish-subscribe for asynchronous systems.
Now has four major core functions
• Message transport (based on performance measurement) in
heterogeneous multi-link fashion
• General publish-subscribe including JMS & JXTA and
support for RTP-based audio/video conferencing
• Filtering for heterogeneous clients
• Federation of multiple instances of Grid services
11/6/2015
uri="http://www.naradabrokering.org" email="[email protected]"
6
Role of Event/Message Brokers



We will use events and messages interchangeably
• An event is a time stamped message
Our systems are built from clients, servers and “event brokers”
• These are logical functions – a given computer can have one
or more of these functions
• In P2P networks, computers typically multifunction; in Grids
one tends to have separate function computers
• Event Brokers “just” provide message/event services; servers
provide traditional distributed object services as Web services
There are functionalities that only depend on event itself and
perhaps the data format; they do not depend on details of
application and can be shared among several applications
• NaradaBrokering is designed to provide these functionalities
• MPI provided such functionalities for all parallel computing
11/6/2015
uri="http://www.naradabrokering.org" email="[email protected]"
7
Engineering Issues Addressed
by Event / Messaging Service





Application level Quality of Service
– e.g. give audio highest priority
Tunnel through firewalls & proxies
Filter messages to slow (collaborative/real-time) clients
Choose Hardware or Software multicast
Scaling of software multicast
• Efficient calculation of destinations and routes.




Integrate synchronous and asynchronous collaboration with
same messaging, control, archiving for all functions
Supports local broker accesses
Transparently replace single server JMS systems with a
distributed solution.
Provides reliable inter-peer group messaging for JXTA
11/6/2015
uri="http://www.naradabrokering.org" email="[email protected]"
8
Features of Event Service


MPI nowadays aims at a microsecond latency
The Event Web Service aims at a millisecond (computer) latency
• Typical distributed system travel times are many milliseconds (to seconds
for Geosynchronous satellites)
• Different performance/functionality trade-off





Messages are not sent directly from P to S but rather from P to
Broker B and from Broker B (via other brokers) to subscriber S.
Synchronous systems: B acts as a real-time router/filterer
• Messages can be archived and software multicast
Asynchronous systems: B acts as a database & workflow engine
Subscription is in each case, roughly equivalent to a database
query
Company X sets up a firewall
• The event service sets up brokers either side of firewall to optimize
transport through the firewall.
11/6/2015
uri="http://www.naradabrokering.org" email="[email protected]"
9
Narada Broker Network
(P2P) Community
For message/events service
Broker
Broker
(P2P) Community
Resource
Hypercube topology
Broker
for brokers?
Broker
Tree for distance education
with teacherBroker
at root
(P2P) Community
Data
base
Software multicast
Broker
(P2P) Community
11/6/2015
uri="http://www.naradabrokering.org" email="[email protected]"
10
Performance in Message-based Architecture I
A
Satellite
UDP
Firewall
HTTP
Fast
Link
Hand-Held
Protocol B2
Software Multicast



B1
Dial-up
Filter
Useful analogies with transportation grids
B3
and parallel computing
In traveling from cities A to B (say 3 separate passengers), one
chooses between and changes transport mechanism at
waystations to optimize cost, time, comfort, scenic beauty …
Waystations are now NB brokers where one chooses transport
protocol
• Able to choose between car, type of car, plane, train etc
• Able to dynamically create waystations to cope with problems
and acts as hubs for multicast messages
• Knows about traffic jams and can assign the “HOV lane”
11/6/2015
uri="http://www.naradabrokering.org" email="[email protected]"
11
Performance in Message-based Architecture II


Application level QoS – can optimize among managed
streams (audio versus video) using performance
subsystem
• This is just a variant of the NP complete load
balancing problem of parallel computing where all
reasonable heuristics worked
• Load-balance in Space-time (strings) not just Space
(particles)
“Performance” needs to measured carefully as
includes QoS
• I delayed shared application update to ensure audio
quality and filtered image to lower resolution
• So “application” has changed to satisfy performance
constraints
11/6/2015
uri="http://www.naradabrokering.org" email="[email protected]"
12
NaradaBrokering Communication




Applications interface to NaradaBrokering through UserChannels
which NB constructs as a set of links between NB Broker waystations
which may need to be dynamically instantiated
UserChannels have publish/subscribe semantics with XML topics
Links implement a single conventional “data” protocol.
• Interface to add new transport protocols within the Framework
• Administrative channel negotiates the best available communication
protocol for each link
Different links can have different underlying transport implementations
• Implementations in the current release include support for
TCP,UDP, Multicast, SSL and RTP.
HTTP, HTTPS support will be available in Feb 2003 release.
• Supports communication through proxies such as iPlanet, Netscape
and Apache.
• Supports communication through firewalls such as Microsoft ISA,
Checkpoint.
11/6/2015
uri="http://www.naradabrokering.org" email="[email protected]"
13
Types of Performance Measurement





“Core Performance”: Equivalent of node Clock Cycle
and node to node MPI Ping/Bandwidth measurements
Cosmic single number which does not satisfy anybody
but nevertheless dominates discussion: Equivalent to
bisection bandwidth or LINPACK
Detailed performance numbers for particular Grid
applications: e.g. Time to assimilate global weather
data and simulate hurricane
We will present “core performance” numbers for
NaradaBrokering running in different modes
Will not discuss “Web Service” performance – rather
performance of Message substrate
11/6/2015
uri="http://www.naradabrokering.org" email="[email protected]"
14
Note on Optimization

Note in parallel computing, couldn’t do much dynamic
optimization as aiming at microsecond latency
• Natural to use hardware routing

In Grid, time scales are different
• 100 millisecond quite normal network latency
• 30 millisecond typical packet time sensitivity (this is one audio
or video frame) but even here can buffer 10-100 frames on
client (conferencing to streaming)
• 1 millisecond is time for a Java server to “think”


Jitter in latency (transit time) due to routing, processing
(in NB) or packet loss recovery is important property
Grid needs and can tolerate significant dynamic
optimization
11/6/2015
uri="http://www.naradabrokering.org" email="[email protected]"
15
Transit delay for message samples in NaradaBrokering
Different communication hops - Internal Machines
5.5
hop-2
hop-3
hop-5
hop-7
5
4.5
4
3.5
3
Sender/receiver/broker - (Pentium-3, 1
GHz, 256 MB RAM). 100 Mbps LAN.
JDK-1.3, Red Hat Linux 7.3
2.5
2
1.5
1
0.5
50
11/6/2015
100
150
200
250
300
350
Message Payload Size
(Bytes)
uri="http://www.naradabrokering.org" email="[email protected]"
400
450
500
16
Transit delay for message samples in NaradaBrokering
Different communication hops - Internal Machines
9
8
7
hop-2
hop-3
hop-5
hop-7
Sender/receiver/broker - (Pentium-3, 1
GHz, 256 MB RAM). 100 Mbps LAN.
JDK-1.3, Red Hat Linux 7.3
6
5
4
3
2
1
1000
11/6/2015
1500
2000
2500
3000
3500
4000
Message Payload Size
(Bytes)
uri="http://www.naradabrokering.org" email="[email protected]"
4500
5000
17
Standard Deviation for message samples in NaradaBrokering
Different communication hops - Internal Machines
0.8
hop-2
hop-3
hop-5
hop-7
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
50
11/6/2015
100
150
200
250
300
350
Message Payload Size
(Bytes)
uri="http://www.naradabrokering.org" email="[email protected]"
400
450
500
18
Standard Deviation for message samples in NaradaBrokering
Different communication hops - Internal Machines
0.8
hop-2
hop-3
hop-5
hop-7
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
1000
11/6/2015
1500
2000
2500
3000
3500
4000
Message Payload Size
(Bytes)
uri="http://www.naradabrokering.org" email="[email protected]"
4500
5000
19
Jitter for message samples in NaradaBrokering
Different communication hops - Internal Machines
0.9
hop-2
hop-3
hop-5
hop-7
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
11/6/2015
0
100
200
300
400 500 600 700
Packet Number
uri="http://www.naradabrokering.org" email="[email protected]"
800
900
20
80
JMF-RTP
NaradaBrokering-RTP
H.263 video file
70
Average bit-rate of 600Kbps.
Frame-rate of 30 frames/sec
60
50
40
Jitter J = J + (|D(i-1, i)| - J)/16
30
20
10
0
0
200
400
600
800
1000
1200
1400
1600
1800
Packet Number
11/6/2015
uri="http://www.naradabrokering.org" email="[email protected]"
21
SSL Overhead I



The test program simulated a SSL client transferring
a 5 MB chunk of data to an SSL server.
Test Environment
• 800 MHz PIII
• JDK 1.3, JSSE 1.0.3
• Numbers are approximate over 5 iterations.
SSL Results
• SSL Socket Creation:
~250 ms
• SSL Handshaking: ~500 ms
• Transfer of 5 MB: ~2050 ms
11/6/2015
uri="http://www.naradabrokering.org" email="[email protected]"
22
SSL Overhead (ISA HTTPS Proxy)






Test Environment was as before except going through Microsoft
ISA’s HTTPS proxy in between the client and the server. The
proxy machine’s configuration was:
• Pentium 900 MHz, 512 MB RAM.
• Microsoft ISA proxy Enterprise Edition allow HTTPS protocol
through
• Proxy => Server network bandwidth 10 Mbps.
SSL Socket Creation: ~250 ms
SSL Handshaking (after 1st one):~400 ms
Transfer of 50 MB:
~50 seconds (1 MB / second)
Proxy CPU Usage:
~7% on the average
Client CPU Usage:
~20% on the average
11/6/2015
uri="http://www.naradabrokering.org" email="[email protected]"
23
JMS Performance: Transit Delay and Std
Deviation (NaradaBrokering & SonicMQ)
Transit Delays for Message Samples in Narada and SonicMQ
Mean
Transit Delay
(MilliSeconds)
30
25
20
15
10
5
0
0
Transit Delay
50
100
150
200
250
Publish Rate 300
(Messages/sec) 350400
Narada
SonicMQ
550
500
450
400
350
300
250 Payload Size
200
150
(Bytes)
100
Standard Deviation in the Message Samples - Narada and SonicMQ
Standard
Deviation
(MilliSeconds)
14
12
10
8
6
4
2
0
0
Standard Deviation
50
100
150
200
250
Publish Rate 300
(Messages/sec) 350400
550
500
450
400
350
300
250 Payload Size
200
150
(Bytes)
100
• NaradaBrokering is Java Message Service (JMS) compliant
• Applications ported include
•Anabas – JMS based distance education system.
•OKC – Online Knowledge Center project at IU
11/6/2015
uri="http://www.naradabrokering.org" email="[email protected]"
Narada
SonicMQ
24
NaradaBrokering
and JXTA
High end "long lived"/
persistent resources
Narada-JXTA provides
JXTA guaranteed long
distance delivery
NARADAJXTA proxy
NARADA
broker cloud
Narada JXTA Event
For duplicate
detection
within
NARADA
Narada Headers
Peers
Interaction UUID
JXTA Interaction
Type
Allows the
NARADA
system to
route event to
specific JXTA
proxies
11/6/2015
JXTA
Rendezvous
PEER
Request/
Dynamic/fluid
Response
peer groups
Peer group id
Peer id
JXTA Event
Payload
Narada Event
Distribution Traces
Present
if the
Request/Response
interaction is
trageted to a
specific
Peer if targeted
Present
Particular peer
at
uri="http://www.naradabrokering.org" email="[email protected]"
25
Transit delay for message samples in Narada, JXTA & Narada-JXTA
Topology - I (2 routers) Internal Machines
140
NaradaBr
Pure JXTA
Narada-JXTA
120
100
80
60
N
NaradaBrokering broker
R
JXTA Rendezvous
NaradaBrokering client
JXTA Peer
N
40
20
0
500
11/6/2015
R
R
(a)
R
R
(b)
N
N
(c)
1000 1500 2000 2500 3000 3500 4000 4500 5000 5500
Message Payload Size
(Bytes)
uri="http://www.naradabrokering.org" email="[email protected]"
26
Transit delay for message samples in Narada, JXTA and Narada-JXTA
Topology - III (8 routers) Internal Machines
300
250
R
R
R
R
R
R
R
200
R
(a)
R
N
R
150
R
R
R
R
R
R
R
R
R
R
N
NaradaBrokering broker
R
JXTA Rendezvous
(b)
100
RN
N
N
50
NaradaBr
Pure JXTA
Narada-JXTA
N
(a)
NR
N
R
R
N
JXTA Peer
N
NaradaBrokering client
(c)
R
0
500
N
N
R
1000 1500 2000
2500 3000 3500 4000 4500 5000 5500
R
R
Message Payload Size
N
(Bytes)
uri="http://www.naradabrokering.org" email="[email protected]"
(b)
N
11/6/2015
27
Transit delay for message samples in NaradaBr, JXTA & Narada-JXTA
Topology - II (6 routers) External Machines
400
R
NaradaBr
Pure JXTA
Narada-JXTA
R
350
R
R
300
R
R
(a)
R
R
N
NaradaBrokering broker
R
JXTA Rendezvous
JXTA Peer
NaradaBrokering client
250
R
N
N
R
R
R
(b)
200
N
N
N
N
150
N
N
(c)
100
50
11/6/2015
100
150
200
250
300
350
400
Message Payload Size
(Bytes)
uri="http://www.naradabrokering.org" email="[email protected]"
450
500
550
28
Narada Performance Web Service
Performance measurements are
used by Links in
• Reconfiguring Connectivity
between nodes
• Deciding underlying transport
protocol
• Determining possible filtering
Each node determines performance
of links of which it is endpoint
Individual node web services are
aggregated as another Web Service



Probably should replace by a more
sophisticated measurement package
Factors measured include

Transit delays, bandwidth, Jitter, Receiving rates.

Performance measurements are
• Spaced out at increasing intervals for healthy
channels.
• Factors selectively measured for unhealthy
channels.
Administrative Interface
• No repeated measurements of bandwidth for
example.
• Injected into Narada
network as XML events
11/6/2015
uri="http://www.naradabrokering.org"
email="[email protected]"
29

Control Channel (TCP)
Specifics tunnel
destination, parameters.
[ SOAP port 80 ]
Config
Specifics default
tunnel destination,
parameters.
Non-Firewall
Proxy
CTL
SSL Tunnel
Server
Proxy
TCP
SSL
SSL Tunnel
Lib
Client Proxy
API
UDP
Fake SSL
Impl.
JSSE
Impl.
WinINET
Impl.
UDP
TCP
Firewall
Proxy
Proxy Detection API
Text Config
WinINET
Detection
Narada Link Firewall Architecture
11/6/2015
Required for MS
Authentication
support.
Firewall
uri="http://www.naradabrokering.org" email="[email protected]"
Required for
Proxy location
detection
30
Stream Media
Types
Works
Start
UDP
Doesn’t Work
Reliable Data
Stream
Start
Works
TCP
Doesn’t Work
WinINET
Try SSL first then HTTP
Windows
?
Doesn’t Work
NaradaBrokering
Link Transport
Yes
Does HTTPS
Firewall
Proxy Exist
Heuristic
Works
Works
Try SSL Over
HTTPS
Proxy
Connection
Complete
Doesn’t Work
Does HTTP
Proxy Exist
Works
Try HTTP
Over HTTP
Proxy
Yes
Doesn’t Work
“Fake
” SSL
Over
Direct
Doesn’t
Work
Try SSL Over
Direct
Doesn’t
Work
Try HTTP
Direct
Works
Works
11/6/2015
uri="http://www.naradabrokering.org" email="[email protected]"
31
Futures










Redo performance with insight of this workshop – we
desperately need a distributed testbed to support scaling tests
Test Scaling with construction of dynamic NB broker network
(what is best topology? Hypercube like?)
Understand clients per broker for various applications
including large multimedia sessions
Complete transport/performance infrastructure
Develop nifty “load balancing heuristics”
Implement Web Service message-based Security
Understand better how to implement distributed XML
database
Develop NB to support federation of Grid Services
Generalize Filtering mechanism
Download from http://www.naradabrokering.org
11/6/2015
uri="http://www.naradabrokering.org" email="[email protected]"
32
Architecture of Message Layer


Need to optimize not only routing of particular messages but
classic publish/subscribe problem of integrating different
requests with related topics (subscribe to sports/basketball/lakers
and sports)
Related to Akamai, AOL … caching and Server optimization
problem
Hypercube of
NB Brokers (logical
not physical)
N≈100 for
Distance
Education
Scale to
billions of
grid nodes?
1-> N Grid Nodes
11/6/2015
uri="http://www.naradabrokering.org" email="[email protected]"
33
NaradaBrokering as an XML database

NB is inevitably a distributed XML database
supporting real-time update and access (JMS uses SQL
like syntax)
• Performance data
• Event Logging
• Published “properties” of publish/subscribe messages
 Publish as XML topic; subscribe using XQuery?

We use NB as a simple XML database for News groups
and other “XML nuggets” see
http://www.xmlnuggets.org
• XML Instance==Message; what is difference between a
message and a database record except performance but
Moore’s law is rewriting rules here

Subscribe a real database (Oracle) to all topics for
reliable storage
11/6/2015
uri="http://www.naradabrokering.org" email="[email protected]"
34
Federation of Services




If you have a service – Notification (as with Grid or JXTA
advertisements), Search, Scheduling, Audio-Video conferencing
….
With a standard which client and server components
Then can build a “server” that services all clients
Alternatively can hierarchically consider collection of existing
Server-client domains
•
•
•
•

IBM or Globus OGSA islands
Sun Grid Engine Schedulers
Polycom/Access Grid A/V sessions
Enterprise Search Engines
NB
Hub
Federation links islands together
• JXTA Search has XML specified federation approach – will try to include
and generalize as a NaradaBrokering federation framework
• DoD High Level Architecture HLA does this for simulation
11/6/2015
uri="http://www.naradabrokering.org" email="[email protected]"
35