Scripting based architecture for Management of Streams and Services in Real-time Grid Applications

Authors

Harshawardhan Gadgil, Geoffrey Fox, Shrideep Pallickara, Marlon Pierce Community Grids Lab, Indiana University, Bloomington Robert Granat NASA, Jet Propulsion Lab, Pasadena

Presented By

Harshawardhan Gadgil [email protected]

Talk Outline

 Introduction to HPSearch Architecture  A quick view of NaradaBrokering middleware  System Goals – RDAHMM Example  Related and Future Work Community Grids Lab Presentation for CCGrid 2005 May 10, 2005 2

HPSearch

  A JavaScript based scripting runtime serving as an administration console – Currently uses Rhino (http://www.mozilla.org/rhino) implementation of JavaScript.

Management viewed as – Setting up the up the broker network, initializing system components) – Querying distributed application Run-time System Metadata (Involves setting  For logging purposes  Monitor metadata to help dynamically rewire system to improve performance May 10, 2005 Community Grids Lab Presentation for CCGrid 2005 3

HPSearch Architecture

Component Summary   Binds URIs to a scripting language – Allows us to manage (manipulate) the Resource identified by the URI    E.g. Read / Write to files, sockets, topics Read from data base, data from FTP/HTTP resource This data can then be streamed to distributed application OR data can be read from a stream and processed / stored “Host-objects” allow us to dynamically access the host system. – Useful for constructing objects that monitor system meta-data, perform management tasks –

PerfMetrics

gathers system performance data and allows us to query it at run-time.

Community Grids Lab Presentation for CCGrid 2005 May 10, 2005 4

HPSearch Kernel Request Handler JavaScript Shell Task Scheduler Flow Handler Web Service EP Other Objects

HPSearch

Architecture Diagram

HPSearch Kernel

Request Handler

URIHandler DBHandler WSDLHandler WSProxyHandler

Files Sockets Topics DataBase Web Service

SOAP/HTTP WSProxy

Service Data buffers sent / received as Narada Events HPSearch Control Events using PUB/SUB on predefined topic Broker Network

WSProxy

Service

. . .

WSProxy

Service Community Grids Lab Presentation for CCGrid 2005 May 10, 2005

HPSearch Kernel

5

NaradaBrokering

 NaradaBrokering – Messaging infrastructure for collaboration, peer-to peer and Grid applications – Implements high-performance protocols (message transit time of 1 to 2 ms per hop) – Order-preserving messages , optimized message transport with QoS and security profiles for sent and received – Support for different underlying protocols TCP, UDP, Multicast, RTP – Discovery Service to locate nearest brokers such as May 10, 2005 Community Grids Lab Presentation for CCGrid 2005 6

HPSearch + NaradaBrokering

Managing Streams  HPSearch uses NaradaBrokering streams to route data – Each stream is represented by a topic name – Components subscribe / publish to specified topic   The WSProxy component automatically maps topics to Input / Output streams Each write (byte[] buffer) and byte[] read() call is mapped to a NaradaBrokering event Community Grids Lab Presentation for CCGrid 2005 May 10, 2005 7

System Goals

   Investigate the use of HPSearch as a management console to deploy system and application components using scripting Use HPSearch in scientific / grid applications which can use streaming data – Data filtering is essential in most cases and we do not want to transfer entire data set when only a small percentage of data would be used,  E.g. Choosing data satisfying certain input constraints.

– Data re-ordering might be required for formatting input data to match the requirements of executable,  E.g. Converting GML to remove XML elements Use publish-subscribe methodologies to connect components.

Community Grids Lab Presentation for CCGrid 2005 May 10, 2005 8

Example

RDAHMM    GPS time series have modes caused by unknown underlying physical processes RDAHMM allows us to identify these modes and time periods where physical processes dominated the sequence without any a-priori knowledge of these processes Help to determine – The actual physical causes – When is the system entering a new mode (perhaps a hint of some important seismic event) May 10, 2005 Community Grids Lab Presentation for CCGrid 2005 9

Example

Application Components     GPS database (Surface displacement time series collected by SCIGN, http://www.scign.org

) Data filter (filters data, removes unwanted components, reorders data as required by the actual RDAHMM executable) RDAHMM executable, performs the time series analysis. This data is transferred to the Graph plotting application.

Matlab based Graph plotting application Community Grids Lab Presentation for CCGrid 2005 May 10, 2005 10

GPS Data

Applications

Streaming Data Filtering Sensor Source HPSearch Engine

Handler Handler

Data Filter Filters the input data to get only the estimate and error values

Handler

Each Handler controls operation of 1 service RDAHMM Analyze the data HPSearch Matlab Plotting Script Output Graph May 10, 2005 Community Grids Lab Presentation for CCGrid 2005 11

Sample Output

Claremont, CA May 10, 2005 Community Grids Lab Presentation for CCGrid 2005 12

Related Work

 Scripting languages have been very popular and successful for Rapid Application Deployment (RAD) – Sash by IBM, allow RAD and handlers for various tasks such as reading from databases, LDAP registries, invoking Web Services and providing GUI – Jython and Matlab are popular in many scientific communities. E.g. GeoDISE – XCAT project at IU, Extreme Lab uses Jython for scripting to deploy distributed components – Karajan (Ant Like scripting) is used for deploying applications over grid – WSRF::Lite uses perl to implement Web Services Resource Framework and host grid services Community Grids Lab Presentation for CCGrid 2005 May 10, 2005 13

Future Work

     Investigate system and management scaling with increasing number of components.

Use of security for streams Negotiation of optimal (high performance) transport More handlers for different aspects of NaradaBrokering’s management (broker / topic discovery, security, replay etc…) Investigate interaction with WS Management and WS - Distributed Management based systems Community Grids Lab Presentation for CCGrid 2005 May 10, 2005 14

To conclude…

 Presented a scripting based architecture for management of data streams and distributed services  Shown how we can use publish subscribe methodologies to connect components that process data in a stream  Create and deploy quick data filtering applications Community Grids Lab Presentation for CCGrid 2005 May 10, 2005 15

More Information

 HPSearch Project Website http://www.hpsearch.org

 NaradaBrokering Project @ IU http://www.naradabrokering.org

 CGL Publications http://grids.ucs.indiana.edu/ptliupages/publications/ May 10, 2005 Community Grids Lab Presentation for CCGrid 2005 16

Questions / Comments

 Any Questions / Comments ?

May 10, 2005

THANKS

for attending the presentation Community Grids Lab Presentation for CCGrid 2005 17