Transcript QuakeSim Work: Web Services, Portlets, Real Time Data Services Marlon Pierce ([email protected]) Contributions: Ahmet Sayar, Galip Aydin, Mehmet Aktas, Harshawardhan Gadgil, Zhigang Qi, Zao Liu Community Grids.

QuakeSim Work: Web Services,
Portlets, Real Time Data
Services
Marlon Pierce ([email protected])
Contributions: Ahmet Sayar, Galip Aydin,
Mehmet Aktas, Harshawardhan Gadgil,
Zhigang Qi, Zao Liu
Community Grids Lab
Indiana University
Project Funding: NASA AIST, ACCESS
QuakeSim Project Overview
• QuakeSim has been funded by CT, AIST, and
ACCESS
• Collaboration of geophysicists and computer
scientists to build cyber-infrastructure for
geophysical research.
• CI research and development includes
– Portlet-based portals
• AJAX enabled
– Geographical Information System services
– Application services to run codes.
QuakeSim Project
Development Overview
•
•
•
Portlet-based portal components allow different portlets to be
exchanged between projects.
– Form-based portlets --> Interactive Maps
– These are clients to Web services
– Share with collaborators of REASoN portal.
Sensor Grid: Topic based publish-subscribe systems support
operations on streaming data.
Web services allow request/response style access to data and codes.
– GIS services (WMS, WFS)
– “Execution grid” services for running codes: RDAHMM, ST_Filter
• Use application specific WSDL on top of generic code management
services.
– GPS daily archive Web Services provided by Scripps.
Evolving Project Philosophy
• Methodology for interoperability between portal
projects exists today.
– Portlets, web services
– Scripps and IU are proving this.
• Must continue to move away from the portal
priesthood and towards a “scientific mashup” model.
– We still need to develop interesting services and client
libraries.
– But we should make it easy for the application scientists to
• Add their own applications.
• Add their own data.
• Make their own web applications (ie mashups)
IU Portlet Development
We use JSR 168 portlets to build
sharable portal plugins.
Portlet Summary
RDAHMM
Set up and run RDAHMM, query Scripps
GRWS GPS Service, maintain persistent
user sessions.
ST_Filter
Similar to RDAHMM portlet; ST_Filter has
much more input.
Station Monitor
Shows GPS stations on a Google Map,
displays last 10 minutes of data.
Real Time RDAHMM
Displays RDAHMM results of last 10
minutes of GPS data in a Google map.
Seismic Archive Query
Portlet
Google Map portlet that shows seismic
events based on your query. *
Fault Query Portlet
Allows you to query the QuakeTables fault
data base for information on faults.*
* Developed, needs to be converted into a portlet
RDAHMM Portlet: Main
Navigation
RDAHMM Project Set Up
RDAHMM GRWS Query
Interface
RDAHMM Results Page
Real Time RDAHMM Portlet
Station Monitor Portlet
ST_Filter Portlets
IU Web Service Work
Web Services
Ant Execution Service
General purpose service for running applications.
Wraps Apache Ant.
RDAHMM Service
Based on Ant service, but provides a RDAHMMspecific API.
ST_Filter Service
Runs ST_Filter, based on Ant service. Will have an
ST_Filter-based WSDL.
Web Feature Service
General purpose, OGC-compliant data service.
Fault and seismic record data.
Web Map Service
Map-generating service. We have built both a OGC
WMS and a Google tiling-style map server
(currently Indiana specific but can be generalized).
Context Service
This is a lightweight metadata management service
that can be used to store user session data. Project
provenance: how did I make this output file?
Sensor Grid Overview
Publish/subscribe infrastructure
for handling real time data.
Real-Time Services for GPS
Observations
• Real-time data processing is supported
by employing filters around
publish/subscribe messaging system.
• The filters are small applications
extended from a generic Filter class to
inherit publish and subscribe
capabilities.
Input Signal
Filter
Output Signal
Filter Chains
NaradaBrokering Topics
Real-Time positions on Google
maps
Real-Time Station Position
Changes
RDAHMM + Real-Time GPS
Integration
SensorGrid Tests
Galip Aydin – Zhigang Qi
11/16/2006
SensorGrid Tests
• Two Major Goals: System Stability and
Scalability
– Ensuring stability of the Filter Services for
continuous operation.
– Finding the maximum number of publishers
(sensors) and clients that can be supported
with a single broker.
– Investigate if system scales for large
number of sensors and clients.
Test Methodology
Ttransfer = (T2 – T1) + (T4 – T3)
• The test system consists of a NaradaBrokering server and a
three-filter chain for publishing, converting and receiving RYO
messages.
• We take 4 timings for determining mean end-to-end delivery
times of GPS measurements.
• The tests were run at least for 24 hours.
1- System Stability Test
System Stability Test
6
5
3
2
1
Tim e of the Day
Transfer Time
Standard Deviation
22:30
21:00
19:30
18:00
16:30
15:00
13:30
12:00
10:30
9:00
7:30
6:00
4:30
3:00
1:30
0
0:00
Time (ms)
4
• The basic system with
three filters and one
broker.
• The average transfer
time shows the
continuous operation
does not degrade the
system performance.
2 – Multiple Publishers Test
Multiple Publishers Test
6
5
Time (ms)
4
3
2
1
Tim e of the Day
Transfer Time
Standard Deviation
• We add more GPS networks by running more
publishers.
• The results show that 1000 publishers can be
supported with no performance loss. This is an
operating system limit.
22:30
21:00
19:30
18:00
16:30
15:00
13:30
12:00
10:30
9:00
7:30
6:00
4:30
3:00
1:30
0:00
0
3 – Multiple Clients Test
1000 Clients
Multiple Subscribers Test
40
35
Time (ms)
30
25
20
15
Adding clients
10
5
0
00 :30
0:
1
00 :30 :00 :30
3:
4
6
7
00
30
00
30
00
30
00
30
00
30
9: 10 : 12 : 13 : 15 : 16 : 18 : 19 : 21 : 22 :
Time Of the Day
Transfer Time
Standard Deviation
• We add more clients by running multiple Simple Filters.
• The system can support as many as 1000 clients with
very low performance decrease.
Extending Scalability
• The limit of the basic system appears to be 1000
clients or publishers.
• This is due to an Operating System restriction of
open file descriptors.
• To overcome this limit we create NaradaBrokering
networks with linking multiple brokers.
• We run 2 brokers to support 1500 clients.
– Number of brokers can be increased indefinitely, so we can
potentially support any number of publishers and
subscribers.
– Still have to test, of course.
4 – Multiple Brokers Test
• Messages published
to first broker can be
received from the
second broker.
• We take timings on
each broker.
• The results show that
the performance is
very good and similar
to single broker test.
4 – Multiple Brokers Test
Multiple Broker Test
Broker 2
Multiple Broker Test
Broker 1
40.00
35.00
35.00
30.00
30.00
Tim e Of The Day
Transit Time
Standard Deviation
Tim e Of The Day
Transfer Time
Standard Deviation
22:30
21:00
19:30
18:00
16:30
15:00
13:30
12:00
10:30
9:00
7:30
6:00
22:30
21:00
19:30
18:00
16:30
15:00
13:30
12:00
10:30
9:00
0.00
7:30
0.00
6:00
5.00
4:30
5.00
3:00
10.00
1:30
10.00
4:30
15.00
3:00
15.00
750 Clients
20.00
1:30
750 Clients
25.00
0:00
Time (ms)
20.00
0:00
Tim (ms)
25.00
Test Results
• The RYO Publisher filter publishes 24-hour
archive of the CRTN_01 GPS network which
contains 9 GPS stations.
• The single broker configuration can support
1000 clients or networks (9000 stations)
• The system can be scaled up by creating
NaradaBrokering broker networks.
• Message order was preserved in all tests.
Federating Map Servers
Zao Liu, Marlon Pierce, Geoffrey
Fox
Community Grids Laboratory
Indiana University
Integrating Map Servers
•
•
•
Geographical Information Systems combine online dynamic
maps and databases.
Many GIS software packages exist
GIS servers around state of Indiana
–
ESRI ArcIMS and ArcMap Server (Marion, Vanderburgh,
Hancock, Kosciusco, Huntington, Tippecanoe)
–
Autodesk MapGuide (Hamilton, Hendricks, Monroe,
Wayne)
–
WTH Mapserver™ Web Mapping Application (Fulton,
Cass, Daviess, City of Huntingburg) based on several
Open Source projects (Minnesota Map Server)
•
Challenge: make 17 different county map servers from different
companies work together.
–
92 counties in Indiana, so potentially 92 different map
servers.
Considerations
• We assume heterogeneity in GIS map and feature servers.
– GIS services are organized bottom-up rather than top-down.
– Local city governments, 92 different county governments, multiple
Indiana state agencies, inter-state (Ohio, Kentucky) consideration,
federal government data providers (Hazus).
– Must find a way to federate existing services.
• We must reconcile ESRI, Autodesk, OGC, Google Map, and
other technical approaches.
– Must try to take advantage of Google, ESRI, etc rather than compete.
• We must have good performance and interactivity.
– Servers must respond quickly--launching queries to 20 different map
servers is very inefficient.
– Clients should have simplicity and interactivity of Google Maps and
similar AJAX style applications.
Caching and Tiling Maps
•
Federation through caching:
– WMS and WFS resources are queried and results are stored on the cache
servers.
– WMS images are stored as tiles.
• These can be assembled into new images on demand (c. f. Google
Maps).
• Projections and styling can be reconciled.
• We can store multiple layers this way.
– We build adapters that can work with ESRI and OGC products; tailor to specific
counties.
•
Serving images as tiles
– Client programs obtain images directly from our tile server.
• That is, don’t go back to the original WMS for every request.
– Similar approaches can be used to mediate WFS requests.
– This works with Google Map-based clients.
– The tile server can re-cache and tile on demand if tile sections are missing.
Map Server Example
Marion and Hancock
county parcel plots
and IDs are overlaid
on IU aerial
photographic images
that are accessed by
this mashup using
Google Map APIs.
We cache and tile all
the images from
several different map
servers. (Marion and
Hancock actually use
different commercial
software.)