Transcript SERVOGrid Basics • SERVOGrid is our project to build a distributed computing infrastructure to support earthquake simulation codes. – We use Web Services for.

SERVOGrid Basics
•
SERVOGrid is our project to build a distributed computing infrastructure to
support earthquake simulation codes.
– We use Web Services for this.
– We build portal interfaces to these web services out of portlets.
• Portlets for Java portals are standardized (JSR 168)
• All major commercial vendors, many open source projects support.
• GEON, NEESGrid, LEAD portal use also.
•
Distributed Computing Infrastructure
– Services for remotely executing applications, managing distributed files in logical
units, monitoring applications, archiving user projects, and so on.
– Variously called “Grids” (DOE), “Cyberinfrastructure” (NSF), “e-Science” (UK eScience program), ….
•
The above is sometimes referred to as an “Execution Grid”.
– We designed this around a web service architecture.
– See links at the end for more information.
•
But what about the data?
– Need ways to access remote data sources through programming interfaces.
– Practice often is to download (via FTP) entire catalogs and hand edit to get data
you want in the format you want.
– We need to automate this.
1
Some SERVO Apps and Their Data
• Disloc: handles multiple arbitrarily dipping dislocations (faults) in an
elastic half-space.
– Relies upon geometric fault models.
• GeoFEST: Three-dimensional viscoelastic finite element model for
calculating nodal displacements and tractions. Allows for realistic
fault geometry and characteristics, material properties, and body
forces.
– Relies upon fault models with geometric and material properties.
• Virtual California: Program to simulate interactions between vertical
strike-slip faults using an elastic layer over a viscoelastic half-space.
– Relies upon fault and fault friction models.
• RDAHMM: Time series analysis program based on Hidden Markov
Modeling. Produces feature vectors and probabilities for
transitioning from one class to another.
– Used to analyze GPS and seismic catalogs.
2
Where Is the Data?
• QuakeTables Fault Database
– SERVO’s fault repository for California.
– Compatible with GeoFEST, Disloc, and VirtualCalifornia
– http://infogroup.usc.edu:8080/public.html
• GPS Data sources and formats (RDAHMM and others).
– JPL: ftp://sideshow.jpl.nasa.gov/pub/mbh
– SOPAC: ftp://garner.ucsd.edu/pub/timeseries
– USGS: http://pasadena.wr.usgs.gov/scign/Analysis/plotdata/
• Seismic Event Data (RDAHMM and others)
– SCSN: http://www.scec.org/ftp/catalogs/SCSN
– SCEDC: http://www.scecd.scec.org/ftp/catalogs/SCEC_DC
– Dinger-Shearer: http://www.scecdc.org/ftp/catalogs/dingershearer/dinger-shearer.catalog
– Haukkson: http://www.scecdc.scec.org/ftp/catalogs/hauksson/Socal
3
What Do We Want to Do?
•
•
•
Web Services provide a programming interfaces to data so that you can
build remote clients.
SERVO portal work houses these Web Service clients.
Portal also houses the clients for the remote simulation services (RDAHMH,
GeoFEST, Potts model codes, etc).
– As we have seen, these codes work directly with data catalogs
•
Typically the codes must process and reformat the data catalog entries in
code-specific ways.
– Done by hand by most developers, which is fine for development.
– But not fine for mature codes or automated applications.
•
Currently, our solution is to harvest data from online catalogs and transform
into GIS compatible formats in DBs.
– Discuss a couple of examples
•
But we would like to experiment with building services that are closer to the
source.
– Finer resolution data products
– Academic exercise in implementing reliable sensor network services and
streaming applications.
•
SensorML family from Open Geospatial Consortium provides the standards.
4
Open GIS Services
• GML abstract data models can encode data but you need services
to interact with the remote data.
• Some example OGC services include
– Web Feature Service: for retrieving GML encode features, like faults,
roads, county boundaries, GPS station locations,….
– Web Map Service: for creating maps out of Web Features
– Sensor Grid Services: for working with streaming, time-stamped data.
• Problems with OGC services
– Not (yet) Web Service compliant
• “Pre” web service, no SOAP or WSDL
• Use instead HTTP GET/POST conventions.
– Often define general Web Service services as specialized standards
• Information services
• Notification services in sensor grids
5
Using GML to Encode GPS
• Collected GPS data for each
site is made available through
online catalogs.
– Using various text formats.
• We have developed GML
descriptions of GPS using
Feature.xsd schema.
• Full GML GPS schema:
http://www.crisisgrid.org/files/g
ps.xsd.
• Supports JPL, SOPAC, and
USGS formats.
• Other efforts: Feature
encodings of Seismic and
Fault catalogs.
– http://www.crisisgrid.org/html/s
ervo_-_gps.html
6
Anatomy of Web Feature Service
• GML provides the data model, but we must still provide a remote
programming interface to the data.
• Web Feature Service provides three major services as described in
OGC specification:
• GetCapabilities: The clients (WMS servers or users) starts with
requesting a document from WFS which describes it’s abilities. When a
getCapabilities request arrives, the server dynamically creates a
capabilities document and returns this.
• This is OGC’s formalization of metadata, so important to GEON, ESG, etc.
• DescribeFeatureType: After the client receives the capabilities
document he/she can request a more detailed description for any of the
features listed in the WFS capabilities document.
• The WFS returns an XML schema that describes the requested feature.
• Metadata about a specific entry.
• GetFeature: The client can ask the WFS to return a particular portion of
any feature data.
• GetFeature requests contain some property names of the feature and a
Filter element to describe the query.
• The WFS extracts the query and bounding box from the filter and queries
the feature databases.
• The results obtained from the DB query are converted the feature’s GML
format and returned to the client as a FeatureCollection.
7
Web Map Services
• OGC Web Map Servers create digital maps from
abstract data sets retrieved from Web Feature Services.
– Layer features are overlayed.
• WMS servers can be daisy chained.
• Our research efforts here are to build Web Service
version of the WMS as well as portlet clients.
• We are developing backward compatibility with existing
WMS.
– Purdue CAAGIS for Indiana maps.
– NASA JPL OnEarth WMS.
• Future plans are to replace current GMT portal tools with
WMS, combine maps with GeoFEST and Virtual
California application output.
8
Next Steps and Questions (Jan)
• We are very interested in
starting a new Sensor
Grid project.
•
– Implement Open
Geospatial Consortium
standards for Sensor Grids
(see supplemental slides).
– We are outlining a more
ambitious integration of
Robert Granat’s RDAHMM
with GPS data.
•
•
• I will be visiting JPL in
early February.
• Some questions we have
are on the right.
•
•
•
•
•
•
•
•
•
Direct (or at least closer) access to GPS
stations?
Any documentation on the filtering process?
What kind of filters are applied on the raw
data? Can it be automated?
How is data stored, in a DB or just in files?
How do the stations communicate with the
servers?
What protocols are being used for transport,
etc?
Are there any web-based interfaces to
stations or to the data stores?
Are the GPS stations configurable?
How often do they send out data, can you
change the interval?
What are the most important data formats
for you? (JPL,SOPAC,USGS)
Are there any other formats we need to
support?
What (other) kinds of applications are there
to use the GPS data?
9
HPSearch Overview
• HPSearch is a system for scripting and
controlling web services and controlling data
streams.
– Uses JavaScript (Rhino) for scripting
– Uses NaradaBrokering for controlling communications
between services.
– Name is historical
• Basic premise is that it can be used to hook
together streams of data to successive filters
(services) that are controlled through a client
environment.
10
Basic HPSearch Parts
• Kernel Components
– Shell: processes the extended JavaScript
– Service Manager: this is a scheduler for load balancing between
several services.
– RequestHandler: monitors individual remote resources (may
include web services but also DBs or plain old URLs).
• WSProxy facilitates wrapping legacy applications as
streaming web services
– Manages both the control invocations from the shell and the NBbased streaming data connections
– Comes with two interfaces
• Runnable: you extend this Java interface for new services.
• Wrapped: this is a legacy/non-WS service such as a perl or matlab
script.
11
HPSearch Architecture
HPSearch Kernel
HPSearch Kernel
Data is read / written from /
to various data resources
Files,
Sockets,
Topics
Request Handler
Request Handler
Javascript Shell
URIHandler
Service Manager
DataBase
DBHandler
Web Service EP
WSDLHandler
Other Objects
WSProxyHandler
Web
Service
WSProxy
Servic
e
Broker Network
WSProxy
Service
Data Flows as a stream
of messages between
WSProxies through the
brokering network
Stream Proxy
Kernel connects to nearest
broker and communicates
with other kernels with
messages
HPSearch
Kernel
Handlers steer web
services via simple
SOAP requests
12
HPSearch for RDAHMM + GPS
Sensor Network / GPS Database
Start Activity,
Web Service
Virtual Data Flow
between various
components
Narada Broker
Data Filter
Links between
Brokers
WSProxy
wrapped service
HPSearch
Kernel
Actual Data Flow between
components takes place
through the brokering
network
Kernel manages
(steers) the
various services
RDAHMM
Matlab Script
Service
distribution
13
More Information
•
My email:
– [email protected]
•
Overview (submitted to ACES special issue of PAGEOPH).
– http://www.servogrid.org/slide/GEM/SERVO/ISERVO_ACES_PAGEOPH.doc
•
QuakeSim Project Page:
– http://quakesim.jpl.nasa.gov/
•
QuakeSim Portal:
– http://www.complexity.ucs.indiana.edu:8282.
•
QuakeTables Fault Database:
– http://infogroup.usc.edu:8080/public.html
•
Community Grids Lab GIS Grid Development:
–
–
–
–
http://www.crisisgrid.org/.
See particularly http://www.crisisgrid.org/html/servo.html for GML information.
See http://www.crisisgrid.org/html/wfs.html for more on Web Feature Service.
WMS and Information Services information coming soon.
14
Supplemental Slides: Sensor
Web Work
Slides Prepared by Galip Aydin
15
Sensor Web
• Goal: To create Web-based sensor networks by
exploiting web-connected sensors (flood
gauges, air pollution monitors etc.).
• This would make all sensors and repositories of
sensor data discoverable, accessible and where
applicable controllable via the WWW.
• OGC describes 5 important encoding and
service standards for a Sensor Web
Implementation (see next slide).
16
OGC-Sensor Web Enablement
1.
2.
3.
4.
5.
SensorML: XML encoding language for sensors. Used to discover, query
and control Web-resident sensors.
Observations & Measurements: The general models and an XML
encoding for what a sensor observes or measures (The value returned by
or derived from a sensor observation -e.g. quantity, count, boolean,
category, ordered category, position-).
Sensor Collection Service: A service to fetch observations from a
sensor or constellation of sensors. Provides real time or archived
observed values. Clients can also obtain information that describes the
associated sensors and platforms.
Sensor Planning Service: A service by which a client can determine
collection feasibility for a desired set of collection requests for one or
more mobile sensors/platforms, or the client may submit collection
requests directly to these sensors/platforms. SPS enables sensor
tasking, acquisition requests, processing and simulation requests, and
registration for alert notification.
Web Notification Service: A service by which a client may conduct a
dialog with one or more other services. Provides a means for Sensor
Planning Services to alert people, software, or other sensor systems of
SPS results or alerts regarding phenomena of interest.
17
OGC-SWE Pieces
User Environment
In Situ Collection Environment
Discover SCS
For Water Quality
User
Need for
Water Quality Data
SWE Client
SCS, WFS,WCS,CPS,WMS
Available SCS
For Water Quality
Collection Feasibility Request or
Collection Request
Confirmation of
Valid Collection
Request or exceptions
+ Session Handle +
Collection Feasibility
Plan
Sensor
Planning
Service
Observation
Request
Observation
Response
Service
Registries
Sensor
Collection
Service
Sensor/Platform
Sensor/Platform
& Observable
& Observable
Registries
Registries
Base
Stations
&
Sensors
Get Water Quality
Observations & Metadata
Local
Archive
&
Catalog
Water Quality
Observations &
Metadata
Observation
Archive
&
Catalog
18
Sensor Web Enhancements 1
• We think that with some changes and additions a
minimal and high performance Sensor Web with live
sensor data capability can be realized.
• We realize that some of these service specifications are
trying to solve problems common to the Web Services
community such as notification.
• There are several Notification specifications for Web
Services such as WS-Notification and WS-Eventing.
Considering the industrial backing and huge WS
community support we can expect these specifications to
evolve into standards.
• Community Grids lab has considerable experience on
Notification issues and currently a WS-Notification
enabled version of NaradaBrokering is being developed.
19
Sensor Web Enhancements 2
• Sensor Planning service is designed as an asset
management service. For our purposes existence of
such a service is not crucial instead an Information
Service with search and discovery capabilities is
preferable.
• We are developing a High Performance Fault Tolerant
Information Service for GIS applications and with little
modifications this service can handle the functionality
expected from Sensor Planning Service.
• The Information Service will provide search and
discovery interfaces and WSDL addresses for Sensor
Collection Services.
20
Sensor Web Enhancements 3
• In some cases immediate access to the sensor data is
crucial. In such cases we expect to have some sort of a
streaming data channel for live sensor data available to
the clients.
• We plan to use Narada Brokering to provide us such a
capability.
• Narada Brokering allows message producers to publish
messages to topics and subscribers of these topics
immediately receives the published messages.
• We can create topics according to the types of the
sensors and subscribe clients to these topics according
to their preferences. Thus Narada Brokering can provide
a streaming channel between sensors and the clients.
21
A sample Sensor Web that utilizes Information Service and Narada Brokering.
22
NaradaBrokering Overview
• NB is software for routing messages that has been
developed by Shrideep Pallickara at CGL for several
years.
– Topic-based publish/subscribe system
• Implements JMS, JXTA, and its own APIs.
• Allows you to construct hierarchical broker networks for
efficient routing.
• Can route “anything”
–
–
–
–
–
HTTP, TCP/IP, UDP
Binary data for collaborative shared display (www.anabas.com)
GridFTP via proxy
SOAP messages
WS-Eventing and WS-ReliableMessaging
• Moving toward supporting SOAP 1.2-style messaging.
23