Transcript Project Mirrion

Simple NOAA Archive Access Portal
SNAAP
Eric Kihn and Rob Prentice
NOAA/NGDC
ESIP Meeting
January 7th, 2009
CLASS –Simple NOAA Archive Access Portal
1
SNAAP Goals

Integrate diverse data archive systems via a standards based,
user focused, web services interface

Create a set of tools that shows the benefit of such an API for
users

Discover new technology and evaluate cutting edge tools for
handling integration and presentation

Document integration of multiple data types and sources
through an single API (time-series, grid, swath, etc..)

Keep it simple. Anything tough is a client problem
CLASS –Simple NOAA Archive Access Portal
2
The Data Environment Now
Web Interface
NMMR/Metadata
Human
Machine
Web Interface
Data Access
SNAAP
Metadata
Metadata
Web Interface Subscriptions
Visualization
Providers
Data
Web Interface
SPIDR WS
Inventory
CLASS ADS
(e.g. AVHRR)
NGDC/SPIDR
(e.g. DMSP, Space Physics)
Order
CLASS –Simple NOAA Archive Access Portal
3
What is the origin of SNAAP?

The project evolved from the goal to put an API on CLASS. It
was quickly realized that the boundary is then arbitrary and it
makes sense to extend the API to other data systems

CLASS (and other systems) already have a functional API for
intra-system communication but the duplicity of systems
causes real headaches

The system is currently limited to archive (non-real time) data
systems following its archive access origins

SNAAP is part of the GEO/GEOSS effort. Its architecture and
capabilities do for NOAA what is being done in GEOSS
(archive)
CLASS –Simple NOAA Archive Access Portal
4
SNAAP Functionality

Search for appropriate data sets
 Discover data by time, type, location, abstract, etc.
 Support for multiple metadata schema (FGDC, IEEE, SPASE)
 Standards based (xQuery and REST)
 This is in already in CLASS (4.3.3) – awaiting policy

Order Data
 Needs a standard way to describe ordering options
 This is done via Ordering Extensions (OE) which are linked to and extend
standard metadata
 Needs some framework for service integration (i.e. transforms) this is
provided via (Data Access and Integration) DAI Actions

Provider Integration Layer
 We have chosen OGSA-DAI for the demo (This could change in production)
 We offer both a SOAP and REST interface (slightly different functionality)
 Its important that the backend retrieval not effect the consumer.
CLASS –Simple NOAA Archive Access Portal
5
Advantages

This approach mirrors that of the highly successful OGC
specification for map services (what, make, retrieve)

It minimizes entry burden for new providers

Its simplicity makes it scalable

It doesn’t preclude or interfere with on-going activity, just
opens up new integration possibilities.
CLASS –Simple NOAA Archive Access Portal
6
Metadata: A Complete Data
Environment is More than Just the
Bits
SNAAP metadata
Virtual Observatory
SNAAP Web
Application
SNAAP API Data
Sources
FGDC records
Ordering
Extentions
Data Request
FGDC
Metadata
OE (1…*)
OGSA-DAI client toolkit
SEARCH in
CLASS
metadata
Wiki Documents
REST API for CLASS
OGSA-DAI for CLASS (Globus
Toolkit 4)
Plugin
CLASS ResourceID
User
Guide
CLASS
Inventory
service
Slideshow
Presentations
Search result:
ResourceID_1
ResourceID_2
ResourceID_3
CLASS –Simple NOAA Archive Access Portal
Visualization
service
Order
service
CLASS products
7
Snapshot of the
SNAAP API architecture
CLASS web
interface
CLASS
metadata
REST API for CLASS
THREDDS
(OPeNDAP)
WCS
(OGC)
GRID client toolkit
OGSA-DAI client toolkit
OGSA-DAI for CLASS (Globus Toolkit 4)
Plugin
OPeNDAP
Plugin
CLASS
Inventory
service
NOMADS
NetCDF
NetCDF
NetCDF
NARR
Visualization
service
Order
service
CLASS products
Plugin
Plugin
Plugin
SABR
ESG - INE - IDEAS – ESSE
SPIDR
Web
Services
Inventory
Preview
image
OLS DMPS
DB cluster
DB cluster
NCEP , ERA-40 , ...
Granule
Granule
Satellite
Granule
CLASS –Simple NOAA Archive Access Portal
8
Why OGSA-DAI?

Standard tool in the Grid community

Includes a “process” for extraction

Built in support for asynchronous transactions

Includes support for domain login

Looked at alternatives like OpenDap, WCS, Custom
applications –documentation of our analysis is available

Problem: Its very complex and not well accepted in NOAA
CLASS –Simple NOAA Archive Access Portal
9
Data Order Flow
2
1
OE
Web Form
XSLT
Servlet
get
Process Document via
SOAP
Client
3
Server
Get Data
Process
Mine
OGSADAI
XML Result
Error
Message
Data
Adapter
SQL
XML
Granule
Time
Series
Storage
CLASS –Simple NOAA Archive Access Portal
10
CLASS Ordering Extensions (OE)
schema upper level
CLASS –Simple NOAA Archive Access Portal
11
CLASS OE schema:
parameters and time
CLASS –Simple NOAA Archive Access Portal
12
CLASS OE schema:
map and ROI
CLASS –Simple NOAA Archive Access Portal
13
CLASS OE schema:
station map
CLASS –Simple NOAA Archive Access Portal
14
Key Goal:
Minimize the Burden on Providers
To hook to SNAAP a provider would :

Create a standard metadata record for the data set. (They
should be doing this anyway).

Create an Ordering extension document. If they are offering
web distribution this is trivial.

Write an adapter to the middle layer, unless they have a
supported service (like JDBC, FTP, etc..) in which case adapt
an existing one.
The provider then:

Continues to serve data and metadata as before plus
SNAAP access.
CLASS –Simple NOAA Archive Access Portal
15
Data Sets and Types

Data Types







Time-series – Sunspot number
Grids – NCEP Reanalysis
Stations – Ionospheric Soundings
Swath - AVHRR
Profiles – Ocean Profile
Maps – Nighttime lights
More?
CLASS –Simple NOAA Archive Access Portal
16
Target Audience

There is a clear understanding that many communities
are already well served by existing data systems

The SNAAP API targets developers who want to add
new functionality to integrated NOAA archives

When operational it will provide fully redundant support
for data discovery and retrieval

It is hoped that by having the data accessible in a
standard open way more innovative client tools will
become available.
CLASS –Simple NOAA Archive Access Portal
17
Visualization screens – 1
time series and map server
CLASS –Simple NOAA Archive Access Portal
18
Visualization screens – 2
NASA World Wind plugins
CLASS –Simple NOAA Archive Access Portal
19
MS Excel and NASA World Wind
interoperability for trajectory data
Excel with track points
NASA World Wind
Drag & Drop controls
Excel with data along tracks
CLASS –Simple NOAA Archive Access Portal
20
Future Plans
Plan transition to operations (CLASS Team)
 This will identify a CLASS build and date for SNAAP inclusion
 This would require some review process (including outside)
and a versioning of OE
 Seek outside review of the OE documentation.

Plan to include non-granule based asynchronus (i.e. models,
transforms) data extractions

Enhance the connection with GEO-IDE

Upgrade to DAI 3.0 - DONE

Expand proto-type to cover more NOAA and non-NOAA data types
and services and document lessons learned

Release the work to the public: Metadata tool is available as
VxOware http://sourceforge.net/projects/vxoware/

Work on single user sign-on.
CLASS –Simple NOAA Archive Access Portal
21
BACK UP SLIDES
CLASS –Simple NOAA Archive Access Portal
22
Asynchronous Ordering
CLASS API Demonstration
CLASS –Simple NOAA Archive Access Portal
23
Orbit Inventory List
CLASS –Simple NOAA Archive Access Portal
24
User Basket (Order & CheckStatus)
CLASS –Simple NOAA Archive Access Portal
25
Order System Entities
•Granule List now using
ECHO XML in the return
•New entity “Data Basket”
– handle to collection of
order
•Demo currently orders all
data centrally will need to
be adapted
CLASS –Simple NOAA Archive Access Portal
26
Order Sequence Diagram
•Data basket ID is
the API handle
•OGSA-DAI
activities handle the
extraction and
packaging
•Status check is
currently by basket
only (XML return)
•Finished granules
return a URL handle
to the entity
CLASS –Simple NOAA Archive Access Portal
27