Transcript Document

Manager’s Survival Guide to
OPeNDAP2
Dr. Matthew Howard
Dept. of Oceanography
Texas A&M University
14-January-2003
This is all about sharing data over the internet
Talk Outline
•
•
•
•
•
Introduction and Brief History
Terms and Definitions
What Resources Will You Need?
Practical Aspects
Available Resources
Introduction and Brief History
•
•
•
•
1992 The Vision
1994-2000 DODS
2000-2003 NVODS
OOS and OPeNDAP2
The Vision
•
•
•
•
1992* WWW and NCSA Mosaic
1993 Fall NASA/NOAA Workshop
Cornillon & Flierl CoIs + 40 others
Distributed Ocean Data System (DODS)
The system would allow one to access subsets of
remote data with their favorite software analysis
package across the network. DODS would subset
the data and convert it from the originator’s data
format to yours automatically.
• Designed by James Gallagher and Glen Flierl 1yr
* Same year consensus reached for GOOS
DODS
• 1994 NOAA/NASA 2 Programmers X 3yrs
• 1997-98 NOAA $1.5M - to continue core
system development.
• 1998-2000 NASA $2.7M to continue core
system development, & entrain users from
other disciplines. Included 13 partners
NASAs, NOAA, AGU, UCAR, NCAR and
6 Universities.
Meanwhile….
National Ocean Partnership Program (NOPP)
• 1997 ONR/NOPP Received Proposals for National
Virtual Ocean Data System (NVODS)
• One team proposed
1) Make 3 distributed data systems (MEL,
NOAAServer & DODS) interoperable
2) Create network-enabled clients for visualization
3) Facilitate data fusion
4) Create regional hubs of data and expertise
• None of the proposals were funded
Back to our story … btw You are here
NOPP 1999 RFP VODHub
• NOPP VODHub Awarded in 2000 $4.5M for 3yr
• “Development of an Integrated Regional,
National, and International Data System for
Oceanography”
• 23 1st tier partners - 27 letters of support
• Coordinate regional efforts (identify & entrain)
• Get DODS data sets online - Populate
• Continue development of core system
NOPP-2000 $4.5M/3yrs - Program Partners
Program Management and Support
Peter Cornillon University of Rhode Island PI
Richard Chinman UCAR Project Manager
Marga ret David son NOAA/CSC Workshop s
Regional Coordin ators
Mark Abbott Oregon State University
Anders Andren U. Wisconsin
Linda Mercer Dept. of Marine Resources State of Maine
Wo rth Nowlin Texas A&M
Data Provide rs
Brad Butman USGS Coastal and Marine P rograms
Don Collins Jet Propulsion L aboratory NASA
James Cummings Naval Research Laboratory Navy
Peter Hamilton SAIC
Herve Roquet Meteo-France
Neville Smith Bureau of Meteorology Research Centre
Ken Tenore University of Maryland
Ken Turgeon Minerals Management Service
System Developement
Glenn Flierl Massachusetts Institute of Technology
Dave Fulker UNIDATA UCAR User support
James Gallagher University of Rhode Island
Chris Lynnes NASA-Goddard DAAC
Ted Habermann NOAA-NGDC GIS
Steve Hanki n NOAA/PMEL WebCrawler &Web Interface
Richard Lawre nce ESRI, Inc GIS
Lola Olsen NASA-Goddard Directory/Search
Series of Meetings and VODHub > NVODS
• Annual Reports
•
First-year Report and Status as of February 2002
• Regional Meetings
•
Synthesis Report
•
Gulf Coast - 31 Oct to 2 Nov 2000
•
Southeast - 17 to 19 Nov 2000
•
Northeast - 8 to 10 Jan 2001
•
West Coast - 17 to 18 January 2001
• Technical Interchange Meeting
•
ESRI/GIS - 7 to 8 Nov 2000
• National Meeting
•
Washington, D.C. - 25 to 27 Apr 2001
• Second Executive Committee Meeting
•
Ocean.US, Arlington, Va. - 26 to 27 Feb 2002
• NVODS/DODS Technical Working Conference
•
Boulder, CO - 9 to 11 Jan. 2002
NVODS and OOS
• After 2001 National meeting VODHub is called
NVODS and program has broader scope (hooked
up with ISOOS).
• GOOS and NVODS and DODS converge
• OOS have emphasis on transporting data from
sensors to users in real and near-real time, and
delayed modes.
• And some choices must be made about how.
Motivation for Using OPeNDAP2
• In draft paper: A Data and Communications
Infrastructure for the U.S. Integrated Sustained
Ocean Observing System (Hankin et al 2002)
wrote:
• “The authors of this report recommend that
OPeNDAP2 be considered as the preferred
middleware solution to achieve the goals of the
DAC subsystem in a rapid and cost effective
manner”.
Terms and Definitions
•
•
•
•
•
DODS and DODS-DAP
NVODS and (OPeNDAP and OPeNDAP2)
DODS Servers and Clients
APIs and CGIs
Metadata
Context for Terms and Definitions
The Data Provider:
Installs free DODS-Server software on their web server
and places properly formatted data in a directory
accessible by the web server.
The Data User:
Obtains a DODS-enabled client program (eg. Matlab or
a web browser) and enters a URL for the requested
data set.
The requested data appears in user’s Matlab session or
browser window.
Terms and Definitions
• DODS - Distributed Ocean Data System
• “a data system intended to allow researchers transparent
access to oceanographic data - stored in any of several
different file formats across the Internet”. (DODS User
Guide, Tom Sgouros March 2000).
• DODS-DAP Data Access Protocol (The heart)
• “An extension to the web. Html provides a basic structure
to move text and pictures over the network. DODS-(DAP)
provides a basic structure to move data over the network.
Both use http as the underlying transport protocol. (P.
Cornillon- Presentation 2002).
• Aka middleware
• Intermediate interchange format
NVODS
• National Virtual Ocean Data System
• NVODS is the realization of the DODS vision.
• NVODS is DODS-DAP plus everything needed to
make it a useful and widely used system such as
catalogs, numerous high-quality desirable data
sets, and popular software clients for most
platforms.
OPeNDAP and OPeNDAP2
• One failure mode for DODS-DAP is the loss of
the lead programmer and programming team that
produced the DODS-DAP.
• Since DAP is useful to other disciplines one way
to protect it against loss is to give it away, ie open
it up to community-wide development.
• OPeNDAP (Open Source Project for a Network
Data Access Protocol) is public non-profit
corporation established to continue DAP
development. This allows other programmers to
become involved which can lead to more rapid
development and wider platform support.
• DODS-DAP will become OPeNDAP2
DODS Servers and Clients
• A DODS-Server is a software program that
reads, converts, subsets, and transmits data.
• Different servers for different data formats
• The DODS-server software resides on a
web server and executes on web server’s
host computer
• More than one server can be installed at a
time
DODS Servers and Clients - cont
• DODS-Clients are software packages used
to examine or manipulate data delivered by
DODS-Servers
• Any web browser can be used to examine
DODS data sets
• High-level commercial packages (eg
Matlab) have been made DODS-aware.
• Free programs exist too.
Terms and Definitions - cont
• CGI (common gateway interface) are software
programs that enhance the capabilities of a web
server.
-DODS-Servers are CGI programs
• API (Application Program Interface)
-Software tools (subroutine libraries) that facilitate
using one program with another.
-Software clients are made DODS-aware by
recompiling with DODS APIs.
Terms and Definitions - continued
• Metadata (Data about the data)
Syntatic metadata (byte stream is 100 real
numbers in 10 by 10 array)
Semantic metadata (data are temperatures in
degrees C, collected 10-May-2002 by Joe Blow,
on 100m grid with corners at X & Y).
Transmitted by DODS in separate structures
(useful to decide if you want the data or not)
Terms and Definitions Summary
•
•
•
•
DODS Vision in wide use = NVODS
DODS-DAP -> OPeNDAP2 (10X12X4)
DODS-Servers = CGI Web Server Programs.
DODS-Clients = Software Packages + DODS
APIs.
• Syntactic metadata (necessary to use data at all)
• Semantic metadata (necessary for interpretation)
What Resources Will You Need?
•
•
•
•
DODS-Server software
Web Server
Data in a servable format
Human resources
Supported Hardware Platforms for
DODS Data Servers
free
Sun Sparc Solaris 2.6
Intel PC Linux RedHat 6.2
Intel PC Linux RedHat 7.0
Intel PC Linux RedHat 7.1
Dec Alpha OSF 4.0f
SGI Irix 6.5 (n32/mip3)
Free
$1k $15K
Web Servers
• Most sites run free Apache Web Server
• Netscape’s FastTrack Web Server works
fine too.
• Limitation is DODS-Server programs.
• OPeNDAP should yield Windows based
server solutions.
• At present UNIX is it.
Data in Servable Format
• Most gridded self-describing formats are
supported such as netCDF, HDF, and maybe
GRIB.
• Matlab binary on some platforms
• DSP on some platforms
• ASCII can be handled by DODS-FreeForm server
and the DODS-JGOFS server or converted to
netCDF using several methods.
• No support for ESRI GIS formats yet.
Human Resources
• Web Server Administrator
download DODS-Servers and copy to
CGI area of web server
• Data formatter/re-formatter
– Programming or scripting ability
• C++ or Java programmer for building DODS-enabled clients
Practical Aspects
• Existing Data Format:
– If netCDF or HDF you are in great shape
– If not you’ll probably need to do some work
• Duplication of effort and files
–
–
–
–
Separate data streams
Files in two formats (for legacy and for DODS)
Double storage and management issues
Ideally serve your files as they are.
• Metadata
– Is your metadata complete? In separate files?
– FGDC requirements
Convert Existing Files to netCDF
cnv
Script
cdl
ncgen
netCDF
Create netCDF files as you go.
Your Program +
netCDF API
netCDF
Practical Aspects
• Problems area
– Security is basically web site security
– Data discovery (advertise, register, web
crawler)
– Data Selection (GUI assisted 4D selection)
– Hardware platform client-server matrix
– GIS/SIS
Outside Resources
•
•
•
•
•
DODS website
UCAR contract help
Regional Centers
URI
Catalog websites