Transcript Selling a Product or Service - HDF-EOS

Access HDF5 Datasets via
OPeNDAP’s Data Access
Protocol (DAP)
MuQun Yang
The HDF Group
7/18/2015
HDF and HDF-EOS Workshop X, Landover, MD
1
OPeNDAP
•
•
•
•
•
•
A software framework that allows simple access to remote data
Client-server model
Servers make the local data accessible remotely
Clients make applications easily access to remote served data
Widely used by Earth Science Community
URL: http://www.opendap.org/
7/18/2015
HDF and HDF-EOS Workshop X, Landover, MD
2
The Data Request
 Data are requested with a URL.
http://www.cdc.noaa.gov/cgi-bin/nph-nc/datasets/Reynolds_sst?sst[10:10][0:90][0:180]
Protocol Machine name
OPeNDAP server Directory File name
Constraint
 The user can impose a constraint on the data that s/he
would like to acquire from a data set by appending a
constraint expression to the end of the URL.
7/18/2015
HDF and HDF-EOS Workshop X, Landover, MD
3
OPeNDAP Data Types
OPeNDAP provides three basic data objects:
 data descriptor structure (.dds) - syntactic
metadata - rigid
 data attribute structure (.das) - semantic
metadata - flexible
 data (.dods) - the actual data in a binary
structure
7/18/2015
HDF and HDF-EOS Workshop X, Landover, MD
4
OPeNDAP Servers
CODAR
netCDF HDF4
Data
Data
CODAR
netCDF
Data
Matlab
DSP
Tables
SQL
FITS
CDF
Flat
Binary
Data
Data
Data
Data
Data
Data
Data
Matlab
HDF4
JGOFS
DSP
FITS
JDBC
CEDAR
General
Data
Data
ESML
FreeFrom
CDF
CEDAR
No fully-supported HDF5 server
7/18/2015
HDF and HDF-EOS Workshop X, Landover, MD
5
OPeNDAP Clients
netCDF C
Ferret
GrADS
netCDF Java
IDV
VisAD
ncBrowse
Matlab
Client
IDL
Client
Matlab
IDL
Access
No HDF5 client
7/18/2015
HDF and HDF-EOS Workshop X, Landover, MD
6
Excel
Why HDF5-OPeNDAP server and
client?
Users!
Users!!
Users!!!
7/18/2015
HDF and HDF-EOS Workshop X, Landover, MD
7
A User Case
Task:
•10 elements of 17 MB HDF5 data
•200 elements of 10 MB NetCDF data
•150 elements of 5 MB HDF4 data
7/18/2015
HDF and HDF-EOS Workshop X, Landover, MD
8
General approach
HDF5 data
FTP
HDF4 data
FTP
NetCDF data
FTP
7/18/2015
HDF and HDF-EOS Workshop X, Landover, MD
9
Approach with OPeNDAP
HDF5 data
FTP
HDF4 data
server
client
NetCDF data
server
7/18/2015
HDF and HDF-EOS Workshop X, Landover, MD
10
Conclusion
• HDF5-DAP server is required
• HDF5-DAP client is strongly recommended
7/18/2015
HDF and HDF-EOS Workshop X, Landover, MD
11
Previous work
- A Prototype of DAP2-HDF5 server
•
•
•
•
Incomplete mapping from HDF5 to OPeNDAP
No comprehensive test suite
Only tested a few examples on two platforms
Incomplete documentation
7/18/2015
HDF and HDF-EOS Workshop X, Landover, MD
12
An example from SSMI
From HDF4
server at JPL
7/18/2015
HDF and HDF-EOS Workshop X, Landover, MD
From HDF5
server at NCSA
13
Current HDF5-OPeNDAP Support
HDF5
DAP
No Client
7/18/2015
Prototyped Server
HDF and HDF-EOS Workshop X, Landover, MD
14
HDF5-OPeNDAP Project
• Two-year project awarded by NASA ROSES NRA
• Funding much less than expected
• Original proposed work kept
• with significant contributions from ESDIS
• Starting date: Nov. 15th, 2006
7/18/2015
HDF and HDF-EOS Workshop X, Landover, MD
15
Proposed Work – software development
•
Implementation
•
•
•
Deployment
•
7/18/2015
HDF5-DAP2 server
HDF5-DAP4 server
Release servers to OPeNDAP community
HDF and HDF-EOS Workshop X, Landover, MD
16
HDF5-DAP2 server
•
•
Update and complete the prototype, including make
files, instructions, documentation, etc
Add a few selected features to handle NASA Aura data
and NPOESS data
-
•
•
•
7/18/2015
Map HDF5 group to DAP(?)
Map Object/regional reference to DAP(?)
Implement a comprehensive test suite
Test on platforms supported by OPeNDAP
Demonstrate with sample NASA data
HDF and HDF-EOS Workshop X, Landover, MD
17
Fully-Supported HDF5 Server
HDF5 server HDF5
DAP
HDF5 server
Matlab/Ferret Client
DAP
7/18/2015
HDF and HDF-EOS Workshop X, Landover, MD
18
HDF5
Proposed Work – software development
•
Implementation
•
•
Deployment
•
•
Release utility to OPeNDAP community
Design study
•
7/18/2015
DAP4 to HDF5 conversion utility
Feasibility of DAP-aware HDF5 library
HDF and HDF-EOS Workshop X, Landover, MD
19
Proposed Work - DAP to HDF5 utility
• A command line tool that converts DAP to HDF5 file
•
•
individual DAP variable
subsetting
Conversion tool
HDF5
HDF5 server
DAP
7/18/2015
HDF and HDF-EOS Workshop X, Landover, MD
20
Proposed work - documentations
•
•
•
•
•
DAP4 specification
HDF5-DAP4 mapping document
DAP4 to HDF5 specification
RFC for DAP-aware HDF5 library
Submit specifications to ESDSWG
7/18/2015
HDF and HDF-EOS Workshop X, Landover, MD
21
Acknowledgement
This work was supported basing upon the Cooperative
Agreement with the National Aeronautics and Space
Administration (NASA) under NASA grant
NNG05GC60A. Any opinions, findings, and
conclusions or recommendations expressed in this
material are those of the author(s) and do not necessarily
reflect the views of NASA.
7/18/2015
HDF and HDF-EOS Workshop X, Landover, MD
22