Transcript HDF5 OPeNDAP Project Update and Demo

HDF5 OPeNDAP Project
Update and Demo
MuQun Yang and Hyo-Kyung Lee
(The HDF Group)
James Gallagher
(OPeNDAP, Inc.)
10/17/2008
HDF and HDF-EOS Workshop XII
1
OPeNDAP Review
• A software framework that allows
simple access to remote data
• Data Access Protocol
• Client-server model
• Widely used by Earth Science
Community
10/17/2008
HDF and HDF-EOS Workshop XII
2
OPeNDAP
Concept
Typical
Remote
Access
View Data
(User)
FTP/HTTP
Remote Data
(HDF4)
Visualization
Tools
tools
(e.g.
(gradsdap)
grads)
Handler
(hdf4_handler)
Client Library
(libnc-dap)
Server
(apache)
DAP Protocol (via http)
10/17/2008
HDF and HDF-EOS Workshop XII
3
OPeNDAP HDF5 Handler
View Data
(User)
Remote HDF5
HDF5 Handler
Server
(apache)
DAP Protocol (via http)
10/17/2008
HDF and HDF-EOS Workshop XII
4
Challenges
Mapping HDF5 to DAP
• Compound Datatype
• Groups
• Object/Regional References
10/17/2008
HDF and HDF-EOS Workshop XII
5
OPeNDAP HDF5 Handler with HDF-EOS5
View Data
(User)
Remote
HDF-EOS5
HDF5
Visualization
tools
(grads)
HDF5 Handler
Client Library
(libnc-dap)
Server
(apache)
DAP Protocol (via http)
10/17/2008
HDF and HDF-EOS Workshop XII
6
Challenges: HDF- EOS5
• Grid with No Geolocation data
 Clients expect Grid with Geolocation data
• Some Attributes stored as Extremely Long
String(s) (e.g. StructMetada.0)
 Clients expect structured attributes
10/17/2008
HDF and HDF-EOS Workshop XII
7
Tweaks for HDF-EOS5
Added two default HDF5 handler
configuration options:
• --enable-eos-grid
• --enable-eos-meta
10/17/2008
HDF and HDF-EOS Workshop XII
8
Challenges: OPeNDAP Clients
• Need special attributes on dataset.
• Need shared geolocation variables.
10/17/2008
HDF and HDF-EOS Workshop XII
9
Tweaks for OPeNDAP Clients
Added two optional handler
configuration options:
• --enable-short-name
• --enable-CF
10/17/2008
HDF and HDF-EOS Workshop XII
10
OPeNDAP HDF5 Handler with HDF-EOS5
View Data
(User)
Remote
HDF-EOS5
HDF5
Visualization
tools
(grads)
HDF5 Handler
w/ CF options
Client Library
(libnc-dap)
Server
(apache)
DAP Protocol (via http)
10/17/2008
HDF and HDF-EOS Workshop XII
11
Day After Server Tweaks
• Finally, Happy Clients!
10/17/2008
HDF and HDF-EOS Workshop XII
12
Problems of Tweaking HDF5 Handler
Remember that we added two optional
HDF5 handler configuration options:
1. --enable-short-name
2. --enable-CF
Causes ambiguity among
variable names
(e.g. /GroupA/ozone vs.
/GroupB/ozone)
Drops some key attributes
(e.g. StructMetdata, HDF_ROOT_GROUP)
10/17/2008
HDF and HDF-EOS Workshop XII
13
HDF5-Friendly OPeNDAP Client Library
View???
Groups
HDF5 Groups
View Data
(User)
Remote
HDF5
Visualization
tools
(grads)
HDF5 Handler
w/ CF options
HDF5-Friendly
Client Library
(liboc-dap)
(libnc-dap)
Server
(apache)
DAP Protocol (via http)
10/17/2008
HDF and HDF-EOS Workshop XII
14
Traditional OPeNDAP
clientin HDF5
Example:
Groups
library: It’s an attribute that I
don’t understand. I’ll ignore it.
HDF5-Friendly
OPeNDAP client
library: I was waiting
for this key attribute to
re-construct HDF5
10/17/2008
HDF and HDF-EOS Workshop XII
15
Example: Reference in HDF5
• Important for NPOESS
• Object / Regional Reference
• Map to DAP URL
10/17/2008
HDF and HDF-EOS Workshop XII
16
Example: dap2h5
View Group/Ref.
Group/Ref.
View Data
(User)
Remote
HDF5
• HDF5-F.
A test Client
application for the client library
HDF5
Handler
• ItLibrary
canApp.
construct HDF5 from
DAP
output
(dap2h5)
HDF5-Friendly
Client Library
(liboc-dap)
Server
(apache)
DAP Protocol (via http)
10/17/2008
HDF and HDF-EOS Workshop XII
17
Our Client Library(prototype)
• One more reason:
Help Clients to view Swath
Visualizing
Visualizing
HDF-EOS5
HDF-EOS5
Swath
Grids
Problem
View???
Swath
Swath
View Data
(User)
Remote
HDF-EOS5
HDF5
Visualization
Tools
tools
(gradsoc)
(grads)
HDF5 Handler
w/ CF option
HDF5-Friendly
Client Library
(liboc-dap)
(libnc-dap)
Server
(apache)
DAP Protocol (via http)
10/17/2008
HDF and HDF-EOS Workshop XII
19
Demo: MLS swath via GrADS
• GrADS coupled with our client
library
• The client library provides grid
mapping from swath data
• GrADS displays swath through
HDF-EOS5 specific client library
API calls
Summary of Client Library (Prototype)
• Finished coding
• Tested with the GrADS visualization
client
• Documented
• Working on a demo DAP to HDF5
tool by using the client library
prototype
• Will test with NCL if time allows
10/17/2008
HDF and HDF-EOS Workshop XII
21
Caution
• Our Client Library is ONLY A PROTOTYPE!
 It does NOT support all DAP data types
 It does NOT support all AURA files
 It does NOT support all Visualization clients
10/17/2008
HDF and HDF-EOS Workshop XII
22
Summary
• HDF5 Access via OPeNDAP
 Is Easy
 Is used by GES DISC to serve Aura
files
• HDF5-Friendly OPeNDAP Client Library
 Serves HDF5 better(EOS swath)
 Benefits visualization clients
10/17/2008
HDF and HDF-EOS Workshop XII
23
Future Work
• HDF5 to DAP2 Mapping Document
• Release HDF5-friendly OPeNDAP
Client Library (Prototype)
• URL: http://hdfgroup.org/projects/opendap
10/17/2008
HDF and HDF-EOS Workshop XII
24
Credits
Mike Folk (The HDF Group)
Robert McGrath (NCSA)
Peter Leonard, Daniel Kahn, Marghi Hopkins (ADNET)
Christopher Lynnes, James Johnson, Denis Nadeau, Fan Fang
(NASA)
Jennifer Adams (GrADS)
Dave Brown (UCAR)
10/17/2008
HDF and HDF-EOS Workshop XII
25
Acknowledgement
This work was supported basing upon the
Cooperative Agreement with the National
Aeronautics and Space Administration (NASA)
under NASA grant NNX06AC83A,
NNX08A077A and NNX06AG75A. Any opinions,
findings, and conclusions or recommendations
expressed in this material are those of the
author(s) and do not necessarily reflect the
views of NASA.
10/17/2008
HDF and HDF-EOS Workshop XII
26