HDF Performance Framework
Download
Report
Transcript HDF Performance Framework
HDF5 OPeNDAP Project
MuQun Yang and Hyo-Kyung Lee (THG)
James Gallagher (OPeNDAP, Inc.)
Question of the Day
• How can I get One Sub from SUBWAY?
• Chicken teriyaki with no onion
• How can I get One Subset of EOS data
image from NASA?
• Today’s satellite image right above the roof
Subway
How can I get One Sub from SUBWAY?
• Easy: anywhere / accepts Visa|MC
• Efficient: your own recipe
• Cool: eat fresh and look good
Dumb-way
How can I get One Subset of EOS data image from NASA?
• Difficult: bring a USB drive and visit
NASA
• Inefficient: download 10 Gbyte and
search for 10 byte dataset
• Not cool: wrong or useless dataset
DAP-way!
How can I get One Subset of EOS data image from NASA?
• Easy: anywhere / accepts IE|FF
• Efficient: WYSIWYG – less fat!
• Cool: Visualization clients – see fresh!
DAP-way Implementation
• OPeNDAP: Free DAP-way Software
• Server-Client Model APIs
• Widely used by Earth Science Community
OPeNDAP server sites
OPeNDAP
• Market Dominance:
•
• http://www.opendap.org
Courtesy of OPeNDAP, Inc.
OPeNDAP Servers
CODAR
netCDF HDF4
Data
Data
CODAR
netCDF
Data
Matlab
DSP
Tables
SQL
FITS
CDF
Flat
Binary
Data
Data
Data
Data
Data
Data
Data
Matlab
HDF4
JGOFS
DSP
FITS
JDBC
Courtesy of OPeNDAP, Inc.
CEDAR
General
Data
Data
ESML
FreeFrom
CDF
CEDAR
OPeNDAP Clients
netCDF C
Ferret
GrADS
netCDF Java
IDV
VisAD
ncBrowse
Matlab
Client
IDL
Client
Matlab
IDL
Courtesy of OPeNDAP, Inc.
Access
Excel
How OPeNDAP works
OPeNDAP
NetCDF
Server
OPeNDAP
Client
OPeNDAP
HDF4 Server
NetCDF
HDF4
Architecture of OPeNDAP Server
• Server 3 (CGI)
• C/C++ based HTTP Server
• Insecure / Inflexible
• Server 4 (Hyrax)
• Java based TomCat Servlet Engine
• Secure / Flexible
Server43(Hyrax)
(CGI) Architecture
Server
Architecture
HTTP Server
DAP
BES Commands
OLFS
BES
XML, GIS, KML
HTML
XML- encapsulated object
Java
Servlet Engine
Unix Daemon
Optional catalogs:
XML, GIS, KML
Data
Store
File system with data
files like HDF,
NetCDF and SQL
Database, …
http://docs.opendap.org/index.php/Hyrax
Courtesy of OPeNDAP, Inc.
Example Usage
OPeNDAP
Client
OPeNDAP
HDF4
HDF4 Server
Actual Content of Data
Semantic Meaning of Data
Syntactic Structure of Data
OPeNDAP Visualization Client
What about HDF5?
OPeNDAP
NetCDF
Server
OPeNDAP
Client
NetCDF
OPeNDAP
HDF4 Server
HDF4
OPeNDAP
HDF5 Server
HDF5
Why Important?
• HDF5: NASA EOS / NPOESS
• NASA’s MISSION:
ACCESS!
• Our MISSION: Build DAP-way Bridge
Goals
•1st: Transform HDF5 access in DAP-way
• 2nd: Yet preserve the beauty of HDF5
OPeNDAP HDF5 Server
Goal: Transform HDF5 access in DAP-way
1. Map HDF5 Datatypes in DAP-way
2. Plus, tweaks for OPeNDAP Clients
3. Plus, tweaks for HDF-EOS Files
Mapping HDF5 in DAP-way
• Prototype server in 2001
• NASA Grant from Nov. 2006
• Support for Compound Datatype
• Support for Group
• Support for References / Links
• First Product Release in Mar. 2008
OPeNDAP HDF5 Server
Goal: Transform HDF5 access in DAP-way
1. Map HDF5 Datatypes in DAP-way
2. Plus, tweaks for OPeNDAP Clients
3. Plus, tweaks for HDF-EOS Files
Subway Customers
• I want 3 inch-long sub
• I need Wasabi sauce
• I eat only wheat bread
• I eat only meat balls
OPeNDAP Clients
Not all OPeNDAP clients are created equal!
• I hate foot-long variable names.
• I need special attributes on dataset.
• I care pre-defined Grid data type.
• I care only well-formed attributes.
Tweaks for OPeNDAP Clients
Two configuration options
• --enable-short-name = cut bread
• --enable-CF = put Wasabi sauce
OPeNDAP HDF5 Server
Goal: Transform HDF5 access in DAP-way
1. Map HDF5 Datatypes in DAP-way
2. Plus, tweaks for OPeNDAP Clients
3. Plus, tweaks for HDF-EOS5 Files
Subway Customers
• I want 3 inch-long
• I need Wasabi sauce
• I eat only wheat bread
• I eat only meat balls
Subway Customers - OK
• I eat only wheat bread
• I eat only meat balls
Subway Suppliers Problem
• No breads, only wheat!
• No meat balls, only meat!
HDF EOS
Not all HDF5 files are created for DAP-way!
• Grid with No Dimension data
• Clients expect Grid with Dimension data
• Metadata Attribute in Two Strings
• Clients expect One Structured Format
Tweaks for HDF-EOS
Two more configuration options
• --enable-eos-grid = bake bread
• --enable-eos-meta = make balls
OPeNDAP HDF5 Server
Goal: Transform HDF5 access in DAP-way
1. Map HDF5 Datatypes in DAP-way
2. Plus, tweaks for OPeNDAP Clients
3. Plus, tweaks for HDF-EOS5 Files
Day After Server Tweaks
• Finally, Happy Clients!
Live Demo
Ozone concentration level over the south pole
Goals
•1st: Transform HDF5 access in DAP-way
• 2nd: Yet Preserve the beauty of HDF5
HDF5 Served in DAP-way
(Yet Preserve the beauty of HDF5)
• There are some things money can’t buy.
For everything else, there is MasterCardTM.
Minimize
• There are some things DAP can’t serve.
For everything else, there is _________?
Maximize
Some Things
(that OPeNDAP HDF5 server can’t serve)
• Hard: Opaque, Bitmap, Enum, 64 bit
Integer, Variable Length types
• Illegal: Reserved characters in DAP are
used in Dataset/Group name in HDF5
For Everything Else
• Let there be HDF5-friendly OPeNDAP
client library!
• Package and Deliver in DAP-way first.
• Then, let the client library handle it.
• Concept of Trojan Horse
Traditional OPeNDAP client
library: It’s an attribute that I
don’t understand. I’ll ignore it.
Example: Group in HDF5
HDF5-friendly
OPeNDAP client library:
I was waiting for this key
attribute to re-construct
HDF5
Example: Reference in HDF5
• Object / Regional Reference
• Map to DAP URL at server
• No de-referencing of URL at client library
• Important for NPOESS
HDF5-Friendly OPeNDAP Client Library
No Latitude and Longitude
• One more reason: Tame Clients
• Only ODC can display Swath properly.
• Easy but Evil OPeNDAP NC Client Library!
easy: nc style API
evil: fixed dimensional attributes
• Our client library must be Easy but Good
Courtesy of NASA
HDF5-Friendly
OPeNDAPClient
Client Library
NC-Friendly
OPeNDAP
Library
View G/R/Swath
???
View NetCDF
View HDF5
Group/Ref/Swath
NetCDF
HDF5
GrADS
OPeNDAP
HDF5
NC Server
Server
libnc-dap
liboc-dap
dapserver
libdap
Summary
• DAP-way access of HDF5
Is Easy / Efficient / Cool
Loses some things
Requires HDF5-friendly OPeNDAP
Client Library
• HDF5-friendly OPeNDAP Client Library
Serves HDF5 better
Tames evil visualization clients
Future Work
• HDF5 DAP Mapping Document
• Finish HDF5-friendly OPeNDAP Client
Library prototype
• Test it on GrADS and display Swath
Project Website
http://hdfdap.hdfgroup.uiuc.edu/joomla
• Beta preview
• Feedbacks are more than welcome!
Credits
Mike Folk (THG)
Robert McGrath (NCSA)
Peter Leonard, Daniel Kahn, Marghi Hopkins (ADNET)
Christopher Lynnes, James Johnson, Denis Nadeau (NASA)
Jennifer Adams (GrADS)
Dave Brown (UCAR)