Transcript ESIP Summer Meeting

The HDF Group
Easy access HDF files via Hyrax
Kent Yang
The HDF Group
July 8 – 11, 2014
ESIP Summer Meeting
1
www.hdfgroup.org
Outline
•
•
•
•
•
Why OPeNDAP?
Hyrax features
Hyrax modules
Demos
Suggestions
July 8 – 11, 2014
ESIP Summer Meeting
2
www.hdfgroup.org
Why OPeNDAP?
• Check metadata remotely (in various forms)
• Obtain the subset of data easily and efficiently
• Hide the original data sources
• netCDF, HDF4, HDF5 or GeoTiFF, GRIB
• Many popular earth science tools can visualize
and analyze the data via OPeNDAP
• OPeNDAP output(including subsets) can be
downloaded as other formats
July 8 – 11, 2014
ESIP Summer Meeting
3
www.hdfgroup.org
Doesn’t help
Great!
July 8 – 11, 2014
ESIP Summer Meeting
4
www.hdfgroup.org
HDF(5) Hyrax modules
• The visualization of HDF(5) data via Hyrax
• What NASA users request the most
July 8 – 11, 2014
ESIP Summer Meeting
5
www.hdfgroup.org
Helpful Hyrax feature – BES keys
• BES keys to dynamic change the program flow
• Easy for users to choose what they want
• Easy for developers to add/revise features
requested by users
July 8 – 11, 2014
ESIP Summer Meeting
6
www.hdfgroup.org
Example on using BES keys
Float
CERES_Unfiltered_Radiances_all_toa_sw_rad[gmt_hour_index = 24][latitude =180][longitude= 360
Float /CERES_Unfiltered_Radiances/all_toa_sw_rad[24][180][360]
July 8 – 11, 2014
ESIP Summer Meeting
7
www.hdfgroup.org
Hyrax modulation
• Core framework implemented in C++
• Just need to provide DDS, DAS and Data to Hyrax
libdap and BES
• A handler can be sophisticated
• HDF handler examples
• Retrieve latitude and longitude for HDF-EOS grid
• Interpolate the latitude and longitude for HDF-EOS
swath with dimension map
• Make sure the dimensions, variable names and
attributes follow CF conventions
July 8 – 11, 2014
ESIP Summer Meeting
8
www.hdfgroup.org
Non-CF
CF
Float /CERES_Unfiltered_Radiances/all_toa_sw_rad[24][180][360]
/CERES_Unfiltered_Radiances/all_toa_sw_rad[24][180][360]
Float
Float CERES_Unfiltered_Radiances_all_toa_sw_rad[gmt_hour_index = 24][latitude =180][longitude= 360]
July 8 – 11, 2014
ESIP Summer Meeting
9
www.hdfgroup.org
Demos – HDF-EOS2 (AIRS grid)
July 8 – 11, 2014
ESIP Summer Meeting
10
www.hdfgroup.org
Demo 2 – HDF-EOS2 (MODIS swath)
Swath with Dimension Map
July 8 – 11, 2014
ESIP Summer Meeting
11
www.hdfgroup.org
Demo 3 – HDF-EOS5 (OMI grid)
July 8 – 11, 2014
ESIP Summer Meeting
12
www.hdfgroup.org
Hyrax modules that help the easy access
• Filenet CDF module
• Can download DAP output as a netCDF-3 or netCDF4 classic files
• NcML module
• Add/delete/revise metadata information of the
data source
• Aggregate granules
July 8 – 11, 2014
ESIP Summer Meeting
13
www.hdfgroup.org
Demo
• GPM level 1B(HDF5)
July 8 – 11, 2014
ESIP Summer Meeting
14
www.hdfgroup.org
HDF(5) and file netCDF demo
• FilenetCDF module can work with the HDF
handlers to download the HDF files or subset of
the files as netCDF-3 or netCDF-4 classic files that
follow the CF conventions
July 8 – 11, 2014
ESIP Summer Meeting
15
www.hdfgroup.org
HDF(5) and NcML demo
• NcML module can be used with HDF handlers to
provide the missing CF information
• AVHRR example
July 8 – 11, 2014
ESIP Summer Meeting
16
www.hdfgroup.org
HDF(5), file netCDF and NcML demo
• HDF handlers, NcML module and file netCDF
module can work together
• AVHRR example
July 8 – 11, 2014
ESIP Summer Meeting
17
www.hdfgroup.org
Summary of modules to serve HDF files
HDF(5)
files
DAP clients
(IDV, Ferret,
Panoply, IDL…...)
Main engine
Alternative output
File netCDF
HDF(5) handlers
Hyrax core
Supplement Info
NcML
netCDF
files
July 8 – 11, 2014
ESIP Summer Meeting
18
www.hdfgroup.org
Demo - Beyond the web service
• BES standalone program
besstandalone
/etc/bes/bes.conf
-i in4-GPM-l1b
> GPM-l1b.nc4
• Connect-cthe
HDF handlers,
the file netCDF
module
and the NcML module
• End users can build Hyrax and use this program as a
command-line tool
• Can it be a standalone tool?
July 8 – 11, 2014
ESIP Summer Meeting
19
www.hdfgroup.org
Future - Performance
• Improve the BES caching mechanism
• Critical for large file access and aggregation service
July 8 – 11, 2014
ESIP Summer Meeting
20
www.hdfgroup.org
The HDF Group
Earth Science Group
Ted Habermann
Aleksandar Jelenak
H. Joe Lee
Joel Plutchak
John Readey
Kent Yang
July 8 – 11, 2014
ESIP Summer Meeting
21
www.hdfgroup.org