Transcript The HDF Group HDF-OPeNDAP Project Update Joe Lee and Kent Yang The HDF Group April 18, 2012 HDF/HDF-EOS Workshop XV www.hdfgroup.org.

The HDF Group
HDF-OPeNDAP
Project Update
Joe Lee and Kent Yang
The HDF Group
April 18, 2012
HDF/HDF-EOS Workshop XV
1
www.hdfgroup.org
Outline
Introduction to OPeNDAP
HDF5 Handler Update
HDF4 Handler Update
Live Demo
April 18, 2012
HDF/HDF-EOS Workshop XV
2
www.hdfgroup.org
Introduction to OPeNDAP
Goal: Background on Update
• Why OPeNDAP?
• What is HDF Handler?
• How important CF conventions?
April 18, 2012
HDF/HDF-EOS Workshop XV
3
www.hdfgroup.org
Access HDF via FTP
Aqua/Aura
Users
IDL
AIRS/OMI
FTP
Local HDF
Files
April 18, 2012
FTP
Clients
FTP
Servers
HDF/HDF-EOS Workshop XV
HDF Files
4
www.hdfgroup.org
OPeNDAP
Typical
remoteconcept
access problem
View Data
(User)
FTP/HTTP
Remote Data
(HDF4/5)
Visualization
Tools
tools
(e.g.,
(gradsdap)
GrADS)
Handler
(hdf4/5_handler)
Client Library
(libnc_dap)
Server
(Hyrax)
Data Access Protocol (DAP) via Internet
April 18, 2012
HDF/HDF-EOS Workshop XV
5
www.hdfgroup.org
Access remote data easily
April 18, 2012
HDF/HDF-EOS Workshop XV
6
www.hdfgroup.org
without worrying about data plans.
April 18, 2012
HDF/HDF-EOS Workshop XV
7
www.hdfgroup.org
Visualize remote data easily, too.
April 18, 2012
HDF/HDF-EOS Workshop XV
8
www.hdfgroup.org
HDF handler is key
Aqua/Aura
HDF
HANDLER
Users
IDV
Hyrax
NetCDF Java
AIRS/OMI
DAP
Visualization
Tools
April 18, 2012
OPeNDAP
Clients
OPeNDAP
Servers
HDF/HDF-EOS Workshop XV
HDF Files
9
www.hdfgroup.org
because if handler doesn’t follow CF
No Visualization Bonus!
April 18, 2012
HDF/HDF-EOS Workshop XV
10
www.hdfgroup.org
If handler gives up extra bonus
handler development is easier.
DAP
DAP Server
HDF / HDF-EOS
April 18, 2012
HDF/HDF-EOS Workshop XV
11
www.hdfgroup.org
What’s involved in extra bonus
Earth Scientist
Visualization
DAP Client
DAP
DAP Server
CF Conventions
HDF / HDF-EOS
April 18, 2012
HDF/HDF-EOS Workshop XV
12
www.hdfgroup.org
How handler developers feel
April 18, 2012
HDF/HDF-EOS Workshop XV
13
www.hdfgroup.org
Because we care all NASA products.
AURA
MISR
?
AQUA
TERRA
NPP
April 18, 2012
HDF/HDF-EOS Workshop XV
14
www.hdfgroup.org
Developers’ Dream
… all data are created equal,
that they shall be visualized
by all client tools easily…
April 18, 2012
HDF/HDF-EOS Workshop XV
15
www.hdfgroup.org
What if all people agreed…
development might have been much easier.
Earth Scientist
Visualization
DAP Client
DAP
DAP Server
Check once.
April 18, 2012
UDF (Imaginary)
HDF/HDF-EOS Workshop XV
16
www.hdfgroup.org
Lessons from handler development
Communicate and work together among
different groups.
Standardize everything from data products
to visualization tools.
Follow conventions and standards.
April 18, 2012
HDF/HDF-EOS Workshop XV
17
www.hdfgroup.org
Outline
Introduction to OPeNDAP
HDF5-OPeNDAP Handler Update
HDF4-OPeNDAP Handler Update
Live Demo
April 18, 2012
HDF/HDF-EOS Workshop XV
18
www.hdfgroup.org
Redesign of HDF5 handler
Goal: Support all NASA HDF5 products.
Modularize for future product support.
Follow CF
conventions robustly.
Product
April 18, 2012
HDF/HDF-EOS Workshop XV
19
www.hdfgroup.org
Why did we redesign?
In 2007,
HDF5 handler
AURA
Grid
was enough for
April 18, 2012
HDF/HDF-EOS Workshop XV
20
www.hdfgroup.org
Why did we redesign?
In 2011,
wasn’t enough for
AURA
Grid
April 18, 2012
AURA
Swath
HDF/HDF-EOS Workshop XV
AURA
ZA
21
GSSTF
www.hdfgroup.org
Why did we redesign?
In 2011,
wasn’t enough for
AURA
Grid
April 18, 2012
AURA
Swath
HDF/HDF-EOS Workshop XV
AURA
ZA
22
GSSTF
www.hdfgroup.org
and more… so we gave up!
ICESat-2
Aquarius
GOSAT
SMAP
April 18, 2012
HDF/HDF-EOS Workshop XV
23
www.hdfgroup.org
We rebuilt one that can scale up
April 18, 2012
HDF/HDF-EOS Workshop XV
24
www.hdfgroup.org
by modularizing the design
HDF5 handler
CF
Generic
HDFEOS5
HDF5
GOSAT
Aquarius
SMAP
with a
April 18, 2012
GRID
SWATH
ZA
big room for future products.
HDF/HDF-EOS Workshop XV
25
www.hdfgroup.org
Key updates on HDF5 handler
version
1.5.x
1.6.x
CF-convention
Support
Range of NASA
Data Products
Primitive
Robust
Limited
All
Mulitiple
Grids/Swath/ZA
Configuration
Options
No
Yes
Static Build
Run-time
Rigid
Flexible
Support for
Future Products
April 18, 2012
HDF/HDF-EOS Workshop XV
26
www.hdfgroup.org
HDF-EOS5 products supported
•
•
•
•
•
April 18, 2012
OMI Swath, Grid
MLS Swath
HIRDLS Swath, ZA
TES Swath, Grid
GSSTF
HDF/HDF-EOS Workshop XV
27
www.hdfgroup.org
HDF5 products supported
•
•
•
•
April 18, 2012
Aquarius Grid
MEaSUREs SeaWiFS DeepBlue
SBUV2 Ozone (O3) Profile
SMAP (Simulation)
HDF/HDF-EOS Workshop XV
28
www.hdfgroup.org
HIRDLS Zonal Average
April 18, 2012
HDF/HDF-EOS Workshop XV
29
www.hdfgroup.org
New TES Swath
April 18, 2012
HDF/HDF-EOS Workshop XV
30
www.hdfgroup.org
MEaSUREs GSSTF
April 18, 2012
HDF/HDF-EOS Workshop XV
31
www.hdfgroup.org
Aquarius Grid (Simulation)
April 18, 2012
HDF/HDF-EOS Workshop XV
32
www.hdfgroup.org
MEaSUREs SeaWiFS DeepBlue
April 18, 2012
HDF/HDF-EOS Workshop XV
33
www.hdfgroup.org
Ozone Profile (BUV)
April 18, 2012
HDF/HDF-EOS Workshop XV
34
www.hdfgroup.org
SMAP Swath (Simulation)
April 18, 2012
HDF/HDF-EOS Workshop XV
35
www.hdfgroup.org
SMAP Swath (Simulation)
April 18, 2012
HDF/HDF-EOS Workshop XV
36
www.hdfgroup.org
No CF magic for some products
Tool / CF Limitation
• GOSAT/acos: 1-D trajectory
• ICESat-2
• OMI L2G: Lat/Lon with 3-D
Java-NetCDF Client Limitation
• Dataset w/ long string
April 18, 2012
HDF/HDF-EOS Workshop XV
37
www.hdfgroup.org
It doesn’t mean handler failure
because
other tools like
can still access and
visualize data.
April 18, 2012
HDF/HDF-EOS Workshop XV
38
www.hdfgroup.org
MATLAB and IDL will work too.
HDF5 Examples under zoo will work.
• Visit http://hdfeos.org/zoo
• Replace local file access with OPeNDAP
URL access.
 MATLAB: Try new OPeNDAP functions.
• The rest will be same.
April 18, 2012
HDF/HDF-EOS Workshop XV
39
www.hdfgroup.org
Read if you can’t see
http://acdisc.sci.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMTO3e.003/2012/OMI-Aura_L3OMTO3e_2012m0416_v0032012m0418t020608.he5.ascii?lon[1029:1:1029],lat[513:1:513],_HDFEOS_GRIDS_OMI_Column_Amount_O3_Data_Fields_ColumnAmountO3[513:1:513][1029:1:1029]
2012m0418t020608.he5 lon,
Dataset: OMI-Aura_L3-OMTO3e_2012m0416_v003-
77.375 lat, 38.375
_HDFEOS_GRIDS_OMI_Column_Amount_O3_Data_Fields_ColumnAmountO3.lon, 77.375
_HDFEOS_GRIDS_OMI_Column_Amount_O3_Data_Fields_ColumnAmountO3._HDFEOS_GRIDS_O
MI_Column_Amount_O3_Data_Fields_ColumnAmountO3[_HDFEOS_GRIDS_OMI_Column_Amount_
O3.lat=38.375],
O3_Data_Fields_ColumnAmount
April 18, 2012
309.6
HDF/HDF-EOS Workshop XV
40
www.hdfgroup.org
or hear from Siri?
April 18, 2012
HDF/HDF-EOS Workshop XV
41
www.hdfgroup.org
Outline
• Introduction
• HDF5-OPeNDAP Handler Update
• HDF4-OPeNDAP Handler Update
• Live Demo
April 18, 2012
HDF/HDF-EOS Workshop XV
42
www.hdfgroup.org
HDF4 OPeNDAP handler update
• Corrected the handling the scale/offset for
MODIS products.
• Added the support for Lambert azimuthal
equal-area (LAMAZ).
• Added the support for Space-oblique
Mercator projection (SOM).
• Other minor bug fixes.
April 18, 2012
HDF/HDF-EOS Workshop XV
43
www.hdfgroup.org
Key updates on HDF4 handler
April 18, 2012
version
3.9.3
3.9.4
Some MODIS
Scale/offset
LAMAZ
projection
support
SOM
projection
support
Wrong
Corrected
No
Yes
No
Yes
HDF/HDF-EOS Workshop XV
44
www.hdfgroup.org
MODIS scale / offset correction
April 18, 2012
HDF/HDF-EOS Workshop XV
45
www.hdfgroup.org
Support for LAMAZ projection
April 18, 2012
HDF/HDF-EOS Workshop XV
46
www.hdfgroup.org
Support for SOM projection
Dataset {
UInt16 Blue_Radiance_RDQIU1[SOMBlockDimD1 = 180][XDimD2 = 128][YDimD3 = 512];
Int32 SOMBlockDimU2[SOMBlockDimD1 = 180];
Float32 LatitudeU3[XDimD2 = 128][YDimD3 = 512];
Float32 LongitudeU4[XDimD2 = 128][YDimD3 = 512];
…
}
Dataset {
UInt16 Blue_Radiance_RDQIU1[SOMBlockDimD1 = 180][XDimD2 = 128][YDimD3 = 512];
Int32 SOMBlockDimU2[SOMBlockDimD1 = 180];
Float32 LatitudeU3[SOMBlockDim = 180][XDimD2 = 128][YDimD3 = 512];
Float32 LongitudeU4[SOMBlockDim = 180][XDimD2 = 128][YDimD3 = 512];
…
}
April 18, 2012
HDF/HDF-EOS Workshop XV
47
www.hdfgroup.org
HDF-EOS2 products supported
GES DISC
• AIRS / MERRA / TOMS
LAADS / LP DAAC / NSIDC
• Many MODIS products
NSIDC
• AMSR_E / NISE / MODIS
LaRC
• MISR / MOPITT / some CERES (TRMM)
April 18, 2012
HDF/HDF-EOS Workshop XV
48
www.hdfgroup.org
HDF4 products supported
GES DISC
• TRMM L1B, L2B Swath
• TRMM L3 Grid 42B & 43B products
OBPG (Ocean Color)
• SeaWiFS / MODIST / MODISA
• CZCS / OCTS
April 18, 2012
HDF/HDF-EOS Workshop XV
49
www.hdfgroup.org
HDF4 products supported
CERES
•
•
•
•
•
•
•
April 18, 2012
CER_AVG
CER_ES4
CER_ISCCP-D2like-Day
CER_ISCCP-D2like-GEO
CER_SRBAVG
CER_SYN
CER_ZAVG
HDF/HDF-EOS Workshop XV
50
www.hdfgroup.org
HDF4 handler limitations
• HDF-EOS2 + additional HDF4 objects
• In some products, some additional HDF4
objects will be ignored.
not listed
• Other HDF4 products
in
previous slides are not tested and may
not be visualized by Java OPeNDAP
clients.
April 18, 2012
HDF/HDF-EOS Workshop XV
51
www.hdfgroup.org
Other limitations
CF conventions Limitation
• _FillValue inside latitude and longitude
Java-NetCDF Limitation
• Java Heap Memory
April 18, 2012
HDF/HDF-EOS Workshop XV
52
www.hdfgroup.org
Outline
• Introduction
• HDF5-OPeNDAP Handler
• HDF4-OPeNDAP Handler
• Live Demo
April 18, 2012
HDF/HDF-EOS Workshop XV
53
www.hdfgroup.org
We’ve updated handlers
HDFEOS2
HDF4
OPeNDAP
HDF
Handlers
HDF5
HDFEOS5
… and will continue.
April 18, 2012
HDF/HDF-EOS Workshop XV
54
www.hdfgroup.org
GES DISC Uses Handlers
April 18, 2012
HDF/HDF-EOS Workshop XV
55
www.hdfgroup.org
Please try HDF4/HDF5 handlers
New hdf5_handler will be released in July.
Source codes and RPMs:
http://opendap.org
Demo HDF5/HDF4 servers:
http://hdfeos.org/zoo/hdf4_handler/
http://hdfeos.org/zoo/hdf5_handler/
April 18, 2012
HDF/HDF-EOS Workshop XV
56
www.hdfgroup.org
and give us feedback.
April 18, 2012
HDF/HDF-EOS Workshop XV
57
www.hdfgroup.org
The HDF Group
Thank you!
April 18, 2012
HDF/HDF-EOS Workshop XV
58
www.hdfgroup.org
Acknowledgements
This work was supported by Subcontract number
114820 under Raytheon Contract number
NNG10HP02C, funded by the National Aeronautics
and Space Administration (NASA) and by
cooperative agreement number NNX08AO77A from
the NASA. Any opinions, findings, conclusions, or
recommendations expressed in this material are
those of the authors and do not necessarily reflect
the views of Raytheon or the National Aeronautics
and Space Administration.
April 18, 2012
HDF/HDF-EOS Workshop XV
59
www.hdfgroup.org
The HDF Group
Questions/comments?
[email protected]
April 18, 2012
HDF/HDF-EOS Workshop XV
60
www.hdfgroup.org