Transcript Title of Presentation - HDF-EOS

Improving the usability of
HDF-EOS2 data
Kent Yang, Joe Lee, Choonghwan Lee
The HDF Group
March 31st, 2009
7/20/2015
Annual briefing to ESDIS
1
EOSDIS Evolution 2015 Vision Tenets
Vision Tenet
Archive Management
EOS Data
Interoperability
Future Data Access
and Processing
Data Pedigree
Cost Control
Vision 2015 Goals
 NASA will ensure safe stewardship of the data through its lifetime.
 The EOS archive holdings are regularly peer reviewed for scientific merit.
 Multiple data and metadata streams can be seamlessly combined.
 Research and value added communities use EOS data interoperably with other
relevant data and systems.
 Processing and data are mobile.
 Data access latency is no longer an impediment.
 Physical location of data storage is irrelevant.
 Finding data is based on common search engines.
 Services invoked by machine-machine interfaces.
 Custom processing provides only the data needed, the way needed.
 Open interfaces and best practice standard protocols universally employed.
 Mechanisms to collect and preserve the pedigree of derived data products are readily
available.
 Data systems evolve into components that allow a fine-grained control over cost
drivers.
User Community
Support
 Expert knowledge is readily accessible to enable researchers to understand and use
the data.
 Community feedback directly to those responsible for a given system element.
IT Currency
 Access to all EOS data through services at least as rich as any contemporary science
information system.
Feb 3, 2005
7/20/2015
Annual briefing to ESDIS
2
Purpose of the talk
• Identify problems to access HDF-EOS2 data
• Share our plan to help improve the EOS2
usability
7/20/2015
Annual briefing to ESDIS
3
The success of EOS mission
MODIS HDF-EOS data to detect sand storm from Gobi china
Courtesy NASA Earth Observatory
7/20/2015
Annual briefing to ESDIS
4
The success of EOS mission
MODIS HDF-EOS data to detect fires in Africa
Courtesy NASA
7/20/2015
Annual briefing to ESDIS
5
Identify problems :
technical part
7/20/2015
Annual briefing to ESDIS
6
Beauty of HDF-EOS2
• Very close to physical model
• Convenient to handle geo-location projections
• Files are generally compact
 No needs to store latitude and longitude for grid
 Reduce the size of latitude and longitude for
swath
7/20/2015
Annual briefing to ESDIS
7
A compact HDF-EOS2 Grid
• MODIS (MYD10A2) Grid file
• Two 2400 X 2400 grid data fields with gzip
compression
• File size: 75 KB
• If lat/lon are kept in the file
• File size may become 44 MB,
500 times bigger than the original file size
7/20/2015
Annual briefing to ESDIS
8
A compact HDF-EOS2 swath
with dimension
withmaps
dimension
maps
coarse_cross
• Small
along file size
406 – Swath
cross
Longitude
w/o dimension
maps
4060
271
2708
0.5 MB
42 MB
(406 X 271 X 4)
(4060 X 2708 X 4)
CloudCover(along, cross)
Latitude(coarse_along, coarse_cross)
Longitude(coarse_along,
coarse_cross)9
7/20/2015
CloudCover(along,
cross)
Annual
briefing
Latitude(along, cross)
to
Longitude(along,ESDIS
cross)
There are issues to access
some HDF-EOS2 files
Aims to improve the usability
7/20/2015
Annual briefing to ESDIS
10
First of all, there should be more
communications among providers, tool
developers, users and data storage
software developers.
Focus on HDF-EOS2 and HDF4 in the
next few slides.
7/20/2015
Annual briefing to ESDIS
11
Schematic of Grid
YDim
XDim
StructMetadata.0 =
“… XDim= 14 YDim = 8
UpperLeftPointMtrs =(-156.0,71.0)LowerRightMtrs =(180.0,-75.0)
… Projection = HE5_GCTP_GEO …”
HDF-EOS5 GRIDS:
Float32 Ozone(YDim, XDim)
7/20/2015
Annual briefing to ESDIS
12
Issue 1: lack of strict checking
in HDF-EOS2 library
7/20/2015
Annual briefing to ESDIS
13
Lack of strict checking in HDF-EOS2 library
• XDim and YDim exist and have values
 .e.g, OMI (OMUVBL3)
 Generally, XDim and YDim do not exist because
they are implied.
 XDim and YDim do not need to have values
because they can be calculated. Values are just
redundant.
7/20/2015
14
Annual
briefing
to
ESDIS
Lack of strict checking in HDF-EOS2 library
StructMetadata
Data Field
…
Band_1KM_RefSB
•Band_1KM_RefSB[4]
Inconsistency in the number of
elements HDF… EOS2 informs and HDF-EOS2 actually reads
 e.g., MODIS (MAC021S0, MAC02QS0,
4 elements
MAC02QS1, MAC021S1)
 We believe that this data
field had four elements
consistent
once and it was appended later.15 elements
INCONSISTENT!
7/20/2015
15
Annual
briefing
to
ESDIS
Lack of strict checking in HDF-EOS2 library
• Wrong grid parameters
 e.g., MISR (MI3DAER)
 Projection is defined by code, projection
parameters, etc.
 For Lambert Azimuth projection, the fifth
parameter specifies “longitude of center of
projection”.
 Its value is entered in DDDMMMSSS.SS format.
 -92970000.00
minutesseconds
970?
degrees -92
0
7/20/2015
Annual briefing to ESDIS
16
Issue 2: lack of standard way
to handle some geolocation
fields
7/20/2015
Annual briefing to ESDIS
17
Schematic of Swath
StructMetadata.0 =
“ … DimensionName=”along”
size= 11
DimensionName=”cross” size= 3
…”
Data fields:
Float32 CloudCover(along,cross)
Geolocation fields:
Float32 lat(along,cross)
Float32 lon(along,cross)
7/20/2015
Annual briefing to ESDIS
18
No standard way to handle geolocation fields
• No standard way to figure out the vertical
coordinate in an HDF-EOS2 file
 A data field
• TAirStd[GeoTrack][GeoXTrack] [StdPressureLev]
 [StdPressureLev]
StdPressureLev is a dimension
 Additional data field supplies values, probably
• pressStd [StdPressureLev]
 How can a user know for sure that the field
pressStd provides the vertical coordinate
information?
7/20/2015
Annual briefing to ESDIS
19
Issue 3: No enforcement for
the use of standard names
7/20/2015
Annual briefing to ESDIS
20
Schematic of Grid
YDim
XDim
StructMetadata.0 =
“… XDim= 14 YDim = 8
UpperLeftPointMtrs =(-156.0,71.0)LowerRightMtrs =(180.0,-75.0)
… Projection = HE5_GCTP_GEO …”
HDF-EOS5 GRIDS:
Float32 Ozone(YDim, XDim)
7/20/2015
Annual briefing to ESDIS
21
No enforcement for the use of standard names
• Use of non-standard dimensions in grid
 e.g., MOPITT (MOP03)
 To be interpreted properly, each data field must
make use of the two predefined dimensions:
XDim and YDim
 However, none of data fields in that file refers to
XDim or YDim. Instead, they refer to nlon and
nlat, which are not standard way.
7/20/2015
Annual briefing to ESDIS
22
No enforcement for the use of standard names
• Object names are not restrictive
 e.g., MODIS (MAC03S0)
 Name of a data field called
• Land/SeaMask
 A dimension called
• nscans*10
 These names may not be safe in other APIs.
Especially, HDF5 does not allow slash(/).
7/20/2015
Annual briefing to ESDIS
23
Other bad practices
• Extreme long attributes
• Chunk size much bigger than dimensional size
for fixed size SDS
• These are not enforced by HDF4 library
7/20/2015
Annual briefing to ESDIS
24
Other technical difficulties
• Hybrid HDF-EOS2 files
 Some data cannot be accessed by HDF-EOS2
library
• Need to help users access these files
7/20/2015
Annual briefing to ESDIS
25
Identify problems :
service part
7/20/2015
Annual briefing to ESDIS
26
A message from DAAC User Survey
I am a student and I need to read
MODIS HDF-EOS data. But it is hard for
me to extract the data. I need a fortran
program to extract the data. I wish
DAAC can provide this.
7/20/2015
Annual briefing to ESDIS
27
DAAC tool sites
• Provide most information users need
7/20/2015
Annual briefing to ESDIS
28
What’s left?
• Some tools are out of date
• On some sites, no C/fortran examples to access
the data
• Information on the tool page out of date
7/20/2015
Annual briefing to ESDIS
29
What’s the problem?
• A small percentage of EOS2 users needs help.
 Access “difficult” files
 Compensate tools/examples for DAACs
7/20/2015
Annual briefing to ESDIS
30
The HDF Group’s plan to DAACs
• Compensate fortran/C example code for DAACs
• C/fortran examples for hybrid files and “difficult”
files
• Evaluate/update tools listed at hdfeos.org
• Evaluate tools at DAACs
• EOS-forum
 Web forum?
 EOS2 user and developer mailing lists?
7/20/2015
Annual briefing to ESDIS
31
Thank You!
7/20/2015
Annual briefing to ESDIS
32
Acknowledgements
This work was supported by the Cooperative
Agreement with the National Aeronautics and
Space Administration (NASA) under NASA grant
NNX06AC83A and NNX08A077A.
Any opinions, findings, conclusions or
recommendations expressed in this material are
those of the author(s) and do not necessarily
reflect the views of NASA.
7/20/2015
Annual briefing to ESDIS
33
Questions/comments?
7/20/2015
Annual briefing to ESDIS
34