Transcript HDF5 in support of heterogeneous databases - HDF-EOS

HDF Update
Mike Folk, Kent Yang, Elena Pourmal
The HDF Group
March 31st, 2009
March 31, 2009
Annual HDF Briefing to ESDIS
1
• HDF Project Update
•
•
•
•
•
•
Helpdesk
HDF Releases
Library release highlights
Quality Assurance
Java Product Highlights
Other Tools Activities
• Outreach
• Related projects and activities
• EOS Support
• OPeNDAP
• netCDF Support
• Other projects of interest
• Platform support issues
HDF Project Update
March 31, 2009
Annual HDF Briefing to ESDIS
3
Helpdesk
statistics
March 31, 2009
Annual HDF Briefing to ESDIS
4
Number of HDF4 and HDF5 issues
since 2000
1000
900
Number of issue
800
700
600
500
HDF4
400
HDF5
300
200
100
0
2000 2001 2002 2003 2004 2005 2006 2007 2008
Years
March 31, 2009
Annual HDF Breifing to ESDIS
5
HDF4 issues by category
120
Number of issues
100
80
2005
2006
60
2007
2008
40
20
0
build
March 31, 2009
library
util
java and doc
Annual HDF Breifing to ESDIS
apps
general
6
HDF5 issues by category
350
300
Number of issues
250
2005
200
2006
2007
150
2008
100
50
0
build
March 31, 2009
library
util
java
doc
Annual HDF Breifing to ESDIS
apps
general
7
HDF issues by domain
600
Number of issues
500
400
2005
2006
300
2007
2008
200
100
0
gov
March 31, 2009
com
edu
net
Annual HDF Breifing to ESDIS
mil
org
8
HDF issues resolution time
45
Percent of all reported issues
40
35
30
25
2005
2006
20
2007
2008
15
10
5
0
< 4 hours < 1 day < 2 days < 1 week
March 31, 2009
<2
weeks
<1
month
Annual HDF Breifing to ESDIS
<6
month
not
closed
9
HDF open and closed bugs
450
400
Number of bugs
350
300
250
Closed
200
Open
150
100
50
0
2004
March 31, 2009
2005
2006
2007
Annual HDF Breifing to ESDIS
2008
10
HDF5 open and closed bugs
350
300
Number of bugs
250
200
Open
Closed
150
100
50
0
2004
March 31, 2009
2005
2006
2007
Annual HDF Breifing to ESDIS
2008
11
HDF4 open and closed bugs
50
45
40
Number of bugs
35
30
Open
25
Closed
20
15
10
5
0
2004
March 31, 2009
2005
2006
2007
Annual HDF Breifing to ESDIS
2008
12
QUESTIONS?
March 31, 2009
Annual HDF Breifing to ESDIS
13
HDF RELEASES
March 31, 2009
Annual HDF Breifing to ESDIS
14
HDF Releases in 2008 - 2009
• HDF4
• HDF4r2.4 (February 2009)
• HDF5
• HDF5-1.8.1 (May 2008)
• HDF5-1.6.8 (November 2008)
• HDF5-1.8.2 (November 2008)
• HDF- Java Products
• Version 2.5 (February 2009)
• H4toH5
• Version 2.0 (June 2008)
• Version 2.1 (February 2009)
March 31, 2009
Annual HDF Breifing to ESDIS
15
HDF LIBRARIES
RELEASE HIGHLIGHTS
March 31, 2009
Annual HDF Breifing to ESDIS
16
HDF4r2.4
• Fixed a flaw in the HDF4 design exposed by
applications that use HDF-NetCDF interface to
access both HDF4 and NetCDF-3 files with
datasets/variables with unlimited dimensions.
• Added several new APIs including API to get
compressed size of data.
• Updated hdp and hrepack to display compression
information.
• Added support for Fedora 10, Ubuntu 8.10, and
OpenSuse 11.1 with GNU C and Fortran.
• Better support for Mac OSX (GNU compilers and
32-bit Intel).
• http://www.hdfgroup.org/release4/obtain.html
March 31, 2009
Annual HDF Breifing to ESDIS
17
HDF5-1.6.8
• Many improvements in tools to address ESDIS
requests
•
•
•
•
Availability of compression information in h5dump
Files comparison with h5diff
Support for strings in h5import
Support for user block in h5repack
• Added compatibility macros (e.g., H5Gcreate1) for
applications to work the same with 1.6.* and 1.8.*
• Support for object and region references in C++
• Bug fixes
•
http://www.hdfgroup.org/HDF5/release/obtain516.
html
March 31, 2009
Annual HDF Breifing to ESDIS
18
HDF5-1.8.1&2
• Tools improvements
• Support for external links
• Support for a simplified way to produce binary output by
h5dump (ESDIS)
• Better support for user block with h5repack
• Improved performance for free space tracking in a
file
• Fixed bugs with file mounting
• Many improvements in configuration, installation
and testing
• Better support for shared C, Fortran and C++
libraries
• http://www.hdfgroup.org/HDF5/release/obtain5.ht
ml
March 31, 2009
Annual HDF Breifing to ESDIS
19
HDF Java Products
• New released is based on JDK 1.6.x, HDF4r2.4 and
HDF5-1.8.2 with 1.6 compatibility
• Java products (i.e. HDFView) can read files created by
HDF5 1.8.2
• New HDFView:
• Supports access to remote HDF5 files stored at the iRODS
server (Windows and Linux only). For information on HDF5iRODS module, see
http://www.hdfgroup.org/projects/irods/index.html.
• Shows large dataset (e.g. 1,000,000 x 100) in a table viewer
• Converts JPEG, GIF, PNG, and BMP images to HDF5 or
HDF4
• Saves HDF4 or HDF5 images to JPEG, GIF, PNG, and BMP
images
• Writes variable length string data
• Filters fill values for images
March 31, 2009
Annual HDF Breifing to ESDIS
20
H4toH5 tools and library
• Based on HDF4r2.4 and HDF5-1.8.2
• h4toh5 utility
• Recognizes HDF-EOS2 files (--with-hdfeos2
configuration option)
• Can generate HDF5 files that can be read by
netCDF-4
• h4toh5 library
• Bug fixes
• Performance improvements
March 31, 2009
Annual HDF Breifing to ESDIS
21
QUALITY ASSURANCE
Some work of interest to ESDIS
March 31, 2009
Annual HDF Breifing to ESDIS
22
QA Efforts
• We test at least 250 different configurations for
HDF software (OSs x compilers x HDF
configurations) every day
• Regression testing of HDF5 released branches
(1.6. and 1.8) and development branch (1.9), and
HDF4
• Platforms
• Linux 32 and 64, Solaris 32 and 64, Free BSD 32 and 64,
AIX 32 and 64 (at LLNL), Windows 32 and 64, Linux
Clusters, SGI Altix
• Backward/forward compatibility testing
• Third part software testing
March 31, 2009
Annual HDF Breifing to ESDIS
23
Backward/forward compatibility
testing
• Goal: assure interoperability between 1.6 and
later versions of the HDF5 library used to
create AURA files
• Regression tests with and without strict format
checks using known problem files (currently MLS
Cloud series with corrupted headers)
• Download sample from subdirectories of OMI, MLS,
HIRDL, TOMS and TES files. Sample consists of
files from the current day of the year for available
years in two of the subdirectories listed for the
above instruments.
March 31, 2009
Annual HDF Breifing to ESDIS
24
Backward/forward compatibility
testing
• FTP sites (about 540 000 files)
• OMI Level 2 files
• ftp://aurapar2u.ecs.nasa.gov
• OMI Level2G, MLS and HIRDL files
• ftp://acdisc.gsfc.nasa.gov
• TOM files
• ftp://acdisc.sci.gsfc.nasa.gov
• TES files
• ftp://l4ftl01.larc.nasa.gov
March 31, 2009
Annual HDF Breifing to ESDIS
25
Backward/forward compatibility
testing
• Each known subdirectory will be visited ~5
times per year.
• hdf5 tools from versions 1.6.9, 1.8.3, and 1.9.0
are checked for proper operation with
downloaded files.
• Downloaded files are copied with h5copy and
repacked with h5repack. The resulting files
are checked again with all 3 versions of the
tools.
March 31, 2009
Annual HDF Breifing to ESDIS
26
Third party software testing
• NetCDF-4
• Daily tests of the netCDF4 snapshot
• Built against current HDF5 v1.8.*
• Tested on Linux 32/64 bit and solaris
• HDF-EOS2 and HDF-EOS5 testing
• HDFEOS2 with h4cc, 32/64 bit Linux, 32 bit solaris
• HDFEOS5 with hdf5 v1.6 and v1.9, 32/64 bit Linux,
32/64 bit solaris
• IDL wrappers
• Created IDL/HDF module binaries based on HDF5
1.8.2 and HDF4.2r3
March 31, 2009
Annual HDF Breifing to ESDIS
27
QUESTIONS?
March 31, 2009
Annual HDF Breifing to ESDIS
28
JAVA PRODUCT
HIGHLIGHTS
March 31, 2009
Annual HDF Briefing to ESDIS
29
HDF Java Products Highlights
• V2.5 released in February 2009
• 50 bug fixes/enhancements
• Support for HDF5 1.8-with-1.6 compatibility
(Does not support new HDF5 1.8 features)
March 31, 2009
Annual HDF Briefing to ESDIS
30
HDFView Highlights
•
•
•
•
Support for iRODS
Show large dataset in table
Write variable length string
Convert image to/from HDF4/5
•
•
•
•
JPEG
GIF
PNG
BMP
March 31, 2009
Annual HDF Briefing to ESDIS
31
Future work for Java
• 26 bugs/enhancements
• hdf-java 2.6 release
• Include latest HDF4 and HDF5 1.8 libraries
• Fix bugs and add new enhancements
• Full support for HDF5 1.8 (next slide)
March 31, 2009
Annual HDF Briefing to ESDIS
32
Future work for Java
• Full support for HDF5 1.8
• Add HDF5 1.8 APIs to the HDF5 Java wrapper
• Initial implementation (done)
• Review and revision is in progress
• Implement and test HDF5 JNI C layer
• Implementation (200+ APIs), 1 month fte
• Testing, 1 month fte
• Support new features in HDF5 1.8
• Revise hdf-java object package
• Add new features to HDFView
April 18-19, 2007
Annual HDF Briefing to ESDIS
33
Project Plan for 1.8 support
March 31, 2009
Annual HDF Briefing to ESDIS
34
OTHER TOOLS
ACTIVITIES
March 31, 2009
Annual HDF Briefing to ESDIS
35
35
Improvements in HDF4 tools
• Display compression information in hdf4 tools
• Preserve unlimited dimensions in hrepack
• Fix bugs in hdp and hdiff reported by DAACs.
March 31, 2009
Annual HDF Briefing to ESDIS
36
HDF5 tools highlights
• 35 bug fixes/enhancements to HDF5 tools
• All bug fixes/enhancements are applied to
both HDF5 1.6 and 1.8 release
March 31, 2009
Annual HDF Briefing to ESDIS
37
Highlights of HDF5 Tools
enhancements
• h5dump
•
•
•
•
Display compression information
Follow external links
Add ability to specify precision output
Use MEMORY option for default binary output
• h5diff
• Display the precision the differences
• Handle NaN values
• h5repack
• Keep the original user block information
• Multiple filters can be invoked
March 31, 2009
Annual HDF Briefing to ESDIS
38
Outreach
March 31, 2009
Annual HDF Briefing to ESDIS
39
Earth Science Related Outreach
• IEEE Transactions on Geosciences and Remote Sensing, vol. 47,
issue 1
• Paper: “Ensuring Long Term Access to Remotely Sensed Data with
Layout Maps” by Duerr, Cao, Folk, Lynnes, Yang.
• ESIP Federation Summer meeting, July 2008
• Presented a poster about OPeNDAP work
• Meeting with ITTVis, Oct. 2008
• Presentation about HDF
• Worked with them to test the IDL-HDF5 module with a new HDF5 release
• 7th NASA ESDSWG conference, Oct. 2008
• Reported on the OPeNDAP project, participated in several working
groups
• AGU Fall meeting, Dec. 2008
• Presentation: “Ensuring Long Term Access to Remotely Sensed HDF4
Data with Layout Maps,” by Folk and Duerr
• American Meteorological Society Annual Meeting, Jan. 2009
• Presentations on OPeNDAP and the HDF-EOS2 to netCDF-4 conversion
tool
March1-2,
31, 2008
2009
April
Annual HDF
Briefing
to ESDIS
Annual
HDF
Briefing to ESDIS
40
40
HDF-EOS Workshop XII
• Tutorials
•
•
•
•
•
•
Introduction to HDF5 Data and Programming Models
Advanced HDF5 Features
Migrating from HDF5 1.6 to 1.8
Introduction to HDF5 Tools
HDFView and HDF Java Products
What will be new in HDF5
• Status
• HDF Status and Developments
• HDF Group Support for NPP/NPOESS
• Applications and discussions
•
•
•
•
•
•
NOAA Science Data Stewardship project update
HDF5-OPeNDAP Project Update and Demo
Independent HDF4 mapping project update
HDF5 IRODS
Users: What are your needs
No programming language left behind
March 31, 2009
Annual HDF Briefing to ESDIS
41
Other outreach - sample
• Presentation “HDF5: Toward a Universal Platform for
Science and Engineering Data,” Folk
• National Renewable Energy Lab Scientific and Technical Data
Management and Analysis Workshop ’08, May 2008
• “HDF Experiences with I/O Bottlenecks,” Folk
• NSF/NARA Collaborative Expedition Workshop: Toward
Scalable Data Management – Overcoming I/O Bottlenecks,
June 10, 2008
• HDF5 Tutorial, Pourmal and Cheng
• 37th SPEEDUP Workshop on HPC, Zurich, Sept. 2008
• “HDF5: Past, Present, and Future”
“Tuning Parallel HDF5 for HPCApplications,”
Koziol, Aydt
• National Energy Research Scientific Computing Center HDF5
Workshop on Scalable Performance
March 31, 2009
Annual HDF Briefing to ESDIS
42
Related projects and
activities
March 31, 2009
Annual HDF Briefing to ESDIS
43
EOS SUPPORT
March 31, 2009
Annual HDF Briefing to ESDIS
44
EOS support
• HDF-EOS2 and HDF-EOS5
• Automatic configuration with szip enabled/disabled
• EOS2 and EOS5 are tested daily with HDF4
and HDF5 development code
• HDF-EOS website
• workshop coordination
• EOS product release
7/18/2015
March 31, 2009
The HDF Annual
Group for
2008
annual
HDF
Briefing
tobriefing
ESDIS
45 45
OPENDAP
March 31, 2009
Annual HDF Briefing to ESDIS
46
OPeNDAP – current status
• HDF5-OPeNDAP handler
• Served OMI and TES data
• HDF5 –awareness OPeNDAP client library
• HDF4-OPeNDAP handler
• Tested with AIRS data and some MODIS data
• More information in the afternoon session
7/18/2015
March 31, 2009
The HDF Annual
Group for
2008
annual
HDF
Briefing
tobriefing
ESDIS
47 47
NETCDF SUPPORT
March 31, 2009
Annual HDF Briefing to ESDIS
48
NetCDF4 support
• Help Unidata NetCDF4 developers
• Frequent communications with NetCDF4
developers
• Test NetCDF4 daily with the HDF5
development code
• NetCDF4 performance benchmark report
•
http://www.hdfgroup.org/pubs/papers/2008
-06_netcdf4_perf_report.pdf
7/18/2015
March 31, 2009
The HDF Annual
Group for
2008
annual
HDF
Briefing
tobriefing
ESDIS
49 49
LONG TERM ACCESS TO
REMOTELY SENSED
HDF4 DATA WITH
LAYOUT MAPS
Discussed later
March 31, 2009
Annual HDF Briefing to ESDIS
50
Other projects of Interest
March 31, 2009
Annual HDF Briefing to ESDIS
51
NOAA SDS
• Enhanced HDF4 to HDF5 conversion tool
• An HDF-EOS2 file can be converted to a netCDF-4 file
• Accumulated knowledge and experience to handle
HDF-EOS2 data
7/18/2015
March1-2,
31, 2008
2009
April
The HDF
Group
forto2008
annualtobriefing
Annual
HDF
Briefing
ESDIS
Annual
HDF
Briefing
ESDIS
52 52
52
Other projects
• “NPOESS HDF5 Development & Support”
• “BioHDF – HDF5 for next gen sequencing”
• Human Genome Research Institute SBIR with
Geospiza, Inc.
• “HDF support for urban military operations”
• Army Engineering Research and Development
Center – Topographic Engineering Center
March 31, 2009
Annual HDF Briefing to ESDIS
53
HDFView ERDC Concept Map Plugin
March 31, 2009
Annual HDF Briefing to ESDIS
54
Platform support issues
March 31, 2009
Annual HDF Briefing to ESDIS
55
HDF4 Platforms Supported
• Systems
• Compilers
• AIX 5.2 (32-bit, 64-bit)
• Free BSD 6.2 (32-bit, 64bit)
• IRIX 64 v6.5 (32-bit, 64-bit)
• Linux 2.4, 2.6 (RH3)
• Linux ia64
• Linux x86_64 (SUSE)
• Sun OS 5.8, 5.10 (32-bit,
64-bit)
• Windows XP, Vista
• Windows XP 64-bit*
• Mac OS X Intel
• * - new platform
March 31, 2009
• IBM C and Fortran
compilers
• GNU gcc 3.4.* and GNU
Fortran
• GNU gcc 3.4 and 4.*
• PGI v 7.* compilers
• Intel C and Fortran
versions 9.1 and 10.00
• SUN WorkShop C and
Fortran
• Visual Studio .NET and
2005 and Intel Fortran
• Visual Studio 2005 (no
fortran)
• GNU gcc 4.0.1 with
gfortran and g95
Annual HDF Briefing to ESDIS
56
HDF4 Platforms to drop/add
• Drop
• Add ?
• Windows XP with
MSVC++ 6.0
• Linux 2.4
• IRIX64 6.5
• SunOS 5.8, 5.9
• HP 11.23 (no access)
• Sun on Intel (no
access)
March 31, 2009
•
•
•
•
•
XT3
Blue Gene
Blue Water
Linux flavors?
?
Annual HDF Briefing to ESDIS
57
HDF5 Platforms Supported
• Systems
•
•
•
•
•
•
•
•
•
•
•
•
• Compilers
AIX 5.2 (32-bit, 64-bit)
Free BSD 6.2 (32-bit, 64-bit)
IRIX 64 v6.5 (32-bit, 64-bit)
Linux 2.4, 2.6 (RH3)
Linux ia64
Linux x86_64 (SUSE)
Sun OS 5.8, 5.10 (32-bit, 64bit)
Cray XT3
Windows XP, Vista
Windows XP 64-bit*
Mac OS X Intel
Alpha Open VMS*
• * - new platform
March 31, 2009
• IBM C and Fortran compilers
• GNU gcc 3.4.* and GNU
Fortran
• GNU gcc 3.4 and 4.*
• PGI v 7.* compilers
• Intel C and Fortran versions
9.1 and 10.00
• SUN WorkShop C and Fortran
• Visual Studio .NET and 2005
and Intel Fortran
• Visual Studio 2005 (no fortran)
• GNU gcc 4.0.1 with gfortran
and g95
• Compaq C and C++
compilers, HP Fortran
compiler
Annual HDF Briefing to ESDIS
58
HDF5 Platforms to drop/add
• Drop
• Add ?
• Windows XP with
MSVC++ 6.0
• Linux 2.4
• IRIX64 6.5
• SunOS 5.8, 5.9
• HP 11.23 (no access)
• Sun on Intel (no
access)
March 31, 2009
•
•
•
•
•
XT3
Blue Gene
Blue Water
Linux flavors?
?
Annual HDF Briefing to ESDIS
59
Platform support questions
• Platform to support for HDF Java products
•
•
•
•
•
•
•
•
Windows XP/Vista
Solaris
Linux
Mac Intel
Mac PowerPC
Linux x86_64
Linux ia64
Solaris 64-bit Sparc
• Any to add? Any to drop?
March 31, 2009
Annual HDF Briefing to ESDIS
60