Transcript The HDF Group HDF Update Mike Folk The HDF Group The 13th HDF and HDF-EOS Workshop November 3-5, 2009 November 3-5, 2009 HDF/HDF-EOS Workshop XIII www.hdfgroup.org.

The HDF Group
HDF Update
Mike Folk
The HDF Group
The 13th HDF and HDF-EOS Workshop
November 3-5, 2009
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
1
www.hdfgroup.org
Topics
What's up with The HDF Group?
Library Update
Tools update
HDF Java Products
Library development in the works
Other activities
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
2
www.hdfgroup.org
The HDF Group
What’s up with The HDF
Group?
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
3
www.hdfgroup.org
The HDF Group
What is
The HDF Group
And why does it exist?
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
4
www.hdfgroup.org
The HDF Group
• Established in 1988
• 18 years at University of Illinois National
Center for Supercomputing Applications
• 4 years an independent non-profit company
“The HDF Group”
• The HDF Group owns HDF4 and HDF5
• Basic HDF4 and HDF5 formats, libraries and
tools are open and free
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
5
www.hdfgroup.org
Data challenges addressed by HDF
• Our ability to organize complex collections of
data
• Efficient and scalable data storage and access
• A growing need to integrate a wide variety of
types of data
• Long term preservation of data
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
6
www.hdfgroup.org
The HDF Group
The HDF Group Mission
To ensure long-term
accessibility of HDF data
through sustainable
development and support of
HDF technologies.
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
7
www.hdfgroup.org
Goals
• Maintain and evolve HDF for sponsors and
communities that depend on it
• Provide support to the HDF communities
through consulting, training, tuning,
development, research
• Sustain The HDF Group for the long term to
assure data access over time
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
8
www.hdfgroup.org
The HDF Group Services
• Helpdesk and Mailing Lists
• Available to all users as a first level of support
• Standard Support
• Rapid issue resolution and advice
• Consulting
• Needs assessment, troubleshooting, design reviews, etc.
• Training
• Tutorials and hands-on practical experience
• Enterprise Support
• Supporting many HDF activities across organizations
• Special Projects
• Adapting customer applications to HDF
• New features and tools
• Research and Development
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
9
www.hdfgroup.org
Members of the HDF support community
•
•
•
•
•
•
•
•
•
NASA – EOS
NOAA/NASA/Riverside Tech – NPOESS
Army Geospatial Center
A leading U.S. aerospace company
NIH/Geospiza (bio software company )
University of Illinois/NCSA
Sandia National Laboratory (2)
Lawrence Berkeley National Lab
Projects for petroleum industry, vehicle testing,
weapons research, others
• “In kind” support
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
10
www.hdfgroup.org
Some areas of increased recent interest
• Improvements
•
•
•
•
Concurrent access
Parallel I/O performance
Real-time write performance
High level language support
• Life sciences
• Sequencing
• Biomedical imaging
• Database integration
• Microsoft products (HPC, .NET, others)
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
11
www.hdfgroup.org
Cool recent application
Imageworks’ Field3D
Spiderman 3
November 3-5, 2009
The Polar Express
HDF/HDF-EOS Workshop XIII
12
www.hdfgroup.org
Topics
What's up with The HDF Group?
Library Update
Tools Update
HDF Java Products
Library development in the works
Other activities
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
13
www.hdfgroup.org
The HDF Group
Basic Library Releases
HDF4
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
14
www.hdfgroup.org
Time-line of the HDF libraries releases
HDF5 1.8
1.8.3
(May 09)
1.8.4
(Nov 09)
1.8.5
(May 10)
HDF5 1.6
1.6.9
(May 09)
1.6.10
(Nov 09)
Dropped
HDF4
4.2r4
(Feb 09)
4.2r5
(Feb 10)
H4-H5
Conversion
Software
2.1
(Feb 09)
2.2
(Feb 10)
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
15
www.hdfgroup.org
HDF5 1.8.3 minor release (May 09)
• New functions
• Improve flexibility when traversing external links
• Validate object identifier
• Enabled data chunk cache properties to be set
per dataset (per file in previous releases)
• Forward/backward compatibility issues
• Modified library to be able to open files with
corrupt root group symbol table messages
• Also corrects corruption errors if found.
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
16
www.hdfgroup.org
HDF5 1.8.4 minor release (Nov 09)
• Modified configure and make process to
properly preserve user's CFLAGS and similar
environment variables.
• Corrected a problem where library would rewrite the superblock in a file opened for R/W
access, even when no changes were made
to the file.
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
17
www.hdfgroup.org
HDF5 1.6 minor releases
• 1.6.9 May 09
• Minor bug fixes
• Same tools improvements as in 1.8.3
• 1.6.10 Nov 09
• Minor bug fixes
• Ability to embed library information in executable
binaries
• This is a last release of 1.6 series
• announced in May 2009 – no response
• This is your last chance!
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
18
www.hdfgroup.org
HDF 4r2.4 minor release (Feb 09)
•
•
•
•
•
Minor bug fixing, enhancements
New routines to get size of compressed data
Support for C shared libraries
Support for 32-bit version on Mac Intel
Updated docs in HTML and PDF
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
19
www.hdfgroup.org
HDF 4r2.5 minor release (Feb 10)
• Minor bug fixes, enhancements
• Support for 64-bit version on Mac Intel
• Restructured and cleaned up source code for
easier maintenance
• Changes in versioning
• Improves ability to maintain
• Becomes similar HDF5 versioning works
• Will use major, minor, release and sub-release
suffix in the names of the source tar balls
• E.g., hdf-4.2.5, hdf-4.2.5-snap0
• Library string will include suffix
• E.g., "HDF Version 4.2 Release 4-snap3, October 18,
2009"
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
20
www.hdfgroup.org
H4-H5 Conversion Software 2.1 (Feb 09)
• Based on HDF4r2.4 and HDF5-1.8.2
• h4toh5 utility
• Recognizes HDF-EOS2 files (--with-hdfeos2
configuration option)
• Can generate HDF5 files that can be read by
netCDF-4
• h4toh5 library
• Bug fixes
• Performance improvements
• http://hdfgroup.org/h4toh5/
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
21
www.hdfgroup.org
H4-H5 Conversion Software 2.2 (Feb 10)
• Based on HDF4r2.5 and HDF5-1.8.4
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
22
www.hdfgroup.org
Topics
What's up with The HDF Group?
Library Update
Tools update
HDF Java Products
Library development in the works
Other activities
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
23
www.hdfgroup.org
Major Improvements for Existing Tools
• H5dump additions
• Ability to show data pointed to by dataset region references.
• More options for dumping data into ASCII
• Compatible with MS Excel
• Compatible with h5import
• h5diff
• Improvements in accuracy, flexibility, and performance
• Some new flags
• Report non-comparable objects
• Avoid NaN detection
• Option to use system epsilon to compare floating-point numbers
• Compares for strict equality first to improve performance
• Treats two INFINITY values as equal
• Fixed segmentation fault problem on variable length strings.
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
24
www.hdfgroup.org
Major Improvements for Existing Tools
• h5stat
• Fixed incorrect statistics on EOS big data files
with corrupted headers.
• h5repack
• Added ability to preserve group creation order
• When chunk size not specified, uses
heuristics to set chunk size
• Fixed problem that 1.8 fails on a file created
with 1.6.
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
25
www.hdfgroup.org
Tool activities in the works
• New tool -- h5tail
• Display new records appended to a dataset
• Improved code quality and testing
• Tools library: general purpose APIs for tools
• Tools library currently only for our developers
• Want to make it public so that people can use it in
their products
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
26
www.hdfgroup.org
Conversion Tools
Please send us your comments and requests
regarding HDF5 conversion tools, such as
•
•
•
•
HDF4 to HDF5
HDF5 to jpeg
HDF5 to XML
HDF5 to other formats?
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
27
www.hdfgroup.org
Topics
What's up with The HDF Group?
Library Update
Tools update
HDF Java Products
Library development in the works
Other activities
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
28
www.hdfgroup.org
HDF-Java 2.6 is on the way
• Includes all HDF java products
• Java Wrapper API
• Java Object API
• HDFView
• Adds new features, such as better support for
dataset region references
• Improves performance
• Release schedule
• Beta 1: end of Nov. 09
• Full release: end of Dec. 09
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
29
www.hdfgroup.org
Full support of HDF5 1.8.x in hdf-java
• Full HDF5 1.8 support will be added to the
release after version 2.6.
• We are looking for input
• RFC:
http://www.hdfgroup.uiuc.edu/RFC/HDF5/hdf-java/
• Java wrapper will be completed March 2010
• Object API and HDFView update to come later
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
30
www.hdfgroup.org
Topics
What's up with The HDF Group?
Library Update
Tools update
HDF Java Products
Library development in the works
Other activities
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
31
www.hdfgroup.org
Single-Writer/Multiple-Reader Access
• Situation: A long-running process is modifying
an HDF5 file and simultaneously other
processes want to inspect data in the file.
• Solution: Single-Writer/Multiple-Reader
(SWMR) File Access.
• Allows simultaneous reading of HDF5 file while
the file is being modified by another process
• No inter-process coordination necessary
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
32
www.hdfgroup.org
Improved Multi-Threaded Concurrency
• Converting from “big lock” on code (entire
library) to locks on internal library data
structures
• Will improve ability to have multiple threads
performing HDF5 operations simultaneously
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
33
www.hdfgroup.org
Other Library Features
• Saving space
• Store Partial Edge Chunks More Efficiently
• Persistent File Free Space tracking/recovery
• Allow a group’s link info to be compressed
• Saving time
• Aggregate neighboring metadata for faster
metadata cache I/O
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
34
www.hdfgroup.org
New chunk indexing methods
Dataset type
Index type
Space
improvements
Speed
improvements
no unlimited
dimensions,
no filters,
no missing
chunks
“implicit”
no actual
chunk index
Same storage
space as
contiguous dataset
storage (no index)
Constant time
lookups
Faster parallel I/O
no unlimited
dimensions
“fixed sized”
smaller chunk
index
Smaller index
overhead
Constant time
lookups
1 unlimited
dimension
“extensible
array”
Smaller index
overhead
Constant time
lookups and
appends
2+ unlimited
dimension
Improved
B-tree*
Smaller index
overhead
Faster
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
35
www.hdfgroup.org
Parallel I/O Improvements
• Project with Lawrence Berkeley Nat’l Lab to
improve HDF5 performance on parallel
applications
• Up to 6x performance improvements on certain
applications (so far)
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
36
www.hdfgroup.org
Topics
What's up with The HDF Group?
Library Update
Tools update
HDF Java Products
Library development in the works
Other activities
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
37
www.hdfgroup.org
The HDF Group
HDF-EOS library
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
38
www.hdfgroup.org
EOS support
• HDF-EOS2 and HDF-EOS5
• Automatic configuration with szip
enabled/disabled
• Now tested daily with HDF4 and HDF5
development code
• Updated the HDF-EOS website
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
39
www.hdfgroup.org
The HDF Group
HDF-EOS5/netCDF-4
Augmentation Tool
Accessing HDF-EOS5 files via netCDF-4 API
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
40
www.hdfgroup.org
The Main Challenge
• Would like netCDF-4 applications to be able
to read and understand HDF-EOS 5 files
• Problem: NetCDF-4 model follows the HDF5
dimension scale model but HDF-EOS5 does
not.
HDFEOS
GRIDS
No HDF5 dimension
CloudFractionAndPressure
scales are associated
Data Fields
with this variable
CloudFraction
CloudPressure
November 3-5, 2009
41
HDF/HD
F-EOS
Worksh
op XIII
www.hdfgroup.org
Our Solution – Augmentation
• Provide dimensions required by netCDF-4
HDFEOS
GRIDS
CloudFractionAndPressure
Data Fields
CloudFraction[XDim][YDim]
CloudPressure[XDim][YDim]
XDim
YDim
November 3-5, 2009
42
HDF/HD
F-EOS
Worksh
op XIII
www.hdfgroup.org
Special values in HDF5
• There are cases where a user may wish to
specify more than one “special” value to
describe non-standard data.
• We provide several examples (C, Fortran, IDL)
on how to store special values
• http://www.hdfgroup.org/pubs/rfcs/
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
43
www.hdfgroup.org
The HDF Group
OPeNDAP
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
44
www.hdfgroup.org
OPeNDAP
• HDF5-OPeNDAP handler
• Served OMI Swath data
• HDF4-OPeNDAP handler
• Tested with some AIRS data and some MODIS
data
• More information in the Thursday morning
session
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
45
www.hdfgroup.org
Swath to Grid conversion Tool
•
•
•
•
Request from NASA GES DISC
Convert Swath to Grid
Support both HDF-EOS2 and TRMM data
Still in the development
MODIS Swath
Converted Grid
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
46
www.hdfgroup.org
The HDF Group
Support for NPP/NPOESS
by
The HDF Group
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
47
www.hdfgroup.org
Priorities for 2008-2009
• Data accessibility and usability
• Developed library of high level APIs to
support NPP/NPOESS data management
• Modified h5dump to display region references
• Modified HDFView to view object and region
references and quality flags
• System maintenance
• User support
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
48
www.hdfgroup.org
NPOESS Project Information
• Project Web site
• http://www.hdfgroup.org/projects/npoess/
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
49
www.hdfgroup.org
HDF4 LAYOUT MAPS
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
50
www.hdfgroup.org
HDF4 Layout Map Project
• Problem
• Long-term readability of HDF data depends
on long-term availability of software
• Proposed solution
• Create a map of the layout of data objects in
an HDF file, allowing a simple reader to be
written to access the data
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
51
www.hdfgroup.org
A Project with the Army Geospatial Center
TRANSFORMING THE
GEOCOMPUTATIONAL BATTLESPACE
FRAMEWORK WITH HDF5
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
52
www.hdfgroup.org
Data Challenges
Military Decision Making
Wide variety
Satellite
Buckeye
November 3-5, 2009
Large scale
Culture
High res.
Stream
HDF/HDF-EOS Workshop XIII
High efficiency
Accuracy
53
Time
www.hdfgroup.org
NIH STTR with Geospiza, Seattle WA
BIOHDF : TOWARD SCALABLE
BIOINFORMATICS
INFRASTRUCTURES
TM
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
54
www.hdfgroup.org
Next Generation DNA Sequencing
NGS is Powerful
“Transforms today’s biology”
“Democratizing genomics”
“Genome center in a mail room”
“Changing the landscape”
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
55
www.hdfgroup.org
… And Daunting
“Prepare for the deluge”
“Byte-ing off more than you can chew”
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
56
www.hdfgroup.org
BioHDF Project
• Goal: Move bioinformatics problems from organizing
and structuring data to asking questions and
visualizing data
• Develop data models and tools to work with NGS data in HDF5
• Create HDF5 domain-specific extensions and library modules to
support the unique aspects of NGS data  BioHDF
• Integrate BioHDF technologies into Geospiza products
• Deliver core BioHDF technologies to the community
as open-source software
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
57
www.hdfgroup.org
The HDF Group
Thank You All
and
Thank You NASA!
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
58
www.hdfgroup.org
Acknowledgements
• This report is based on work supported by
cooperative agreement number NNX08AO77A
from the National Aeronautics and Space
Administration (NASA).
• Any opinions, findings, conclusions, or
recommendations expressed in this material
are those of the author[s] and do not
necessarily reflect the views of the National
Aeronautics and Space Administration.
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
59
www.hdfgroup.org
The HDF Group
Questions/comments?
November 3-5, 2009
HDF/HDF-EOS Workshop XIII
60
www.hdfgroup.org