Transcript The HDF Group HDF Project Update Mike Folk, Elena Pourmal, Kent Yang The HDF Group Briefing to ESDIS October 16-17, 2012 10/16/2012 HDF Briefing to NASA www.hdfgroup.org.

The HDF Group
HDF Project Update
Mike Folk, Elena Pourmal, Kent Yang
The HDF Group
Briefing to ESDIS
October 16-17, 2012
10/16/2012
HDF Briefing to NASA
1
www.hdfgroup.org
Project Goals
• Support for NASA HDF data providers and
data users, and the EOSDIS Core System
(ECS)
• Maintenance and quality assurance for HDF4
and HDF5 libraries and utilities
• Evolving the HDF4 and HDF5 libraries and
utilities to meet new NASA EOS requirement
• Integration with complementary technologies
and application domains
• Prototype study of web technologies to access
HDF data
10/16/2012
HDF Briefing to NASA
2
www.hdfgroup.org
The HDF Group
Support activities for NASA
HDF data providers and
users and ECS
10/16/2012
HDF Briefing to NASA
3
www.hdfgroup.org
Activities
• Help Desk
• Outreach
• Mailing lists, Web site
• Conferences, Workshops, Other
• EOS support
• netCDF support and collaboration
• Later today:
• HDF OPeNDAP support
• Support for NASA data products
• SMAP, ICESat-2, MEaSURES)
10/16/2012
HDF Briefing to NASA
4
www.hdfgroup.org
HELP DESK
10/16/2012
HDF Briefing to NASA
5
www.hdfgroup.org
The HDF Group Help Desk
• High-priority support for ESDIS users
•
•
•
•
Fast response
Problem investigation and resolution
Performance tuning
Access to HDF developers
• Basic support for entire HDF user community,
as resources permit
• [email protected]
10/16/2012
HDF Briefing to NASA
6
www.hdfgroup.org
Help Desk issues resolution time
HDF Issues Resolution Time
60
Number of issues
50
40
2010
2011
2012*
30
20
10
0
< 4 < 1 day < 2
hours
days
<1
<2
<1
<6
not
week weeks month month closed
* Number of messages up do August 31, 2012
10/16/2012
HDF Briefing to NASA
7
www.hdfgroup.org
Help Desk issues by years
HDF4 and HDF5 issues by years
1000
900
Number of issues
800
700
600
500
HDF5
HDF4
400
300
200
100
0
* Number of messages up do August 31, 2012
10/16/2012
HDF Briefing to NASA
8
www.hdfgroup.org
OUTREACH
10/16/2012
HDF Briefing to NASA
9
www.hdfgroup.org
Outreach
• Web pages improvements
• FAQ support for HDF4 and HDF5
http://www.hdfgroup.org/HDF5-FAQ.html
http://www.hdfgroup.org/HDF-FAQ.html
• Later today: Detailed talk on support for EOS
(Kent)
• Organizing material based on the ESDIS
survey results and feedback from the users
• Example: Tools by category (work in progress)
• How do I edit, copy, import/export data with
HDF?
10/16/2012
HDF Briefing to NASA
10
www.hdfgroup.org
Tools by category
10/16/2012
HDF Briefing to NASA
11
www.hdfgroup.org
Outreach
• Mailing list and archives
[email protected]
http://www.hdfgroup.org/news/
[email protected]
http://mail.hdfgroup.org/pipermail/hdfforum_hdfgroup.org/
10/16/2012
HDF Briefing to NASA
12
www.hdfgroup.org
[email protected]
• Started in Summer 2007
• Monitored by the members of the HDF group
• Number of messages:
2012 to Sep 30
2011
2010
703
1,334
1,721
• Most discussed topics:
• Parallel HDF5
• Advanced HDF5 features (drivers, complex
types, performance)
• .NET and C++ wrappers, HL libraries
10/16/2012
HDF Briefing to NASA
13
www.hdfgroup.org
[email protected]
• Members of the FORUM help with
• Releases testing on a regular basis
• Maintaining CMake build systems on platforms
beyond Windows
• Answering questions
• The Help Desk focuses on ESDIS and other
paying customers while referring others to the
FORUM for topics that require domain
knowledge or specific HDF5 usage
• Reporting defects
• Securing funding for The HDF Group
10/16/2012
HDF Briefing to NASA
14
www.hdfgroup.org
Conferences, Workshops, Other
• 2011 IDV/THREDDS Workshop, July 2011
• Learned how to use IDV and THREDDS
• 10th NASA ESDSWG Conference, Nov 2011
• Poster on HDF and netCDF interoperability
• 2012 ESIP Summer Meeting, July 2012
• Presentations
• HDF/HDF-EOS Data for Dummies
• What does Open Source mean for HDF?
• Poster
• What’s new at hdfgroup.org?
10/16/2012
HDF Briefing to NASA
15
www.hdfgroup.org
Conferences, Workshops, Other
• 15th HDF and HDF-EOS Workshop, April 2012
• Five tutorials, nine presentations and lightning
talks, and one special session
• Materials:
http://www.hdfeos.org/workshops/ws15/agenda.php
10/16/2012
HDF Briefing to NASA
16
www.hdfgroup.org
Conferences, Workshops, Other
• Advised ICESat-2, MOPITT and several
MEaSURES projects on interoperability with
netCDF-4
• Participated in the SMAP ISO-19115 work
• Participated in ESDSWG SPG telecons
• Joined several ESDSWG mailing lists related
to HDF
10/16/2012
HDF Briefing to NASA
17
www.hdfgroup.org
NETCDF SUPPORT AND
COLLABORATION
10/16/2012
HDF Briefing to NASA
18
www.hdfgroup.org
NetCDF-4 support
• Help Unidata netCDF-4 developers
• Frequent communications with netCDF-4
developers
• Provided a fix for ncdump critical to the ESDIS
users
• Test netCDF-4 daily with the HDF5
development code
• Plan to participate in Unidata Advanced
netCDF Workshop on October 26, 2012
10/16/2012
HDF Briefing to NASA
19
www.hdfgroup.org
EOS SUPPORT
10/16/2012
HDF Briefing to NASA
20
www.hdfgroup.org
EOS Support
• Submitted several bugs and the corresponding
fixes to the HDF-EOS team
• EOS2 and EOS5 are tested daily with HDF4
and HDF5 development code
• HDF-EOS website
• Actively maintain the website
• Improve the quality of the example codes and
plots to access NASA HDF and HDF-EOS
data via MATLAB, IDL and NCL
• More information in a separate talk
10/16/2012
HDF Briefing to NASA
21
www.hdfgroup.org
The HDF Group
Maintenance and quality
assurance for HDF4 and
HDF5 libraries and utilities
10/16/2012
HDF Briefing to NASA
22
www.hdfgroup.org
Activities
• Manage and resolve issues
• Perform maintenance releases of HDF
software
• Support OSs and compilers identified by ESDIS
• Address issues and software evolution
requests from ESDIS
• Perform SQE and SQA activities
• Put in place missing processes
• Improve existing processes
• Build in software quality by performing testing
and using the best software development
practices
10/16/2012
HDF Briefing to NASA
23
www.hdfgroup.org
HDF ISSUES
10/16/2012
HDF Briefing to NASA
24
www.hdfgroup.org
Issues and their priorities
• Must Fix
• Fix after “Must Fix”
• Data corruption
• Portability
• Backward and Forward
Compatibility
• Funded Requests
•
•
•
•
Power User Requests
Tools
Library issues
Build Infrastructure
• When resources
permit
• Wrappers
• HL Libraries
• Other
Need your input on priorities!
10/16/2012
HDF Briefing to NASA
25
www.hdfgroup.org
HDF4 opened and closed issues
HDF4 open and closed issues
70
Number of Issues
60
50
40
Opened
Closed
30
20
10
0
2003 2004 2005 2006 2007 2008 2009 2010 2011 2012
10/16/2012
HDF Briefing to NASA
26
www.hdfgroup.org
HDF5 opened and closed issues
HDF5 opened and closed issues
600
Number of issues
500
400
300
Opened
Closed
Closed High Priority
200
100
0
10/16/2012
HDF Briefing to NASA
27
www.hdfgroup.org
SQE AND SQA AT THE HDF
GROUP
10/16/2012
HDF Briefing to NASA
28
www.hdfgroup.org
Activities
• CMMI appraisal and process improvement
action plan (Dec. 2011)
• Created SQE position
• Focus on
• Implementing CMMI best practices and
procedures
• Enforcing current practices and procedures
• Focused on improving software testing
10/16/2012
HDF Briefing to NASA
29
www.hdfgroup.org
Activities
• Worked on establishing a culture of code
reviews for the HDF libraries and tools
• Improved releases planning and monitoring for
HDF5
• Used static analysis tools to identify defects
• Coverity (used at THG)
• Klocwork (used at LLNL)
10/16/2012
HDF Briefing to NASA
30
www.hdfgroup.org
HDF MAINTENANCE
RELEASES
10/16/2012
HDF Briefing to NASA
31
www.hdfgroup.org
General maintenance for 2011-2012
•
•
•
•
Implemented CMake for all HDF software
Worked on closing reported issues
Focused on performance enhancements
Improved HDF documentation
• http://www.hdfgroup.org/HDF5/doc/Advanced.html
• Explored the needs for internal documentation
• Preventive measures:
• Coverity static analysis; Rational tools; valgrind
• Have been exploring new testing tools to sustain
testing for HDF products (CDash, Bamboo, Cloud)
10/16/2012
HDF Briefing to NASA
32
www.hdfgroup.org
Maintenance Releases 2011 – 2012
2011
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
HDF4
HDF5
1.8.7
1.8.8
HDF-Java
2.8
h4h5 tools
2012
Jan Feb Mar Apr
HDF4
4.2.7
May
Jun Jul
patch1
HDF5
Aug Sep
Oct
Nov
CF
1.8.10
4.2.8
1.8.9
HDFJava
h4h5
tools
Dec
CF
2.9
2.2.1
CF – code freeze
6 weeks for HDF5
4 weeks for other products
10/16/2012
HDF Briefing to NASA
33
www.hdfgroup.org
HDF4 maintenance releases
• HDF 4.2.7 released in February 2012
• Extra functions to support H4 mapping project
• Support for Linux PPC64 with IBM XL Fortran
• Minor bug fixes and documentation improvements
• HDF 4.2.8 release in August 2012
• Support for H4mapping project
• Information about releases available from
http://www.hdfgroup.org/ftp/HDF/prev-releases/ReleaseFiles/
10/16/2012
HDF Briefing to NASA
34
www.hdfgroup.org
HDF5 maintenance releases
• HDF5 1.8.7-1.8.9
• Many improvements in tools
• ESDIS request to handle “/” in attributes names
• Fortran 2003 support
• Fortran wrappers for Dimension Scales APIs
• Information about releases available from
http://www.hdfgroup.org/ftp/HDF5/prev-releases/ReleaseFiles/
• HDF5 1.8.10 November 15, 2012
• Many improvements in h5diff, minor bug fixe
10/16/2012
HDF Briefing to NASA
35
www.hdfgroup.org
H4h5tools maintenance releases
• 2.2.1 released February, 15th, 2012
•
•
•
•
10/16/2012
Based on HDF 4.2.7 and HDF5 1.8.8
Fixed bugs reported by ESDIS users
Upgraded to HDF5 1.8 APIs
Updated documents
HDF Briefing to NASA
36
www.hdfgroup.org
Java maintenance releases
• 2.8 release December 2011
• Addressed all issues reported by ESDIS users
• 2.9 release December 2012
• Based on HDF 4.2.8 and HDF5 1.8.10
• ESDIS requests:
• Show groups/attributes in creation order
• Export data to a binary/ascii file without having
to open the object in the TableView
10/16/2012
HDF Briefing to NASA
37
www.hdfgroup.org
FUTURE DIRECTIONS
10/16/2012
HDF Briefing to NASA
38
www.hdfgroup.org
General maintenance for 2013-2014
• Document and improve maintenance processes
and procedures
• Focus on documentation for HDF4 and HDF5
• Improve quality
• Establish code review procedures for all HDF
products
• Expand static analysis to all HDF software
packages
• Enhance testing framework
10/16/2012
HDF Briefing to NASA
39
www.hdfgroup.org
Future Maintenance Releases
2013
Jan Feb Mar
HDF4
CF
Apr
May
CF
1.8.11
Jun Jul
Aug Sep
Oct
Nov
CF
1.8.12
4.2.9
HDF5
HDFJava
CF
h4h5
tools
CF
2.10
2.2.2
2014
Jan Feb Mar Apr
HDF4
CF
May
Jun Jul
Aug Sep
Oct
Nov
CF
1.8.14
Dec
4.2.10
HDF5
CF
1.8.13
HDFJava
h4h5
tools
Dec
CF
CF
2.11
2.2.3
CF – code freeze
6 weeks for HDF5
4 weeks for other products
10/16/2012
HDF Briefing to NASA
40
www.hdfgroup.org
Future HDF4 maintenance releases
• HDF 4.2.9 planned for February 2013
• Close as many issues as possible
• HDF 4.2.10 planned for February 2014
• Complete “self-configuration” work for library
maintainability and portability
• ESDIS priorities?
10/16/2012
HDF Briefing to NASA
41
www.hdfgroup.org
Future HDF5 maintenance releases
• HDF5 1.8.11 May 2013
• Features sponsored by European Synchrotron
Community (PSI/DECTRIS and DESY)
• Writing compressed chunks directly to HDF5 file
bypassing library internals
• Finished prototype; achieved order of magnitude boost in
I/O
• Support for dynamically loaded third party
libraries (compression and other filters)
• Preliminary studies soon; several prototypes are already
available
10/16/2012
HDF Briefing to NASA
42
www.hdfgroup.org
Future HDF5 maintenance releases
• How soon will MATLAB and IDL be able to
support 1.8.11?
• ESDIS priorities?
10/16/2012
HDF Briefing to NASA
43
www.hdfgroup.org
Future h4h5tools maintenance releases
• 2.2.2 release March 2013
• Use latest HDF4 and HDF5 releases
• Address any ESDIS issues if arise
• 2.2.3 release March 2014
• Possible improvements
• Conversion of dimension scales
• Leveraging HDF5 1.8 features
• ESDIS priorities?
10/16/2012
HDF Briefing to NASA
44
www.hdfgroup.org
Future Java maintenance releases
• 2.10 release December 2013
• We are gathering requirements (next slides)
10/16/2012
HDF Briefing to NASA
45
www.hdfgroup.org
TOOLS UPDATE
10/16/2012
HDF Briefing to NASA
46
www.hdfgroup.org
HDFVIEW USER SURVEY
10/16/2012
HDF Briefing to NASA
47
www.hdfgroup.org
Background
• HDFView was created for browsing and editing
HDF files on a local machine
• The most popular HDF tool
• August 2012 downloads
• HDFView – 2,805
• HDF5 binaries for all platforms – 1,073
• Twelve releases since the first release in 1998
• Time to reevaluate its capabilities and find out
users’ expectations
10/16/2012
HDF Briefing to NASA
48
www.hdfgroup.org
HDFView Survey Goals
• Understand users needs and future directions
for of HDFView maintenance and evolution
• New requirements
• Current deficiencies
• Most desired improvements
• Help with tasks prioritization
10/16/2012
HDF Briefing to NASA
49
www.hdfgroup.org
HDFView Survey
• Conducted a survey at
https://www.surveymonkey.com/s/PLGRLJY
• Over 40 questions in 8 different areas
• About 200 responses
10/16/2012
HDF Briefing to NASA
50
www.hdfgroup.org
Requested Improvements Areas
• Import/export capabilities
• Binary, text, Excel, tree-shape (XML, JSON),
GeoTIFF, TIFF, KML (Google Earth), PNG,
netCDF
• Access
• Cloud, OPeNDAP, mobile devices
• Data discovery/search
• Values, metadata, file structure
• Data presentation
• Plots, images, data with dimension scales
10/16/2012
HDF Briefing to NASA
51
www.hdfgroup.org
Requested Improvements Areas
• Usability
• HDF5 functionality
• Performance with large files
• Support for all HDF5 datatypes
• Support for custom filters
• Data handling
• Displaying, copying, selecting, exporting, importing
• GUI features
• One working window
• Java vs. new technologies (QT, C++, ?)
• UNIX remote display
• Platform-specific requests
• Mac OS X (installation, shortcuts)
10/16/2012
HDF Briefing to NASA
52
www.hdfgroup.org
HDFView Survey
• Report with raw data is available from
http://www.hdfgroup.org/hdf-java-html/hdfview/
10/16/2012
HDF Briefing to NASA
53
www.hdfgroup.org
HDF TOOLS PLAN
10/16/2012
HDF Briefing to NASA
54
www.hdfgroup.org
Plans for HDFView
• Plan for and address high-priority items for
ESDIS from the survey
• Quality improvements
• Clean issues database (37 right now)
• Finish unit testing work (HDF4 JNI)
• Set up GUI automatic testing
• Performance improvements
• Address large files issues
• Evolution
• Add support for dimension scales
• Add support for misc. HDF5 features
10/16/2012
HDF Briefing to NASA
55
www.hdfgroup.org
HDFView Survey
• The ESDIS 2013 priority list as identified by
the ESDSWG Executive Council available from
http://earthdata.nasa.gov/wiki/main/index.php/The_ESDIS_2013_Priority_List
• Topics relevant to HDFView
• ESDIS standards: GeoTIFF and KML
• Cloud and mobile computing
• ESDS conventions for HDF5
• CF conventions
• netCDF interoperability
• What are ESDIS priorities for HDFView?
10/16/2012
HDF Briefing to NASA
56
www.hdfgroup.org
Plans for command-line tools
• Regular maintenance
• Clean up issues database
• Focus on quality and performance
• Focus on tools to close a gap between HDF5 and
XML
• Settle on new XML schema
• H5gen to generate HDF5 from XML
• Enhance XML output generated by h5dump
• Possible open source experiment?
• Small Apache project to learn to create tools that are
developed and maintained primarily by the open
source community.
• ESDIS priorities?
10/16/2012
HDF Briefing to NASA
57
www.hdfgroup.org
Open Source Initiative Action Plan
Mission Statement
Stakeholders
• Conduct a small Apache project that will
enable us to learn to create tools for earth
science data users that is developed and
maintained primarily by the open source
community.
Approach
Outcomes, Deliverables, Milestones
• Achieve approval from NASA to make this
project a part of the ESDIS project (EED
Task 2 Rev 8).
• Identify a need that is likely to have
community support. For example, a need
for tools to convert HDF files to certain other
formats.
• Learn to create and lead an Apache
Incubator Project.
• Create an Apache Incubator project.
• Develop tools.
• Graduate the project from incubation.
10/16/2012
• Key groups, organizations, individuals that will be
impacted
• Earth science data users.
• EOS projects, Decadal Survey Missions, other
NASA Earth Science data activities
• All HDF communities
• Who needs to be involved in developing deliverables.
• Developers and end users who work with NASA
earth science data.
• Specific outcome: the company learns to create an
open source project in which HDF software is
developed and used by the community
• Work products
• An Apache Incubator project
• An Apache project supporting the software
• Tools
• Recommendation report
• Milestones
• Periodic reports on steps in the approach.
• Incubator project established.
• Apache project achieved.
• Final report
HDF Briefing to NASA
58
www.hdfgroup.org
PLATFORM SUPPORT ISSUES
10/16/2012
HDF Briefing to NASA
59
www.hdfgroup.org
HDF4 platforms supported
• Systems
• Compilers
• Linux 2.6 (CentOS-5)
Linux 2.6 x86_64
• Linux PowerPC64
• Solaris 5.10
• 32/64bit Windows XP,
7 and Cygwin
• Mac OS Intel 10.6.*
and 10.7.* 32/64-bit
10/16/2012
HDF Briefing to NASA
• IBM C and Fortran cmp
• GNU C and Fortran up
to 4.6.2, g95 (4.0.3)
• Intel C and Fortran v12
• PGI C and Fortran v11
• Sun C 5.10 and Fortran
95 8.4
• VS 2008 Intel Fortran
10
• VS 2010 Intel Fortran
10
60
www.hdfgroup.org
HDF5 platforms supported
• Systems
• Compilers
• Same as for HDF4
• AIX 5.3
10/16/2012
HDF Briefing to NASA
• Same as for HDF4
• IBM compilers
61
www.hdfgroup.org
HDF4 and HDF5 platforms to drop
• Systems
• Compilers
• Mac 10.6.*
• Windows XP
• Last releases HDF 4.2.9
and HDF5 1.8.10
10/16/2012
HDF Briefing to NASA
• VS2008 with Intel 11
and MSVS build
environment
• We will use “CMake
only” for Windows for all
HDF products starting in
Spring 2013
62
www.hdfgroup.org
HDF4 and HDF5 platforms to add
• Systems
• Compilers
• Mac 10.8.*
• How important is 32-bit
on 64-bit Mac OS X?
• Windows 8
• Clang?
• VS 2012
• Spring 2013
• Spring 2013
• Flavors of Linux?
We are using virtualization very successfully.
Can add any Linux or Windows flavors.
Just let us know!
10/16/2012
HDF Briefing to NASA
63
www.hdfgroup.org
The HDF Group
Integration with
complementary
technologies and
application domains
10/16/2012
HDF Briefing to NASA
64
www.hdfgroup.org
OPENDAP
10/16/2012
HDF Briefing to NASA
65
www.hdfgroup.org
OPeNDAP – current status
• HDF5-OPeNDAP handler
• Re-engineered to support more NASA HDF5
and HDF-EOS5 products
• HDF4-OPeNDAP handler
• Enhanced the support of many NASA HDFEOS2 and HDF4 products
• More information in a separate talk
10/16/2012
HDF Briefing to NASA
66
www.hdfgroup.org
TOOLS TO IMPROVE
INTEROPERABILITY
BETWEEN HDF AND NETCDF
10/16/2012
HDF Briefing to NASA
67
www.hdfgroup.org
HDF and netCDF interoperabilty tools
•
•
•
•
•
HDF-EOS5 augmentation tool
HDF-EOS2 dumper tool
HDF-EOS5 to netCDF-4 conversion tool
HDF4/HDF-EOS2 to CF conversion library
More information in a separate talk
10/16/2012
HDF Briefing to NASA
68
www.hdfgroup.org
The HDF Group
HDF4 Layout Maps
10/16/2012
HDF Briefing to NASA
69
www.hdfgroup.org
HDF4 Layouts Map
Provide tools and support to ensure long-term
access to EOS data via HDF4 File Content
Maps.
• H4mapwriter release 1.0.3 in June 2012
• More information in a separate talk
10/16/2012
HDF Briefing to NASA
70
www.hdfgroup.org
The HDF Group
Thank You!
10/16/2012
HDF Briefing to NASA
71
www.hdfgroup.org
Acknowledgements
This work was supported by Subcontract number
114820 under Raytheon Contract number
NNG10HP02C, funded by the National Aeronautics
and Space Administration (NASA). Any opinions,
findings, conclusions, or recommendations
expressed in this material are those of the authors
and do not necessarily reflect the views of Raytheon
or the National Aeronautics and Space
Administration.
10/16/2012
HDF Briefing to NASA
72
www.hdfgroup.org