Transcript HDF-EOS Status, Related Tools and Issues Abe Taaheri, Raytheon IIS 2009 THG Briefing to ESDIS Page 1

HDF-EOS Status,
Related Tools and Issues
Abe Taaheri, Raytheon IIS
2009 THG Briefing to ESDIS
Page 1
TOOLKIT / HDF-EOS Support
•
Maintenance support is funded on EMD Evolution
through May 22, 2009
- Bridge contract until EODDIS Evolution &
Development (EED) award in 2010
•
Current support






HDF-EOS 2 & 5
SDP/MTD Toolkit
HDF-EOS plug-in for HDFView
HDF-EOS2 to GeoTIFF Converter (HEG)
Other Toolkit/HDF-EOS related software
Toolkit/HDF-EOS/HEG helpdesk
Page 2
ECS Support of HDF-EOS
• Both HDF4 and HDF5 based flavors of HDF-EOS are part
of ECS baseline.
• OS’s: Solaris (9, 10), Irix6.5 (64, n32),
Linux (including 64-bit Opteron and Itanium),
Mac OS X (Power PC and Mac Intel),
Windows XP(MTD Toolkit, HDF-EOS2 & 5, HEG, HDFView)
• added full Cygwin support for next release
• Compilers: Fortran 77/90 & g77/pgf90 , C, C++, gcc, g++,
Intel Fortran (SGI Altix), gfortran (Mac Intel)
Page 3
Downloadable Software
• SDP/MTD Toolkit (Science Data Processing Tools)
• HDF-EOS2 (HDF4 based)
• HDF-EOS5 (HDF5 based)
• heconvert (converts HDF4 - based
Grid/Point/Swath structures to HDF-EOS 5
equivalents)
Page 4
Downloadable Applications
• HEG
- GeoTIFF conversion
- Spatial subsetting, Field Subsetting
- reprojection
- stitching, sub-sampling and more ….
• DumpMetadata (Dump HDF-EOS metadata)
• HDFView Plug-in, Java-based browser
(HDF4, HDF5, HDF-EOS 2 and 5 access)
• HE5View* (HDF5 - based browser)
• EOSView* (HDF4 - based browser)
* Dropped support for OS updates on July 2006
Page 5
Software Releases
Toolkit/HDF-EOS:
• Last release – March 2008
– TOOLKIT 5.2.15
– HDF-EOS 2.15, HDF- EOS5.1.11
with HDF4.2r3, HDF5-1.6.7, Szip2.1
• Next release – May 2009
– TOOLKIT 5.2.16
– HDF-EOS 2.16, HDF-EOS5.1.12
with new releases of HDF4 (2r4), 5 (1.8.3)
Page 6
Software Releases (Cont.)
HEG (HDF-EOS2 to GeoTIFF Converter):
• Last release – August 2008
– Version 2.9
• Next release – May 2009
– Version 2.10
– Already have released Datapool version to
the DAACs (Feb. 2009)
Page 7
Software Releases (Cont.)
HDF-EOS Plug-in for HDFView:
• Last release – September 2008
– Version 2.4
• Next release – June 2009
– For HDFView Version 2.5
Page 8
Software Releases (Cont.)
HECONVERT (HDF-EOS2 to HDF-EOS5
• Last releases
Converter):
– 2001, 2004 Version ?
– Oct. 2008 Version 2.0
DumpMetadata (ASCII Dump of HDF-EOS metadata)
• Last release
–Dec. 2008 Version 1.0
Page 9
SDP Toolkit
•
Science Data Production (SDP) Toolkit is a collection of
tools used by
- Science software developers who produce
code to process instrument data.
- HDF-EOS users who need to
• Access metadata in HDF-EOS files
• Perform Time/Date conversion
• Access Digital Elevation Model Data
• Access Ancillary Data, such as Digital Chart
of the World database (DCW), Olson World
Ecosystem files, DEM data files, etc.
Page 10
SDP Toolkit (cont.)
•
Tools provided in (SDP, MTD) Toolkit:
-
AA
CBP
CSC
CUC
DEM
EPH
GCT
IO
MEM
MET
PC
SMF
TD
(Ancillary Data Access)
(CELESTIAL Body Position)
(Coordinate System Conversion)
(Constant and Unit Conversions)
(Digital Elevation Model Access)
(Ephemeris Data Access)
(Geo Coordinate Transformation)
(Input Output (File I/O))
(Memory Management)
(Metadata Access)
(Process Control)
(Status Message File (Error/Status))
(Time and Date Conversion)
Page 11
What is HDF-EOS ?
• HDF-EOS is an extension to HDF which
standardizes storage of and access to common
Earth Sciences data structures
–By providing a single interface to data structures
common to earth science
–By providing a container for EOS inventory,
archive and product specific metadata
–By establishing a consistent relationship
between geolocation and science data.
Page 12
HDF-EOS 2
• HDF4 - based, storage format for EOS standard
products.
• Used operationally by MODIS, MISR, ASTER,
Landsat, AIRS and other EOS instruments
• Support for Grid/Point/Swath structures
• HDF4 and HDF-EOS 2 will be supported by
THG/NASA for the fore-seeable future.
Page 13
HDF-EOS 5
• Based on HDF5, a complete rewrite of HDF-EOS2
with a different user interface.
–First released in 2000.
–Format used by EOS Aura instruments
• Designed to ‘resemble’ HDF-EOS 2 to the
maximum extent possible.
–Supports same data structures
–Added prefix ‘HE5_’ to HDF-EOS 2 functions.
Page 14
HDFView
•
HDFView: A Java-based browser from THG
for HDF4 and HDF5 – based data.
•
HDF-EOS plug-in extends HDFView to browse
HDF-EOS data in both versions.
•
The extension is a plug-in module and does
not change THG HDFView functionality. All
copy, save, editing, image display, etc. for
HDF4 and 5 objects remains.
Page 15
HDFView
•
Identical look and feel when displaying HDF
and HDF-EOS objects.
•
Users can access a complete breakdown of
HDF-EOS objects. e.g. grid information,
projection info., dimensions, datafields, and
attributes are displayed.
•
Current Support: SUN, SGI, LINUX, Mac,
WINDOWS platforms
Page 16
Swath, Grid, and Point Objects
Page 17
AMSR-E Swath
Tableview Features in Version 2.4
Page 18
HDF-EOS to GeoTIFF Converter (HEG)
HEG: Utility that converts EOSDIS data from
HDF-EOS format to common Geographical
Information System (GIS)-compatible formats.
• Two versions available:
– A downloadable desktop version
– Access through NASA archive online storage (Data Pools)
– Implemented also in some prototypes:
▪ Stand-alone Data Pool
▪ OGC (Open Geospatial Consortium) for Web Services Chaining
▪ DOWS (Deploy OGC Web Services) on the DAACs
Page 19
HEG Functionality
• Conversion:
– HDF-EOS (Swath or Grid ) to a single-band
or multi-band GeoTIFF’s
– HDF-EOS Swath to HDF-EOS Grid
– HDF-EOS Swath or Grid to generic Binary
– Metadata preservation/creation.
– Allows file selection from users local storage
Page 20
HEG Functionality
• Subsetting (spatial, field, band)
• Stitching (mosaicing) + subsetting + Reprojecting
• Reprojection ( to UTM, PS, TM, STP, LCC, LAMAZ,
GEO, SIN, Albers)
• Subsampling (Subsample stacks)
• Running from Command Line (batch jobs)
or with Java-based GUI
Page 21
HEG Functionality
• Currently supports MODIS, MISR, ASTER, AIRS and AMSR-E
products on TERRA and AQUA (>110 products),
• Operable on Sun, SGI, Win, Linux, MAC
• Integrated into ECS Data Pool
– Reduces the transfer time of HDF-EOS data sets (if subset is requested)
– Provides the end-user with the exact file required by their application
• User Interface
– Portable, written in Java.
– Not dependent on COTS (eg. IDL).
Page 22
Mt. Etna Eruption: MISR L1B
Stitched Blocks
Page 23
2004 Hurricane Francis, MODIS
MOD021KM, Multi-Band GeoTIFF, 3 bands:
EV_1KM_RefSB, Bands 1,2,5
Page 24
AST_L1B, 2003 California Fires
Geo. projection, Multi-Band GeoTIFF,
(3 bands: Visible Near Infrared (VNIR))
….. and Subsetted
Two Swaths Stitched …..
Page 25
Recent Major Development/Fixes
TOOLKIT/HDF-EOS:
 SZIP Insatallation: Default decoder,
Choice for encoder
 Choice for configuring HDF4 with/without SZIP
 Choice for disabling HDF4’s internal netCDF
 Choice for installing Toolkit/HDF-EOS in 32 or
64-bit modes in 64-bit Linux platforms
(except for Itanium; no –m32 flag)
Page 26
Recent Major Development/Fixes
 Support for new releases of HDF4/HDF5/SZIP
(HDF4.2r4, HDF5-1.8.X, SZIP-2.1)
 Auto Configuration capability (like HDF) for
HDF-EOS2 and HDF-EOS5
 Support for Mac Intel platform.
 Support for CYGWIN
Page 27
Recent Major Development/Fixes
 Added APIs GDij2ll and GDll2ij to HDF-EOS2 and
HDF-EOS5
 Added HDF_EOS 2 & 5 utilities for dumping grid
lat/lon into ASCII or binary files
 Bug fixes such as:
- HDFEOS5: Attribute data type inquiry (code returned not useful for reuse)
- HDFEOS5: Tiling/comp. consistency between dataset and structmetadata
- HDF-EOS2: Correct diminfo for a field (value in structmetadata is wrong?)
- HDF-EOS5: wrong datatype info for INT8 data (C code)
- Toolkit: Problem in pccheck utility( hangs if a blank space at a line start )
- Fix for ODL core dump (too many file names for Input Pointer)
Page 28
Recent Major Development/Fixes
HDF-EOS Plug-in for HDFView:
Current Support:
–
–
–
SUN, LINUX, WINDOWS,
and MAC (Power and Intel)
Auto installation of plug-in into HDFView
running self-extracting archives.
Adding to plug-in the fixes and enhancement
that went into HDFView2.4 and 2.5
Display image for selected pixels in the viewed
tables
Page 29
Recent Major Development/Fixes
–
–
–
–
–
–
Display Lat/Lon for a selected pixel in the
data field tables
Full Mac OSX support
Improved Help Window
New icons for activating Lat/Lon display for
Swath & Grid
Format conversion: HDF-EOS2 -> HDF-EOS5
Correction for compression and tiling values for
grid fields
Page 30
Recent Major Development/Fixes
HEG:
Product Support:
AMSR-E:
- Land products (CEA Projection added)
- AMSR-E 5 day Snow, and Monthly Snow
- Weekly Ocean, and Monthly Ocean
- Daily 12 Km, and Daily 6 Km Sea Ice
- L2 Rain, and Rain Grid
- ASTR_07XT, ASTR_09XT, ASTR_L1BE,
MOD29E1D
- More new versions and products tested. Will
update Users’ Guide and HEG web site
Page 31
Recent Major Development/Fixes
• Stitching fixes:
− MOD14A1 3-D fields (band selection)
− TM grids
• Multi-band geotif/HDF-EOS in Datapool
• Other Fixes:
− Interpolation correction with AIRS products
containing fillvalue in the data fields
− Pixel size unit consistency in conversion and stitch
GUIs.
Page 32
Recent Major Development/Fixes
− Correction for droping a column when
stitching MODIS SIN tiles.
− Fixes for processing with default pixel size instead
of ignoring it for Geographic projection
− Spatial subsetting fix for global AMSR-E garanules
− Fix for ignoring re-projection of type NN
interpolation after stitching
− support pixel resampling for gridded data
Page 33
Future Plans
TOOLKIT/HDF-EOS:
• Auto Configuration for TOOLKIT
• XML output for Metadata in TOOLKIT
• Update of DEM land-sea mask
• Support for HDF5-1.8.X
• Validation tool for HDF-EOS5 files
Page 34
Future Plans (cont.)
HDF-EOS Plug-in for HDFView:
• More functionalities :
- Ability to modify input HDF-EOS2 granules.
- Ability to cut/paste objects, modify/delete attributes.
- Ability to create new grid/swath
HEG:
- Support for AMSR-E Level 3 Land (CEA projection)
- New MRT (MODIS Reprojection Tool) release updates
- Support HEG for MAC Intel
Page 35
Availability
• Access to libraries, applications, Documentation:
TOOLKIT:
http://newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.html
HDFView:
http://newsroom.gsfc.nasa.gov/sdptoolkit/HDFView/
HDFView_hdfeos_plugin.html
HEG:
http://newsroom.gsfc.nasa.gov/sdptoolkit/HEG/HEGHome.html
• Email
– [email protected]
– [email protected]
Page 36