Transcript HDF5 Tools Update

HDF5 Tools Update
Peter Cao - The HDF Group
[email protected]
November 6, 2007
This report is based upon work supported in part by a Cooperative Agreement with NASA under NASA NNX06AC83A. Any opinions,
findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the
views of the National Aeronautics and Space Administration.
Outline
Overview of current tools
New tools in HDF5
HDF-Java 2.4 release
11/6/07
HDF and HDF-EOS Workshop XI, Landover, MD
1
HDF5 Command Line Tools
Readers
h5dump, h5diff, h5ls
• new tools: h5check, h5stat
•
Writers
•h5repack,
h5repart, h5import, h5jam/h5unjam
•new tool: h5copy, h5mkgrp
Converters
•h4toh5,
11/6/07
h5toh4, gif2h5, h52gif
HDF and HDF-EOS Workshop XI, Landover, MD
2
Where to start
Tools located at bin/ directory
11/6/07
1.
Use “-h” option for help
2.
Online help
3.
Email to THG help desk
http://www.hdfgroup.org/hdf5tools.html
[email protected]
HDF and HDF-EOS Workshop XI, Landover, MD
3
h5dump
Dumps file content to stdout in
•
•
•
11/6/07
ASCII text
XML
Binary
HDF and HDF-EOS Workshop XI, Landover, MD
4
4
h5dump -H SDS.h5
HDF5 "SDS.h5" {
GROUP "/" {
GROUP "Floats" {
DATASET "FloatArray" {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 4, 3 ) / ( 4, 3 )
}
}
}
DATASET "IntArray" {
DATATYPE H5T_STD_I32LE
DATASPACE SIMPLE { ( 5, 6 ) / ( 5, 6 ) }
}
}
}
11/6/07
HDF and HDF-EOS Workshop XI, Landover, MD
5
h5dump -d /Floats/FloatArray SDS.h5
HDF5 "SDS.h5" {
DATASET "/Floats/FloatArray" {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 4, 3 ) / ( 4, 3 ) }
DATA {
(0,0): 0.01, 0.02, 0.03,
(1,0): 0.1, 0.2, 0.3,
(2,0): 1, 2, 3,
(3,0): 10, 20, 30
}
}
}
11/6/07
HDF and HDF-EOS Workshop XI, Landover, MD
6
h5dump -x SDS.h5
11/6/07
HDF and HDF-EOS Workshop XI, Landover, MD
7
h5dump Binary Output
-b F, --binary=[MEMORY|FILE|LE|BE]
h5dump -d /IntArray -o out_le.bin -b LE SDS.h5
0000000 00000000 00000001 00000002 00000003 00000004 00000005
0000030 0000000a 0000000b 0000000c 0000000d 0000000e 0000000f
0000060 00000014 00000015 00000016 00000017 00000018 00000019
0000110 0000001e 0000001f 00000020 00000021 00000022 00000023
0000140 00000028 00000029 0000002a 0000002b 0000002c 0000002d
11/6/07
HDF and HDF-EOS Workshop XI, Landover, MD
8
h5diff
Show Differences
Between
two objects or two files
11/6/07
HDF and HDF-EOS Workshop XI, Landover, MD
9
h5diff SDS.h5 SDS2.h5
Dataset: </IntArray> and </IntArray>
5 differences found
11/6/07
HDF and HDF-EOS Workshop XI, Landover, MD
10
Report differences
h5diff SDS.h5 SDS2.h5 -r /IntArray
Dataset:
</IntArray> and </IntArray>
position
IntArray
IntArray
difference
-----------------------------------------------------[ 0 0 ]
0
10
10
[ 1 0 ]
10
100
90
[ 2 0 ]
20
200
180
[ 3 0 ]
30
300
270
[ 4 0 ]
40
400
360
5 differences found
11/6/07
HDF and HDF-EOS Workshop XI, Landover, MD
11
h5repack
Copies a file to a new file with
•
•
•
11/6/07
Compression filter
Data layout
Other user options
HDF and HDF-EOS Workshop XI, Landover, MD
12
h5repack: new filters
-f {GZIP|SZIP|SHUF|FLET|NBIT|SOFF|NONE}
For example
h5repack -i SDS.h5 -o out.h5 -f GZIP=6
11/6/07
HDF and HDF-EOS Workshop XI, Landover, MD
13
h5repack: data layout
-l {CHUNK| COMPA| CONTI}
For example
h5repack -i SDS.h5 -o out.h5 -l CHUNK=2x3
11/6/07
HDF and HDF-EOS Workshop XI, Landover, MD
14
h5repack: latest version of file format
h5repack options
File size in bytes
Original file
3,167,264
--latest
2,267,346
--latest –compact=9
2,162,254
--latest –compact=9 –ssize=50:dtype
299,882
The testing file was created with HDF5 1.6.5. The root group of the file contains 100 groups. Each group at the
root contains 9 sub groups. Each sub group contains one compound dataset. The compound dataset has 40
members of 32 bit integer
11/6/07
HDF and HDF-EOS Workshop XI, Landover, MD
15
h5repart
Repartitions a file into a family of files
For example
h5repart -m 200m int16kx16k.h5 part200m%d.h5
977 MB
11/6/07
200 MB
part200m0.h5
200 MB
part200m1.h5
200 MB
part200m2.h5
200 MB
part200m3.h5
177 MB
part200m4.h5
HDF and HDF-EOS Workshop XI, Landover, MD
16
h5import
Imports binary/ASCII data into an HDF5 file
•
h5import infile -c config_file [infile -c config_file2 ...] -outfile outfile
For example
h5import float5x4x2.txt -c First_set.conf -o First_set.h5
PATH work/First-set
INPUT-CLASS TEXTFP
RANK 3
DIMENSION-SIZES 5 2 4
OUTPUT-CLASS FP
OUTPUT-SIZE 64
OUTPUT-ARCHITECTURE IEEE
OUTPUT-BYTE-ORDER LE
CHUNKED-DIMENSION-SIZES 2 2 2
MAXIMUM-DIMENSIONS 8 8 -1
11/6/07
GROUP "/" {
GROUP "work" {
DATASET "First-set" {
DATATYPE H5T_IEEE_F64LE
DATASPACE SIMPLE { ( 5, 2, 4 ) / ( 8, 8, H5S_UNLIMITED
)}
DATA {
(0,0,0): 1.01, 1.02, 1.03, 1.04,
(0,1,0): 1.11, 1.12, 1.13, 1.14,
(1,0,0): 1.21, 1.22, 1.23, 1.24,
(1,1,0): 1.31, 1.32, 1.33, 1.34,
(2,0,0): 1.41, 1.42, 1.43, 1.44,
…
}
}
}
}}
HDF and HDF-EOS Workshop XI, Landover, MD
17
h5jam/h5unjam
•
h5jam
h5jam -u test_ub.txt -i test_ub.h5
•
h5unjam
h5unjam -i test_ub.h5 -o out_ub.txt -o out_ub.h5
User Block
Jam
HDF5
unjam
User Block
11/6/07
HDF5
HDF and HDF-EOS Workshop XI, Landover, MD
18
h5ls
•
Lists selected information about file objects in the
specified format
For Example,
h5ls -r SDS2.h5
/Floats
/Floats/DoubleArray
/Floats/FloatArray
/Floats/subs
/IntArray
11/6/07
Group
Dataset {10, 5}
Dataset {4, 3}
Group
Dataset {5, 6}
HDF and HDF-EOS Workshop XI, Landover, MD
19
gif2h5 / h52gif
gif2h5 – converts a GIF file into HDF5
gif2h5 apollo17_earth.gif apollo17_earth.h5
h52gif – converts an HDF5 file into GIF
h52gif apollo17_earth.h5 apollo17_earth2.gif -i /apollo17_earth.gif/Image0 -p "/apollo17_earth.gif/Global Palette"
11/6/07
HDF and HDF-EOS Workshop XI, Landover, MD
20
h5toh4 / h4toh5
h5toh4 -- Converts an HDF5 file to an HDF4 file
h4toh5 -- Converts an HDF4 file to an HDF5 file
http://www.hdfgroup.org/h4toh5/
11/6/07
HDF and HDF-EOS Workshop XI, Landover, MD
21
New tools
h5copy
h5mkgrp
h5stat
h5check
11/6/07
HDF and HDF-EOS Workshop XI, Landover, MD
22
h5copy
Copies an object within a file/cross files
/
/
IntArray
Floats
FloatArray
FloatArray
11/6/07
HDF and HDF-EOS Workshop XI, Landover, MD
23
h5copy -f shallow
/
floats
/
64-bit
f32
floats
integers
/
i1
i2
f32
64-bit
f1
floats
f2
f32
64-bit
f1
11/6/07
HDF and HDF-EOS Workshop XI, Landover, MD
f2
24
h5copy -f soft
/
/
dset_SL
f1
/f1
f1
dset_SL
/f1
/
dset_SL
/f1
11/6/07
HDF and HDF-EOS Workshop XI, Landover, MD
25
h5copy -f ref
/
d1
d2
/
dset_ref
679
1287
d2
d1
/
dset_ref
1895
763
dset_ref
0
0
11/6/07
HDF and HDF-EOS Workshop XI, Landover, MD
26
h5mkgrp
Creates groups
h5mkgrp test.h5 /info /data /data/test1 /data/test2
/
info
data
test1
11/6/07
test2
HDF and HDF-EOS Workshop XI, Landover, MD
27
h5stat
Prints statistics about HDF5 file
Header size
• Raw data size
• Attributes, datatypes, and etc.
•
Helps
To troubleshoot overhead in
HDF5 files
• To choose specific object’s
properties and storage strategies
•
11/6/07
HDF and HDF-EOS Workshop XI, Landover, MD
28
h5check
A validation tool that verifies if an HDF5 file is
encoded according to the HDF5 File Format
Specification
11/6/07
HDF and HDF-EOS Workshop XI, Landover, MD
29
Why h5check?
•
Verify if the file is compliant with the File Format
•
As a verification tool required by the application of
HDF5 File Format to be an ANSI standard
•
Serves as a watch dog that the HDF5 library
implementation is compliant with the File Format
11/6/07
HDF and HDF-EOS Workshop XI, Landover, MD
30
Example: a compliant file
•
% h5check example1.h5
•
•
•
•
•
•
•
•
•
•
11/6/07
VALIDATING example1.h5
FOUND super block signature
VALIDATING the super block at 0...
VALIDATING the object header at 928...
VALIDATING the btree at 384...
FOUND btree signature.
VALIDATING the local heap at 96...
FOUND local heap signature.
…
Result: File is in compliance.
HDF and HDF-EOS Workshop XI, Landover, MD
31
Example: a non-compliant file
h5check invalid2.h5
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
11/6/07
FOUND super block signature
VALIDATING the super block at 0...
VALIDATING the object header at 928...
VALIDATING the btree at 384...
FOUND btree signature.
VALIDATING the SNOD at 1248...
FOUND SNOD signature.
VALIDATING the object header at 976...
check_sym(at 1248): Errors from check_obj_header()
decode_validate_messages(): Failure in type->decode().
H5O_sdspace_decode(): Bad version number in simple dataspace message.
VALIDATING the local heap at 96...
FOUND local heap signature.
Main(): Errors from check_obj_header().
decode_validate_messages(): Failure in type->decode().
H5O_attr_decode(): Can't decode attribute dataspace.
H5O_sdspace_decode(): Bad version number in simple dataspace message.
…
Result: File is not in compliance.
HDF and HDF-EOS Workshop XI, Landover, MD
32
Implementation Status
Basic file format components
1.6 - DONE
1.8 - IN PROGRESS
11/6/07
HDF and HDF-EOS Workshop XI, Landover, MD
33
HDF-Java 2.4
HDF-Java 2.4 Released !!
http://www.hdfgroup.org/hdf-java-html
ftp://ftp.hdfgroup.org/HDF5/hdf-java
11/6/07
HDF and HDF-EOS Workshop XI, Landover, MD
34
HDF-Java 2.4 Release
•
•
Build with HDF4 2r2 and HDF5 1.6.6
New platforms
Mac intel
Linux 64-bit AMD
Solaris 64-bit
•
•
•
•
36 new enhancements and 44 bugs fixed
Fixed memory leak
Test suite
Enhanced documentation
11/6/07
HDF and HDF-EOS Workshop XI, Landover, MD
35
Memory leak in old version
while (true)
{
H5File f = new H5File
(fname, H5File.READ);
f.open();
f.close();
}
11/6/07
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
HDF and HDF-EOS Workshop XI, Landover, MD
36