Transcript Advanced Utilities Extending ncgen to support the netCDF-4 Data Model Dr. Dennis Heimbigner

Advanced Utilities
Extending ncgen to support the
netCDF-4 Data Model
Dr. Dennis Heimbigner
Unidata netCDF Workshop
August 3-4, 2009
Overview












The NCGEN4 Utility
NCGEN4 Command Synopsis
netCDF4 in CDL: Types
netCDF4 in CDL: Typed Attributes
netCDF4 in CDL: Groups
Scope Rules
Specifying Data Constants
Rules for Using Braces
Special Attributes
Debugging Note: The Cycle
Building and Installing NCGEN4
Extended Example
The NCGEN4 Utility





NCGEN4 extends the CDL language
Includes all of the netCDF-4 data model
True inverse of ncdump
Includes the special attributes (chunking, etc)
Supports binary, C, NOT FORTRAN (yet)


Experimental: Java and NcML
Can produce 4 kinds of binary netCDF files:




netcdf-3 (classic) 32 bit
netcdf-3 (classic) 64 bit
netcdf-3 (classic, but stored in netcdf-4 file format)
netcdf-4 (supports full netcdf-4 model)
NCGEN4 Command Synopsis
Primary Command Line Options
[-b]
Create a (binary) netCDF file (default)
[-o <file>]
Name for the binary netCDF file created
[-k <file_format>] Format of the file to be created
1 => classic 32 bit
2 => classic 64 bit
3 => netcdf-4/CDM
4 => classic, but stored in an enhanced file format
[-x]
Don't initialize data with fill values
[-l <language>]
Specify output language to use when generating
source code to create or define a netCDF file
matching the CDL specification.
Input CDL file
<filename>
netCDF4 in CDL: Types








New section called “types:”
Consistent with the output of ncdump
Supports the new primitive data types:
ubyte, ushort, uint, string, int64 (LL),
uint64 (ULL)
Supports the new user-defined types:
int enum enum_t {off=0,on=1,unknown=2};
opaque(11) opaque_t;
compound cmpd_t { vlen_t f1; enum_t f2;};
int(*) vlen_t;
netCDF4 in CDL: Typed Attributes







vlen_t v:attr = {17, 18, 19};
Attribute typing is optional (=> type inferred)
Warning! x:attr = “abc”; is inferred to be type
char, not string
Instead say string x:attr = “abc”;
Why? for backward compatibility with ncgen
Good practice to add “_t” to the end of all type
names
Why? Because X :attr = … might be interpreted
incorrectly; is X a type or variable?
netCDF4 in CDL: Groups






group: g {…}
A group can itself contain dimensions, types,
variables, and groups
Name prefixing allows references to types and
dimensions that reside in other groups
Example: /g/cmpd_t
=> Do not use ‘/’ in your names
Pretty much like the Unix file system
Or Windows, but using forward slashes
Scope Rules


Scope rules determine how references to a
dimension or type without a prefix are
interpreted
General rule:
1. Look in immediately enclosing group
2. Look in the parent of the immediately enclosing
group and so on up the enclosing groups


For dimensions, if not found => error
For types, continue to search the whole
group tree to find a unique match, then
error if not found
Specifying Data Constants

Constants for user defined types require the
use of braces {…} in certain places.
dimensions:
d=2;
types:
int(*) vlen_t;
compound cmpd_t { int64 f1; string f2;};
variables:
vlen_t v1(d);
cmpd_t v2(d);
data:
v1 = {7, 8, 9}, {17,18,19};
v2 = {107LL, “abc”}, {1234567LL, “xyz”};
Rules for Using Braces

The top level is automatically assumed to be a list
of items, so it should not be inside {...}






Different than C constants lists
Instances of UNLIMITED dimensions (other than
as the first dimension) must be surrounded by {...}
in order to specify the size.
Instances of VLENs must be surrounded by {...} in
order to specify the size.
Compound instances must be embedded in {...}
Compound fields may optionally be embedded in
{...}.
No other use of braces is allowed.
Special Attributes



Special attributes specified in an ncgen4 CDL
file will be properly handled
Consistent with ncdump -s
Global special attributes

“_Format” – specify the netCDF file format





“classic”
“64-bit offset”
“netCDF-4”
“netCDF-4 classic model”
Overridden by the -k flag
Special Attributes (cont.)

Per-variable special attributes







“_ChunkSizes” – list of chunk sizes 1 per dimension
“_DeflateLevel” – compression level: integer (0-9)
“_Endianness” – “big” or “little”
“_Fletcher32” – “true” or “false” to set check summing
“_NoFill” – “true” or “false” to set persistent NoFill
property
“_Shuffle” – “true” or “false” to set shuffle filter
“_Storage” – “contiguous” or “chunked” to set storage
mode
Debugging Note





Use the “Cycle”, Luke
Use ncgen/ncgen4 to convert your <file>.cdl
to <file>.nc
Then use ncdump to convert your <file>.nc
to <file2>.cdl
Compare <file>.cdl to <file2>.cdl
Watch out for UNLIMITED!
dimensions: u = unlimited; variables: v1(u); v2(u);
data: v1 = {1,2,3,4}; v2 = {7,8};
Ncdump produces v2 = {7,8,_,_};
Building and Installing NCGEN4


Easy: add --enable-ncgen4 to your list of
./configure flags
Ncgen4 will be installed along with ncdump
and the original ncgen
Extended Example
netcdf foo {
types:
ubyte enum enum_t {Clear = 0, Cumulonimbus = 1, Stratus = 2};
opaque(11) opaque_t;
int(*) vlen_t;
dimensions:
lat = 10; lon = 5; time = unlimited ;
variables:
long
float
double
long
string
ubyte
lat(lat), lon(lon), time(time);
Z(time,lat,lon), t(time,lat,lon);
p(time,lat,lon);
rh(time,lat,lon);
country(time,lat,lon);
tag;
Extended Example (cont.)
// variable attributes
lat:long_name = "latitude";
lat:units = "degrees_north";
lon:long_name = "longitude";
lon:units = "degrees_east";
time:units = "seconds since 1992-1-1 00:00:00";
// typed variable attributes
string Z:units = "geopotential meters";
float Z:valid_range = 0., 5000.;
double p:_FillValue = -9999.;
long rh:_FillValue = -1;
vlen_t :globalatt = {17, 18, 19};
Extended Example (cont.)
data:
lat = 0, 10, 20, 30, 40, 50, 60, 70, 80, 90;
lon = -140, -118, -96, -84, -52;
group g {
types:
compound cmpd_t { vlen_t f1; enum_t f2;};
} // group g
group h {
variables:
/g/cmpd_t compoundvar;
data:
compoundvar = { {3,4,5}, Stratus } ;
} // group h
}
Questions?