Transcript Integrating netCDF and OPeNDAP

Utilities for netCDF-4
Dr. Dennis Heimbigner
Unidata Advanced netCDF Workshop
July 25, 2011
Overview










Extending CDL to support netCDF-4 Model
netCDF4 in CDL: Types
netCDF4 in CDL: Typed Attributes
netCDF4 in CDL: Groups
Scope Rules
Specifying Data Constants
Special Attributes
NCGEN for netCDF4
NCGEN Command Synopsis
Extended Example
Extending CDL to Support
netCDF-4 Model


ncdump and ncgen both support CDL
extended to cover the netCDF-4 model.
CDL now has the following new elements:






Extended set of atomic types
User defined types
Typed Attributes
Groups
Special attributes
Extended Example
netCDF4 in CDL: Types


Supports the new atomic data types:
ubyte, ushort, uint, string, int64, uint64
New section called “types:” for user-defined
types
 int enum enum_t {t=0,f=1,other=2};
 opaque(11) opaque_t;
 compound cmpd_t { int64 f1; enum_t f2;};
 int(*) vlen_t;
netCDF4 in CDL: Typed Attributes





vlen_t v:attr = {17, 18, 19};
Attribute typing is optional (=> type inferred)
Warning! x:attr = “abc”; is inferred to be type char,
not string
 Instead say string x:attr = “abc”;
 Why? for backward compatibility with ncgen3
Good practice to add “_t” to the end of all type
names
Why? Because e.g. X :attr = … is ambiguous

Is X a type for a global attribute or a variable?
netCDF4 in CDL: Groups

Syntax: group: g {…} [Note the colon]

A group can itself contain types, dimensions,
variables, and (nested) groups

Name prefixing allows references to types and
dimensions that reside in other groups

Example: /g/cmpd_t
=> Do not use ‘/’ in your names

Pretty much like the Unix file system

Or Windows, but using forward slashes
Scope Rules


Scope rules determine how references to a
dimension or type without a prefix are
interpreted
General rule:
1. Look in immediately enclosing group
2. Look in the parent of the immediately
enclosing group and so on up the
enclosing groups


For dimensions, if not found => error
For types, continue to search the whole file
to find a unique match, then error if not
found
Specifying Data Constants

Constants for user defined types require the
use of braces {…} in certain places.
dimensions:
d=2;
types:
int(*) vlen_t;
compound cmpd_t { int64 f1; string f2;};
variables:
vlen_t v1(d);
cmpd_t v2(d);
data:
v1 = {7, 8, 9}, {17,18,19, 20, 21};
v2 = {107, “abc”}, {1234567, “xyz”};
Rules for Using Braces







The top level is automatically assumed to be a list of items,
so it should not be inside {...} (Different than C constants lists).
UNLIMITED dimension instances (other than the outer
dimension) must be surrounded by {...} in order to specify
the size.
Vlen instances must be surrounded by {...} in order to
specify the size.
Compound instances must be embedded in {...}
Non-scalar fields of compound instances must be
embedded in {...}.
Datalists associated with attributes are implicitly a vector
(i.e., a list) of values of the type of the attribute and the
above rules must apply with that in mind.
No other use of braces is allowed (esp. for arrays)
Special Attributes



Special attributes specified in an ncgen CDL file
will be properly handled
Consistent with ncdump -s
Global special attributes

“_Format” – specify the netCDF file format





“classic” (equivalent to –k1)
“64-bit offset” (-k2)
“netCDF-4” (-k3)
“netCDF-4 classic model” (-k4)
_Format is overridden by command line -k flag
Special Attributes (cont.)









Per-variable special attributes
_Storage – constant string “contiguous” or “chunked”
to set storage mode
_ChunkSizes – 1-d array of integer chunk sizes;1 per
dimension
_DeflateLevel – (integer range 0-9) compression level
_Endianness – constant strings “big” or “little”
_Fletcher32 – boolean to set check summing
_NoFill – boolean sets persistent NoFill property
_Shuffle – boolean to enable/disable shuffle filter
Note: boolean value can be “true”, “1”, “false”, or “0”
NCGEN Command Synopsis
Primary Command Line Option Extensions
[-k <file_format>] Format of the file to be created
1 => classic 32 bit
2 => classic 64 bit
3 => netcdf-4/CDM
4 => classic, but stored in netcdf-4 file format
[-l <language>]
Currently C (full netcdf-4 support);
Java, F77 (netcdf-3 only);
default => binary output with full netcdf-4
support
<filename>
Input CDL file
Extended Example

An extended example is included in the ncgen
man page

http://www.unidata.ucar.edu/software/netcdf/docs/ncgen-man-1.html
Questions?
Debugging Note
CDL:
dimensions: u = unlimited; variables: v1(u); v2(u);
data: v1 = {1,2,3,4}; v2 = {7,8};
Ncdump produces v2 = {7,8,_,_};





Use the “Cycle”
Use ncgen to convert your <file>.cdl to <file>.nc
Then use ncdump to convert your <file>.nc to
<file2>.cdl
Compare <file>.cdl to <file2>.cdl
Watch out for UNLIMITED!