Transcript GRIB2 vs NetCDF: Evaluation of the Technical Aspects Bruce Wright (IT Architect) © Crown copyright Met Office.

GRIB2 vs NetCDF: Evaluation
of the Technical Aspects
Bruce Wright (IT Architect)
© Crown copyright Met Office
Contents
This presentation covers the following areas
• Data Format Overview
• GRIB vs NetCDF
• Summary
© Crown copyright Met Office
Data Format Overview
© Crown copyright Met Office
GRIB
• Developed by WMO for the exchange of gridded data
• Allowing detailed description of a huge variety of grids,
parameters, processes, represented by codes that
reference external tables
• Portable, implemented as octets (groups of 8 bits):
Section
Name
Length
Notes
0
Indicator
16 bytes
"GRIB" initially
1
Identification
21 bytes
2*
Local Use
3 **
Grid definition
52 bytes?
4 ***
Product definition
14-50 bytes?
5 ***
Data
representation
typically 21 bytes
Dependent on compression
6 ***
Bit-map
6 bytes + bitmap
Optional section
7 ***
Data
5 bytes + data
8
End
4 bytes
© Crown copyright Met Office
Optional section - may contain anything
"7777"
*, **, *** - can be nested
NetCDF
• Developed by Unidata to facilitate the access and sharing of
array-orientated data in a form that was self-describing and
portable
• Simple, but flexible, data model based around variables,
dimensions and attributes.
• Climate and Forecast (CF) convention (developed for
sharing climate model and NWP forecast data) offers a
widely-used standard for metadata (esp. through standard
names)
• Users interact with the data and metadata in a file through
an API that allows things like querying the number of
variables (data objects) in a file, reading or writing slices of
data to the variables, defining the variables in a file after
creation, etc.
• Unidata provide a library of routines for interacting with
NetCDF files
© Crown copyright Met Office
GRIB vs NetCDF
© Crown copyright Met Office
Interoperability
GRIB
NetCDF
Standard
WMO standard
Used by NMSs and other
areas of operational
meteorology (e.g. Aviation)
De facto standard
Used by wider fluid earth
sciences community (esp.
research & academia)
Machine
Independent
Yes
Yes
SelfDescribing
No – interpretation requires: Yes – low-level definition of
dimensions, variables and
- WMO-agreed tables
attributes
- Local tables
COTS Support Limited
Mainly WMO-sponsored
Transmission
(& Archiving)
© Crown copyright Met Office
Sequential – record-based
access
More extensive
ERSI ArcView 9.2
OGC web service standard
Direct Access – whole file
required
Metadata
GRIB
NetCDF (CF)
Comprehensive
Yes
Standard range of
descriptors agreed out of
considerable experience
Less so than GRIB for
operational meteorology
…but potential to be
extremely comprehensive
Flexible
Yes, but…
Use of local tables can be
‘clumsy’
Highly
Use of global and variable
attributes
Human
Readable
No
Yes
SelfDescribing
No – refers to external
tables
Yes
Extensible
Yes, but…
WMO process slow
Local table limitations
Highly
CF managed through active
community mailing list
© Crown copyright Met Office
Interface
GRIB
NetCDF (CF)
Standard
Interface
No, but…
ECMWF GRIB API
NCEP software…
Yes (from Unidata)
NetCDF is really an
interface, rather than a
format
Languages
ECMWF – C, FORTRAN
77, Java
NCEP – C, FORTRAN 90,
(Tcl/Tk GUI, MS-Windows
degrib)
C, C++, FORTRAN 77,
FORTRAN 90
Java
Bindings to Perl, python,
ruby, IDL, Tcl, (PV-Wave),
MATLAB, R, OpenDX
Platforms
ECMWF has tested on
Linux, AIX, HP-UX, IRIX
Most platforms
Installation &
Use
?
Easy Linux install, learning
curve but not severe
© Crown copyright Met Office
Compression
GRIB
NetCDF (CF)
Packing
Good
Simple packing
Complex, with different
algorithms
Crude packing only
Lossless
Compression
Proposals to include LZW,
Weather-Huffman,
JPEG2000
Fraunhofer Institute
developed compression
(factor 2.5 on 16 bit data)
No for NetCDF3, but Cornell
developed extension
NetCDF4 (HDF5) supports
various
Licencing
Issues
None known, except for
specialist solutions
Possible limitations for
some algorithms
© Crown copyright Met Office
General
GRIB
NetCDF
Dimensions
2D fields only
GRIB2 allows these to be
grouped for higher
dimensions
N-dimensional
e.g. 3 spatial dimensions
and time in a single array
Small Reads
Whole file must be parsed
to index file
Efficient, as ‘header’
provides direct access
Initial Write
Sequential format allow
efficient streaming
File has to be constructed
using API
© Crown copyright Met Office
Summary
© Crown copyright Met Office
Summary
• CF-compliant NetCDF (CF-NetCDF)
• NetCDF interface developed by Unidata to facilitate the access and
sharing of array-orientated data in a form that was self-describing and
portable (the format being machine-independent)
• Highly flexible data management solution for multi-dimensional
gridded data
• Very widely used in atmospheric and oceanographic sciences
community
• CF (Climate-Forecast) metadata convention offers, arguably, the best
metadata standard available within this community
• GRIB2
• Developed as WMO standard to provide an efficient, machineindependent format for the exchange of gridded data by National Met
Services
• No standard interface, although several have been developed
• Metadata is code-based, needing to cross-reference external tables,
with a highly specified metadata ‘vocabulary’ and layout and no
indexing of the data
© Crown copyright Met Office