Transcript Slide 1
Introduction to Exchange
Formats
First Data Management Training Workshop
12-17 February 2007, Oostende, Belgium
Sissy IONA
First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium
1
Exchange Data Formats in SeaDataNet
CF-compliant NetCDF binary format (for gridded fields and multidimensional data types)
ODV ASCII spreadsheet format (for profiles, time series)
MEDATLAS ASCII format (optional)
2
First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium
NetCDF data format
•
NetCDF (network Common Data Form) is an interface to create, access, and
share array-oriented data in a form that is self-describing and portable. "Selfdescribing" means that a dataset includes information defining the data it contains.
"Portable" means that the data in a dataset is represented in a form that can be
accessed by computers with different ways of storing integers, characters, and
floating-point numbers. Using the netCDF interface for creating new datasets makes
the data portable. Using the netCDF interface in software for data access,
management, analysis, and display can make the software more generally useful.
The netCDF software includes C, Fortran, C++, Java and other interfaces for
accessing netCDF data. These libraries are available for many common computing
platforms.”
(NetCDF Users Guide, V3.6.1, 2006, http://www.unidata.ucar.edu/software/netcdf)
•
Principally, designed for gridded data but extended to other observational data.
•
NetCDF software was developed at the Unidata Program Center in Boulder,
Colorado. It is freeley available at the above UCAR’s website.
3
First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium
CF metadata Convention
•
•
•
The Climate and Forecast (CF) Conventions is a standard designed to promote the
processing and sharing of NetCDF files.
The conventions locate data in space-time and as a function of other independent
variables, facilitate processing and graphics.
Identify data sufficiently to enable users of data from different sources to decide what
is comparable and to distinguish variables in archives.
•
The CF conventions generalize and extend the COARDS conventions.
•
The CF conventions can be contained in other formats like XML.
•
•
The CF conventions includes features that describe:
Grid-cell boundaries
Horizontal grids other than latitude-longitude
Recording common statistical operations
Standardised identification of physical quantities
Non-spatiotemporal axes
Climatological statistics
Data compression
Full documentation available on http://www.cgd.ucar.edu/cms/eaton/cf-metadata/
4
First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium
NetCDF data file structure and main CF features
A NetCDF data file has four basic components:
Dimensions: gives the dimension information for the variables. Dimensions can be
spatial, temporal or any other quantity. The dimensions can contain regularly or
irregularly space steps and there is no limit in the number of dimensions. The
‘UNLIMITED’ dimension occurs when the dimension is ‘time’ and data are still being
taken.
Variables: contain information on the actual variables as well as on each dimension.
Coordinates also contained in variables. The variables should be named using the CF
Standard Names, otherwise ‘long_name’ attribute should be used to describe the
variable. CF requires all variables to have units unless they contain dimensionless
numbers.
Global attributes: must include as much information as possible. CF conventions
provide some basic “discovery” metadata in global attributes:
title : description of what is in the dataset
Institution : that produces the data
source : the method of production of the original data - model, observational
history : modifications to the original data
references : publications or web references
Comment : any other information
Data: stores the data. CF conventions suggest the ‘_FillValue’ attribute for missing
data and defined as the same type of the variable that replaces.
5
First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium
Example of gridded data
PCMDI Sea Surface temperature data set in NetCDF format following the CF conventions:
-------------------------------------------------------------------------------------------------------------------------------------------
netcdf tos_O1_2001-2002 {
filename must have “.nc” extension
dimensions:
lon = 180 ;
lat = 170 ;
time = UNLIMITED ; // (24 currently)
bnds = 2 ;
number of vertices
variables:
double lon(lon);
coordinate variable of 1 dimension
lon:standard_name = "longitude”; legal value from CF Standard_Name table
lon:long_name = "longitude" ;
lon:units = "degrees_east”;
compliant with CF/COARDS/Udunits
lon:axis = "X”;
legal value of axis are X,Y,Z and T
lon:bounds = "lon_bnds“;
represent the cell
lon:original_units = "degrees_east" ;
double lon_bnds(lon, bnds);
define the extent of the cell and the grid point location
double lat(lat) ;
lat:standard_name = "latitude" ;
lat:long_name = "latitude" ;
lat:units = "degrees_north" ;
lat:axis = "Y" ;
lat:bounds = "lat_bnds" ;
lat:original_units = "degrees_north" ;
double lat_bnds(lat, bnds) ;
7
First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium
Example of gridded data
double time(time) ;
time:standard_name = "time" ;
time:long_name = "time" ;
time:units = "days since 2001-1-1”;
the time units must contain a reference time
time:axis = "T" ;
time:calendar = "360_day“;
defines the permitted values of year,month,day
time:bounds = "time_bnds" ;
time:original_units = "seconds since 2001-1-1" ;
double time_bnds(time, bnds) ;
float tos(time, lat, lon) ;
tos:standard_name = "sea_surface_temperature" ;
tos:long_name = "Sea Surface Temperature" ;
tos:units = "K" ;
tos:cell_methods = "time: mean (interval: 30 minutes)" ; defines the variation within the cells
tos:_FillValue = 1.e+20f ;
tos:missing_value = 1.e+20f ;
tos:original_name = "sosstsst" ;
tos:original_units = "degC" ;
tos:history = " At 16:37:23 on 01/11/2005: CMOR altered the data in the following ways: added
2.73150E+02 to yield output units; Cyclical dimension was output starting at a different lon;" ;
8
First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium
Example of gridded data
// global attributes:
:title = "IPSL model output prepared for IPCC Fourth Assessment SRES A2 experiment" ;
:institution = "IPSL (Institut Pierre Simon Laplace, Paris, France)" ;
:source = "IPSL-CM4_v1 (2003) : atmosphere : LMDZ (IPSL-CM4_IPCC, 96x71x19) ; ocean ORCA2
(ipsl_cm4_v1_8, 2x2L31); sea ice LIM (ipsl_cm4_v" ;
:contact = "Sebastien Denvil, [email protected]" ;
:project_id = "IPCC Fourth Assessment" ;
:table_id = "Table O1 (13 November 2004)" ;
:experiment_id = "SRES A2 experiment" ;
:realization = 1 ;
:cmor_version = 0.96f ;
:Conventions = "CF-1.0" ;
Identification of convention
:history = "YYYY/MM/JJ: data generated; YYYY/MM/JJ+1 data transformed At 16:37:23 on 01/11/2005,
CMOR rewrote data to comply with CF standards and IPCC Fourth Assessment requirements" ;
:references = "Dufresne et al, Journal of Climate, 2015, vol XX, p 136" ;
:comment = "Test drive" ;
data:
lon = 1, 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29, 31, 33, 35, 37,
39, 41, 43, 45, 47, 49, 51, 53, 55, 57, 59, 61, 63, 65, 67, 69, 71, 73,
........................................................................................................................
lon_bnds =
-0, 2,
2, 4,
4, 6,
........................................................................................................................
9
First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium
Example of multi-dimensional data
OceanSITES ADCP time series in NetCDF format compliant with the CF/COARDS standard:
------------------------------------------------------------------------------------------------------------------------------------------netcdf gebco2005_leg1_150K_1_fhv1 {
dimensions:
N_DATE_TIME = 390 ;
N_LEVEL = 40 ;
DATE_TIME = 14 ;
CONST1 = 1 ;
CONST2 = 2 ;
STRING32 = 32 ;
STRINGFILT = 30 ;
STRINGFILTUSES = 1 ;
variables:
char REFERENCE_DATE_TIME(DATE_TIME) ;
REFERENCE_DATE_TIME:long_name = "Date of reference for Julian days" ;
REFERENCE_DATE_TIME:convention = "YYYYMMDDHHMISS" ;
REFERENCE_DATE_TIME:_FillValue = " " ;
double JULD(N_DATE_TIME) ;
JULD:long_name = "Julian days relative to REFERENCE_DATE_TIME" ;
JULD:_FillValue = -999999. ;
float CAS_DATE_FLAG(N_DATE_TIME) ;
CAS_DATE_FLAG:long_name = "Flag on ADCP current Juld" ;
CAS_DATE_FLAG:valid_min = 0s ;
CAS_DATE_FLAG:valid_max = 10s ;
CAS_DATE_FLAG:_FillValue = -99999.f ;
double JULD_ADCP(N_DATE_TIME) ;
JULD_ADCP:long_name = "Julian days ADCP relative to REFERENCE_DATE_TIME" ;
JULD_ADCP:_FillValue = -999999. ;
char DATE_TIME_UTC(N_DATE_TIME, DATE_TIME) ;
DATE_TIME_UTC:long_name = "ASCII gregorian date and time" ;
DATE_TIME_UTC:convention = "YYYYMMDDHHMISS" ;
DATE_TIME_UTC:_FillValue = " " ;
float LATITUDE(N_DATE_TIME) ;
LATITUDE:long_name = "Latitude of each location" ;
LATITUDE:units = "degree_north" ;
LATITUDE:valid_min = -90.f ;
LATITUDE:valid_max = 90.f ;
LATITUDE:_FillValue = -999999.f ;
float LONGITUDE(N_DATE_TIME) ;
LONGITUDE:long_name = "Longitude of each location" ;
LONGITUDE:units = "degree_east" ;
LONGITUDE:valid_min = -180.f ;
LONGITUDE:valid_max = 180.f ;
LONGITUDE:_FillValue = -999999.f ;
First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium
10
Example of multi-dimensional data
float UVEL_SHIP(N_DATE_TIME) ;
UVEL_SHIP:units = "meter per second" ;
UVEL_SHIP:long_name = "Eastward ship velocity" ;
UVEL_SHIP:valid_min = -20.f ;
UVEL_SHIP:valid_max = 20.f ;
UVEL_SHIP:_FillValue = -999999.f ;
float VVEL_SHIP(N_DATE_TIME) ;
VVEL_SHIP:long_name = "Northward ship velocity" ;
VVEL_SHIP:units = "meter per second" ;
VVEL_SHIP:valid_min = -20.f ;
VVEL_SHIP:valid_max = 20.f ;
VVEL_SHIP:_FillValue = -999999.f ;
float DEPH(N_LEVEL) ;
DEPH:long_name = "Depth of bin center" ;
DEPH:units = "meter" ;
DEPH:valid_min = -12000.f ;
DEPH:valid_max = 0.f ;
DEPH:_FillValue = -999999.f ;
float TEMP_ADCP(N_DATE_TIME) ;
TEMP_ADCP:long_name = "ADCP transducer temperature" ;
TEMP_ADCP:units = "degree_Celsius" ;
TEMP_ADCP:valid_min = -5.f ;
TEMP_ADCP:valid_max = 45.f ;
TEMP_ADCP:_FillValue = -999999.f ;
float HDG(N_DATE_TIME) ;
HDG:long_name = "Ship heading" ;
HDG:units = "degree" ;
HDG:valid_min = -360.f ;
HDG:valid_max = 360.f ;
HDG:_FillValue = -999999.f ;
float PTCH(N_DATE_TIME) ;
PTCH:long_name = "Ship pitch" ;
PTCH:units = "degree" ;
PTCH:valid_min = -360.f ;
PTCH:valid_max = 360.f ;
PTCH:_FillValue = -999999.f ;
float ROLL(N_DATE_TIME) ;
ROLL:long_name = "Ship roll" ;
ROLL:units = "degree" ;
ROLL:valid_min = -360.f ;
ROLL:valid_max = 360.f ;
ROLL:_FillValue = -999999.f ;
11
First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium
Example of multi-dimensional data
float CAS_CURRENT_FLAG(N_DATE_TIME, N_LEVEL) ;
CAS_CURRENT_FLAG:long_name = "Flag on ADCP current (U,V,W)" ;
CAS_CURRENT_FLAG:valid_min = 0s ;
CAS_CURRENT_FLAG:valid_max = 10s ;
CAS_CURRENT_FLAG:_FillValue = -99999.f ;
float VVEL_ADCP(N_DATE_TIME, N_LEVEL) ;
VVEL_ADCP:long_name = "Northward absolute ADCP current velocity" ;
VVEL_ADCP:units = "meter per second" ;
VVEL_ADCP:valid_min = -20.f ;
VVEL_ADCP:valid_max = 20.f ;
VVEL_ADCP:_FillValue = -999999.f ;
float UVEL_ADCP(N_DATE_TIME, N_LEVEL) ;
UVEL_ADCP:long_name = "Eastward absolute ADCP current velocity" ;
UVEL_ADCP:units = "meter per second" ;
UVEL_ADCP:valid_min = -20.f ;
UVEL_ADCP:valid_max = 20.f ;
UVEL_ADCP:_FillValue = -999999.f ;
float WVEL_ADCP(N_DATE_TIME, N_LEVEL) ;
WVEL_ADCP:long_name = "Upward absolute ADCP current velocity" ;
WVEL_ADCP:units = "meter per second" ;
WVEL_ADCP:valid_min = -20.f ;
WVEL_ADCP:valid_max = 20.f ;
WVEL_ADCP:_FillValue = -999999.f ;
float EVEL_ADCP(N_DATE_TIME, N_LEVEL) ;
EVEL_ADCP:long_name = "Absolute ADCP current velocity error" ;
EVEL_ADCP:units = "meter per second" ;
EVEL_ADCP:valid_min = -20.f ;
EVEL_ADCP:valid_max = 20.f ;
EVEL_ADCP:_FillValue = -999999.f ;
float PGOOD_ADCP_B1(N_DATE_TIME, N_LEVEL) ;
PGOOD_ADCP_B1:long_name = "Percent of good data before ensemble averaging" ;
PGOOD_ADCP_B1:units = "percent" ;
PGOOD_ADCP_B1:comment = "Number of good data = PGOOD * NB_ENS_AVE" ;
PGOOD_ADCP_B1:valid_min = 0.f ;
PGOOD_ADCP_B1:valid_max = 100.f ;
PGOOD_ADCP_B1:_FillValue = -999999.f ;
float PGOOD_ADCP_B2(N_DATE_TIME, N_LEVEL) ;
PGOOD_ADCP_B2:long_name = "Percent of good data before ensemble averaging" ;
PGOOD_ADCP_B2:units = "percent" ;
PGOOD_ADCP_B2:comment = "Number of good data = PGOOD * NB_ENS_AVE" ;
PGOOD_ADCP_B2:valid_min = 0.f ;
PGOOD_ADCP_B2:valid_max = 100.f ;
PGOOD_ADCP_B2:_FillValue = -999999.f ;
…………………………………………………………………………………………………………………………..
First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium
12
Example of multi-dimensional data
…………………………………………………………………………………………………………………………..
Float VVEL_ADCP_CORTIDE(N_DATE_TIME, N_LEVEL) ;
VVEL_ADCP_CORTIDE:units = "meter per second" ;
VVEL_ADCP_CORTIDE:long_name = "Northward absolute velocity corrected for tide" ;
VVEL_ADCP_CORTIDE:valid_min = -20.f ;
VVEL_ADCP_CORTIDE:valid_max = 20.f ;
VVEL_ADCP_CORTIDE:_FillValue = -999999.f ;
char FILT_TYPE(STRINGFILT) ;
FILT_TYPE:long_name = "Type of filtering" ;
char FILT_FLAGS(STRINGFILTUSES) ;
FILT_FLAGS:long_name = "List of flags used before filtering" ;
// global attributes:
:DATE_CREATION = "26-Oct-2005" ;
:SOFTWARE = "CascadeVersion 5.3" ;
:ADCP_CONSTRUCTOR = "RDI" ;
:ADCP_TYPE = "Ocean Surveyor" ;
:CONVENTIONS = "OceanSite dictionary" ;
:CRUISE_NAME = "gebco2005" ;
:PLATFORM_NUMBER = "FABB" ;
:PLATFORM_NAME = "BEAUTEMPS-BEAUPRE" ;
:DATE_UPDATE = "26-Oct-2005" ;
:DATA_TYPE = "SADCP" ;
:FORMAT_VERSION = "1.0" ;
:NAVIGATION_REFERENCE = "GPS Navigation" ;
:HEADING_REFERENCE = "GPS Heading" ;
data:
REFERENCE_DATE_TIME = "19500101000000" ;
JULD = 20361.5601215279, 20361.5670601851, 20361.5739872684,
20361.5809317129, 20361.5878761574, 20361.5948379631, 20361.6017824076,
20361.6087268516, 20361.6156712961, 20361.6226157406, 20361.6295775464,
20361.6365219909, 20361.6434664354, 20361.6503935186, 20361.6573379631,
20361.6642997684, 20361.6712442129, 20361.6781886574, 20361.6851157406,
20361.6920601851, 20361.6990219909, 20361.7059664354, 20361.7129108794,
…………………………………………………………………………………………………………………………..
13
First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium
ODV Generic Spreadsheet format
ODV (Ocean Data View) is a simple, flexible, user friendly ASCII format.
It consist of 3 parts:
Comment lines
Header line
Data lines: up to 50 values
The ODV QC flags are: 0=good, 1=unknown, 4=questionable, 8=bad
It supports files in ASCII and binary NetCDF format: water profile data, benthic
core profile data, Surface underway data, Moored instrument time series
Developed by Schlitzer, R., http://odv.awi.de
14
First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium
Example of a CTD vertical profile
// Data from: C:/sissy/ODV/tests/New_ODV_Collection (Feb/09/2007 11:59:08)
// Extracted: Feb/09/2007 12:00:17 by Iona@SISSY
Cruise;Station;Type;mon/day/yr;hh:mm;Longitude [degrees_east];Latitude [degrees_north];Bot.
Depth [m];Pressure [decibar];QF;Temperature [degrees Celsius];QF;Salinity [psu];QF;Oxygen
[ml/l];QF
36AE_GN36199402802;00010;C;6/1/1994;06:30;22.5;36.25;2386;8;0;19.734;0;38.703;0;'9999';8
36AE_GN36199402802;00010;C;6/1/1994;06:30;22.5;36.25;2386;9;0;19.683;0;38.702;0;'9999';8
36AE_GN36199402802;00010;C;6/1/1994;06:30;22.5;36.25;2386;10;0;19.62;0;38.713;0;'9999';8
36AE_GN36199402802;00010;C;6/1/1994;06:30;22.5;36.25;2386;11;0;19.442;0;38.717;0;'9999';8
36AE_GN36199402802;00010;C;6/1/1994;06:30;22.5;36.25;2386;12;0;19.052;0;38.728;0;'9999';8
36AE_GN36199402802;00010;C;6/1/1994;06:30;22.5;36.25;2386;13;0;18.662;0;38.739;0;'9999';8
36AE_GN36199402802;00010;C;6/1/1994;06:30;22.5;36.25;2386;14;0;18.401;0;38.808;0;'9999';8
……………………………………………
36AE_GN36199402802;00020;C;6/1/1994;12:00;22.5;36;3100;1;0;20.924;0;38.556;0;'9999';8
36AE_GN36199402802;00020;C;6/1/1994;12:00;22.5;36;3100;2;0;20.871;0;38.547;0;'9999';8
36AE_GN36199402802;00020;C;6/1/1994;12:00;22.5;36;3100;3;0;20.888;0;38.536;0;'9999';8
36AE_GN36199402802;00020;C;6/1/1994;12:00;22.5;36;3100;4;0;20.836;0;38.541;0;'9999';8
36AE_GN36199402802;00020;C;6/1/1994;12:00;22.5;36;3100;5;0;20.772;0;38.541;0;'9999';8
36AE_GN36199402802;00020;C;6/1/1994;12:00;22.5;36;3100;6;0;20.674;0;38.546;0;'9999';8
36AE_GN36199402802;00020;C;6/1/1994;12:00;22.5;36;3100;7;0;20.597;0;38.539;0;'9999';8
36AE_GN36199402802;00020;C;6/1/1994;12:00;22.5;36;3100;8;0;20.527;0;38.538;0;'9999';8
……………………………………………
15
First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium
Example of a Current Meter time series
/ Data from: C:/sissy/ODV/tests/New_ODV_Collection2 (Feb/09/2007 12:12:51)
// Extracted: Feb/09/2007 12:14:15 by Iona@SISSY
Cruise;;;Type;mon/day/yr;hh:mm;Longitude [degrees_east];Latitude [degrees_north];Bot. Depth [m];Time [years since 1900-01-01
00:00];QF;Year;QF;Month;QF;Day;QF;Time [hhmmss];QF;Current East Component [m/s];QF;Current North Component [m/s];QF;Horizontal Current Speed
[m/s];QF;Direction rel. true North [degree];QF
36AE_GN36199724603;M1N21;C;3/23/1997;08:00;24.6;40.0725;500;97.2228;1;1997;0;3;0;23;0;80000;0;0.009;0;-0.015;0;0.017;0;149.4;0
36AE_GN36199724603;M1N21;C;3/23/1997;08:00;24.6;40.0725;500;97.2229;1;1997;0;3;0;23;0;83000;0;0.008;0;-0.012;0;0.014;0;146.3;0
36AE_GN36199724603;M1N21;C;3/23/1997;08:00;24.6;40.0725;500;97.2229;1;1997;0;3;0;23;0;90000;0;0.013;0;-0.011;0;0.017;0;128.8;0
36AE_GN36199724603;M1N21;C;3/23/1997;08:00;24.6;40.0725;500;97.223;1;1997;0;3;0;23;0;93000;0;0.011;0;-0.009;0;0.014;0;128.8;0
36AE_GN36199724603;M1N21;C;3/23/1997;08:00;24.6;40.0725;500;97.2231;1;1997;0;3;0;23;0;100000;0;0.011;0;-0.013;0;0.017;0;137.5;0
36AE_GN36199724603;M1N21;C;3/23/1997;08:00;24.6;40.0725;500;97.2231;1;1997;0;3;0;23;0;103000;0;0;0;-0.014;0;0.014;0;180.6;0
36AE_GN36199724603;M1N21;C;3/23/1997;08:00;24.6;40.0725;500;97.2232;1;1997;0;3;0;23;0;110000;0;0.005;0;-0.013;0;0.014;0;157.1;0
36AE_GN36199724603;M1N21;C;3/23/1997;08:00;24.6;40.0725;500;97.2232;1;1997;0;3;0;23;0;113000;0;0.008;0;-0.012;0;0.014;0;146.9;0
36AE_GN36199724603;M1N21;C;3/23/1997;08:00;24.6;40.0725;500;97.2233;1;1997;0;3;0;23;0;120000;0;0.01;0;-0.014;0;0.017;0;144.5;0
36AE_GN36199724603;M1N21;C;3/23/1997;08:00;24.6;40.0725;500;97.2233;1;1997;0;3;0;23;0;123000;0;0.011;0;-0.008;0;0.014;0;125.6;0
36AE_GN36199724603;M1N21;C;3/23/1997;08:00;24.6;40.0725;500;97.2234;1;1997;0;3;0;23;0;130000;0;0.007;0;-0.012;0;0.014;0;151.5;0
36AE_GN36199724603;M1N21;C;3/23/1997;08:00;24.6;40.0725;500;97.2235;1;1997;0;3;0;23;0;133000;0;0.007;0;-0.012;0;0.014;0;148;0
36AE_GN36199724603;M1N21;C;3/23/1997;08:00;24.6;40.0725;500;97.2235;1;1997;0;3;0;23;0;140000;0;0.013;0;-0.006;0;0.014;0;116.5;0
36AE_GN36199724603;M1N21;C;3/23/1997;08:00;24.6;40.0725;500;97.2236;1;1997;0;3;0;23;0;143000;0;0.022;0;-0.007;0;0.023;0;107;0
36AE_GN36199724603;M1N21;C;3/23/1997;08:00;24.6;40.0725;500;97.2236;1;1997;0;3;0;23;0;150000;0;0.027;0;-0.005;0;0.028;0;101.1;0
36AE_GN36199724603;M1N21;C;3/23/1997;08:00;24.6;40.0725;500;97.2237;1;1997;0;3;0;23;0;153000;0;0.03;0;-0.009;0;0.031;0;106;0
16
First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium
MEDATLAS format
MEDATLAS format designed by the MEDATLAS and MODB consortia in conformity
with the ICES/IOC GETADE recommendations in the frame of the EU MAST II
programme.
It is an auto-descriptive ASCII format.
Main characteristics of MEDATLAS format:
- facilitate the reading of the data (and not to optimize the data archiving neither to
speed up the data processing).
- is independent of the computer.
- is flexible and accept (almost) any number of different parameters.
- keeps track of the history of the data including the data collection and the processing.
- allows the processing of each profile independently. Therefore the date, time and
geographical co-ordinate must be reported on each profile header (and not in separate
files)
-real (floating) numbers must remain in the same way as they have been transmitted,
not reformatted into integers. The number of decimals must implicity indicate the
accuracy of the measurements
17
First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium
Data and Metadata Structure
A MEDATLAS file includes:
- data from the same cruise
- data measured with the same instrument (CTD, Bottle, Current Meter, etc)
A MEDATLAS file consists of three parts:
a cruise header based on the international ROSCOP information
a station header including the cruise reference, the originator station reference within
the cruise, date, location, list of observed parameters with units
the data of the station
(example of a time series data file)
The sequence ‘station header + data records' is repeated for each profile.
The archived parameters are the observed. Derived parameters (eg. density) are not
archived.
Detailed description of the format and the main fields in the “Medar-Medatlas Protocol,
Part I: Exchange format and quality checks for observed profiles”
18
First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium
MEDATLAS format identifiers
Cruise reference (unique):
FI35199745003 (String of 13 Characters, No blanks, ‘0’ instead)
FI
data centre code
35
GF3 country code of the data source
1997
year of the beginning of the cruise
45003
assigned to the cruise by the data centre
Station reference (unique):
FI3519974500300011 (String of 18 Characters, No blanks, ‘0’ instead)
FI35199745003
0001
cruise reference
station name
1 cast number
21
First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium
Reference codes
Reference codes are used to describe the metadata:
1. IOC/GF3 Country codes (http://www.ices.dk/ocean/codes/Country.htm)
2. ICES Ship codes (http://www.ices.dk/ocean/codes/shipcodes.htm)
3. Extended GF3 codes for parameters and units (UNESCO/IOC/IODE, 1987 : GF3 A
General Formatting System for Geo-Referenced Data - Manual and Guides 17)
4. IOC/ROSCOP codes for data types (http://www.ices.dk/ocean/roscop/par-cod.htm)
5. Mediterranean and Black Seas region from International Hydrographic Bureau
definitions (I.H.B.) (http://www.ices.dk/Ocean/codes/ihb.htm)
6. MEDAR/MEDATLAS Data Centres and Confidentiality Codes (MEDAR/MEDATLAS
protocol).
7. Quality flags
22
First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium
References/links
-
“Climate and forecast (CF) metadata convention” by J. Gregory, Univ. of Reading, Met Office Hadley
Centre, UK
-
Coriolis data-center, Volume 2/2, V1.2, 2001
-
Jonathan Gregory, 2003, “The CF metadata standard”
-
“Medar-Medatlas Protocol, Part I: Exchange format and quality checks for observed profiles”, V3, 2001
-
NetCDF Users Guide, V3.6.1, 2006
-
OceanSITES User’s Manual, V1.0, 2006
-
Schlitzer, R., Ocean Data View, http://odv.awi.de, 2006
-
http://badc.nerc.ac.uk
-
http://www.cgd.ucar.edu/cms/eaton/cf-metadata/
-
http://ferret.wrc.noaa.gov/noaa_coop/coop_cdf_profile.html
-
http://www.ices.dk/ocean
-
http://www.unidata.ucar.edu/software/netcdf
26
First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium