Transcript WP5_Presentation_A.Crise

JERICO WP5
Data management and distribution
Alessandro Crise
Istituto Nazionale di Oceanografia e di Geofisica Sperimentale
JERICO Project Kick-Off meeting
Maison de la Recherche
Paris, 24-25 May 2011
JERICO WP5
Data management and distribution: main goal
The main goal is to design the JERICO Data Flow and
Management infrastructure based on the
implementation of an end-to-end ‘open and free’
conduit from data to users and upon the guidelines
proposed in the GOOS Report 148 (2005).
The implementation will be consistent and
complementary with the major European initiatives for
the establishment and coordination of infrastructures
for the management and distribution of data and
products in Europe which can use JERICO data
(SeaDataNet, EuroGOOS Regional Centres, MyOcean).
YET ANOTHER DATA MANAGEMENT SYSTEM?: NO!
WP5 Structure
JERICO will need to ensure that the flow of
real-time and delayed mode will be reliable,
accessible and easy to distribute.
The WP5 structure:
• Creating value for measured data: from raw
data issued from the sensors to validated data
stored and marked.
• Facilitating data and meta-data flow to
existing data repository infrastructures
(SeaDataNet);
• Helping in delivering real-time JERICO data to
ROOSs and My Ocean
Task 5.1: Quality assurance: create value for measured data
Task responsible:
OGS
Partners:
HCMR
Duration
M1-M42
• This task is aimed at adding value to
observations by associating them with
characteristic uncertainties.
• By definition, the “expected” accuracy
associated with an observation is an
attribute of the measurement chain that
produces it.
Task 5.1: Quality assurance objectives
The objectives are:
Collecting pre-existing knowledge regarding
the key identified sensors and critically revise
state-of-art literature on in lab and in situ
performances of the most popular sensors (in
conjunction with WP4)
defining procedures for ascribing a total
uncertainty to measurements that will account
for the actual circumstances and conditions
under which the measurements were taken.
Testing and verifying uncertainty computational
procedures
Task 5.1: Quality assurance methodologies
. Methodology
A novel model of a-priori uncertainty will be proposed as a
sum of fixed and time-dependent source of error and
uncertainty (precision and accuracy, drifts, experimental
uncertainties, the representativity error, etc.)
The variables temperature, salinity and
fluorescence/chlorophyll will be considered at first.
Controlled experiments will be performed in order to
document and validate the different uncertainty components
composing the final uncertainty estimates.
All the relevant assessments carried out in WP4 will be
included as basic information for the combined uncertainty
estimation.
WP4.1 will provide an updated summary of these
achievements to WP5.1, no later than the dates set down in
the WP4 D4.2 delivery schedule (M14)
Task 5.2: Harmonization of Delayed Mode data management
procedures with SEADATANET
Task responsible: IFREMER,
Partners:
HCMR, MUMM, OGS
Duration:
M1-M42
• This task will define, establish and oversee
the data management infrastructure for
dealing with delayed-mode data in JERICO.
• The goal is to provide JERICO users with
relevant informations for seamless data
integration with the EU SEADATANET
initiative.
Task 5.2 Methodology
-
-
-
-
-
-
-
Survey the existing delayed-mode data handling practices of JERICO
partners.
Formulate a viable proposal for a common JERICO delayed-mode data
management platform that can serve to reinforce the SEADATANET
effort;
the guidelines established by DG-Mare/EMODNET and WISE-Marine
will be
also taken in account when doing this.
Create common vocabularies for JERICO delayed-mode data formats
and meta-databases, building upon the work that has already been
done in SEADATANET.
Implement delayed-mode data and metadata formats that are
compatible with those of SEADATANET (in compliance with the EU
INSPIRE Directive);
the Sensor Web Enablement family of the OGC consortium family of
standards (SensorML, Sensor Registry, O&M) will be considered in the
implementation.
Ensure easy sharing and secure archival of delayed-mode data within
JERICO by employing common data transport formats and storage
criteria making use of the experience gained by SEADATANET.
Task 5.2 Metadata implementation
Discovery metadata
To let all know what have been deployed during the
project (where, what, how, who…)
To help to retrieve data related to observatories
To be compliant with European directives (Inspire)
Enabling technology
Use of SeaDataNet EDIOS directory to describe
observatories
Typing / Generation of metadata using SeaDataNet
procedures and tools (Mikado) complemented with
automatic tools developed in Emodnet-Physics to
better use the RT Netcdf files provided in RT by the
ROOSs
Task 5.2 Data implementation
Delayed mode data
Delayed mode quality check
Uniform file format (based on OceanSites NetCDF, compatibility
with CF convention)
+ distribution
in ASCII flat file format (Ocean Data View)
Full metadata (Common Data Index) generated by Mikado
for long term reuse
Data access via web portal (all users)
Task 5.3 Harmonization of Real Time data anagement
procedures with MyOcean and EuroGOOS & the institution of
data access services for JERICO TOP activities
Task responsible: IFREMER,
Partners: CNR, NIVA, IMR, HCMR,PUERTO, SMHI
Duration: M1-M42
• This task will define, establish and oversee the
data management infrastructure for dealing with
real-time data in JERICO.
• The goal is to provide for seamless integration
with the EuroGOOS and EU Marine Core Service’s
MyOcean component and in order to render
JERICO real-time data accessible for use in
generating downstream services and products.
Task 5.3 Methodology
•
Survey the existing real-time data handling practices of JERICO
partners.
•
Formulate a viable proposal for a common JERICO real-time data
management platform that can serve to reinforce EuroGOOS and
MyOcean
•
Create common vocabularies for JERICO real-time data formats and
metadatabases, based on any existing
•
recommendations by MyOcean/EuroGOOS for doing this.
•
Implement real-time data and metadata formats based on any existing
recommendations by EuroGOOS and MyOcean for doing this.
•
Reconcile, wherever possible, the data quality assurance
procedures/protocols for real-time data amongst the JERICO partners
(link to WP4, WP5.1).
•
Ensure easy exchange of real-time data within JERICO by employing
common data transport formats and storage criteria, based on any
existing recommendations by EuroGOOS/MyOcean for doing this.
WP5.3 Implementation
Real-Time data
Real-Time Quality check based on
EuroGOOS/MyOcean procedure that would need to
be extended to new parameters whenever possible
Uniform file format (based on OceanSites NetCDF,
compatibility with CF convention)
ftp file server (with an index file) per region
Oceanotron (OpenDap protocol) for direct access by
remote software
WP5 in JERICO Data flow
WP5 act as mediator in JERICO Data Flow Model
A priori quality assurance (WP4)
Data management procedures (WP5)
• Time-dependent data uncertainty estimation
• Real-time data management
• Delayed mode data management
Value added products and data dissemination (WP6)
EuroGOOS ROOSs engagement (WP2)
Demonstation of data provision (WP7)
SeaDataNet
MyOcean
Emodnet Physical Parameters
EuroGOOS
MyOcean Insitu TAC implementation of
EuroGOOS regional approach
Arctic: IMR / Norway
All parameters
North West Shelves: BSH/Germany
•Temp & Salinity : BSH/Germany
•Current: SMHI/Sweden
•Sea Level: DMI/Denmark
Global Ocean :
Coriolis/France
•Link with international
network s:
Coriolis/France
•Argo:
Coriolis,BODC,BSH
•European
Vessels:
South West
Shelves: Puertos Del
NIVA/Norway
Estado/Spain
•Mooring : PdE
•Underway data: Coriolis/France
Baltic Sea: SMHI/Sweden
•Temp & Salinity : BSH/Germany
•Current: SMHI/Sweden
•Sea Level: DMI/Denmark
•Bio : FMI-Syke/Finland
Black Sea
IOBAS/Bulgaria
All parameters
Mediterranean Sea :
HCMR/Greece
•Mooring : HCMR/Greece
•XBT/CTD: ENEA/Italy
•Drifter & Argo: OGS/Italy
•Glider and Argo:
MyOcean In-situ TAC role
Limited number of parameters:
Physical: T&S, current, sea level
Biogeochemical: Chlorophyll/Fluorescence, Oxygen,
Nutrients
Integrate in-situ data in product accessible through
global and regional portals
Common format
Common NRT QC
Common Quality flags
Common distribution tools
Single access point
Ensure a minimum level of quality on the data delivered
In Near real time ( 24h to a week)
In delayed mode
Assessment of the products at basin scale
EMODNET PHYSICAL PARAMETERS
The European Commission, represented by DG
MARE, concluded a service contracts aiming to:
Create a prototype components of the European
Marine Observation and Data Network
(EMODnet).
Provide access to archived and real-time data on
physical variables in Europe's seas and oceans
determine how well the data meet the users
needs.
THE EMODNET PP PILLARS
SeaDataNet
JERICO WP5
EuroGOOS
MyOcean
EMODnet PP
EMODnet PP aims to bridge
the communities behind
MyOcean, EuroGOOS and
SeaDataNet in order to
strengthen their position
inside the Commission
agenda on IS and expand in
future the participation to all
partners
EuroGOOS SEPRISE network
NEW FEATURES
Add new stations and models
Add profiles
Make use of the results from ECOOP
Add data from FerryBox and RV
Common quality control
Link to EDIOS
SEPRISE data in SDN ODV format
Put data on GTS
E-HYPE
Cooperation between MyOcean in-situ TAC and SeaDataNet
MoU signed Beginning 2010
Operational
users
Marine governmental agencies and
research institutes (including NODCs)
MyOcean in-situ TAC
Direct users
real-time
Re-analysis
users
Observing
systems
Data integration
(QC, « best copy »,
archival, re-analysis
datasets
Vocabularies
Metadata (EDIOS,
CDI, CAMIOON,
SensorML)
Technical WG
Metadata
(e.g. EDIOS)
Other
observations
(e.g. scientific
cruises…)
Yearly regional products
(climatologies, datasets)
SeaDataNet
International organisations
Direct users
(IOC/IODE, ICES, ICSU World Data System…)
Users
JERICO role in the European Data Management landscape
Operational
users
JERICO Coastal
MyOcean in-situ TAC
Observatory Network (WP7)
Re-analysis
users
Observing
Network
‘EMECOlike’ Web –
enabled
Datatools
Information
(QC, « best copy »,
archival, re-analysis
datasets
Vocabularies
Metadata (EDIOS,
CDI, CAMIOON,
SensorML)
Technical WG
‘EMECO’ like
& evidence pipeline
(WP6)
Assessment
Web
database
Data integration
Yearly regional products
(climatologies, datasets)
Portal
delayed-mode
SeaDataNet
International organisations
Direct users
Data
XML, CSV
(IOC/IODE, ICES, ICSU World Data System…)
Users
WP5 Milestones/Deliverables
ACTIONS
To propose a formal agreement with SDN,
EuroGOOS/MyO for JERICO data provision, to
ensure more visibility to JERICO project
To find an agreement with DG
MARE/EMODNET for the availability to JERICO
of the developments achieved/to be achieved
in EMODNET Physical Parameter lot.
Thanks!