Transcript Title
www.seadatanet.org
SeaDataNet mid-term review – Bologna – September 2008, 19 OBSERVATIONS & PR ÉVISIONS CÔTIÈRES
SeaDataNet
Pan-European Infrastructure for Ocean and Marine Data Management
Gilbert Maudire and the SeaDataNet Consortium
OBSERVATIONS & PR ÉVISIONS CÔTIÈRES SeaDataNet mid-term review – Bologna – September 2008, 19
Data is of vital importance for marine research and support to public decision
Marine observation and data sets status
o observation conducted by more than 1100 active scientific laboratories in the countries of partners, using various sensors and platforms o not always quality controlled using comparable procedures o not always easily accessible
Necessity of large background work in order to :
secure them in long term archives document available observations and data sets (metadata) improve coherence of data sets (references, quality, redundancy) make data sets accessible easily (formats, on line distribution) 2
www.seadatanet.org
OBSERVATIONS & PR ÉVISIONS CÔTIÈRES SeaDataNet mid-term review – Bologna – September 2008, 19
What is SeaDataNet?
EU Sixth Framework Programme
Integrated Research Infrastructure Initiative (I3)
1 April 2006 to 31 March 2011 (5 years)
Objectives
o To
network existing oceanographic data centres
already nationally funded o To develop an
efficient distributed pan-European marine data management Infrastructure
(a “unique Virtual Data Centre”) o To
provide on-line access to distributed databases of standardised quality
by using adapted communication & information technology
This project integrates and develops several EU past initiatives and international cooperation, in particular Medar/Medatlas, Edmed, Edios, Sea-Search etc.
3
www.seadatanet.org
OBSERVATIONS & PR ÉVISIONS CÔTIÈRES SeaDataNet mid-term review – Bologna – September 2008, 19
49 Partners
National Oceanographic Data Centres
o divisions of major national marine research institutes in 35 countries o experienced as leaders of data Management of many scientific projects and of concerted actions (e.g. MTPII-MATER, CANIGO, OMEX, EDIOS, MERSEA, MEDAR/MEDATLAS, SEA SEARCH)
3 satellite data centres
3 scientific modelling centres
2 SMEs experts in software development
3 International organisations with strong regional interest/cooperation
4
www.seadatanet.org
OBSERVATIONS & PR ÉVISIONS CÔTIÈRES SeaDataNet mid-term review – Bologna – September 2008, 19
SeaDataNet managed data
Primary focus on water column parameters
o Physical o Chemical o Bio-chemical (non species dependant)
Projects will be proposed for other domains by involved communities in strong relationships with SeaDataNet
Several SeaDataNet partners are already involved in those domains o Geophysical and geological data o Biology (species dependant) 5
www.seadatanet.org
OBSERVATIONS & PR ÉVISIONS CÔTIÈRES SeaDataNet mid-term review – Bologna – September 2008, 19
SeaDataNet services
Online services : web portal and web services
o o o
Discovery services
• Metadata directories • Common vocabularies : parameters, units, …
Delivery services
• Security services : Authentication, Authorization & Administration (AAA) • Data download and access including reformatting in standardized formats
Viewing services
• Web map services, data visualization.
Offline services : « back office » services
o o o o
Safeguarding within the national repositories (NODCs) Quality Control and desktop viewing services
• ODV analysis software
Product generation adapted to each regions Monitoring and statistics : usage & system performance www.seadatanet.org
6
OBSERVATIONS & PR ÉVISIONS CÔTIÈRES SeaDataNet mid-term review – Bologna – September 2008, 19
Infrastructure versions & planning
Version 0 – 2006-2007
Continuation and maintenance of existing Sea-Search system :
the data access needs several different requests to each data centres and the data sets are delivered in different formats o
Operated at the start of the project Version 1 – 2008-2010 Setup of the integrated online data services to users :
networking of 10 “interoperable” data centres of the Technical Task Team
unique request
to the interconnected data centres and the data sets are delivered with a
unique format
o
Presently under test and progressive integration of 10 data centres during 2008
Version 2 – 2010-2011 Improved version of the system (ease of use, data visualisation, data access) and by step, all data centres connected for providing online trans-national access to the data, metadata and products
o
Operational in 2010 www.seadatanet.org
7
OBSERVATIONS & PR ÉVISIONS CÔTIÈRES SeaDataNet mid-term review – Bologna – September 2008, 19
Discovery services
• • •
All catalogues (European metadata directories)
are compliant with ISO19115 standard family for spatial metadata make use of common vocabularies and references are accessible and linked through web services • ease data access using various selection criterions adapted to different users (scientists, observatory and fleet managers, …) 8
www.seadatanet.org
OBSERVATIONS & PR ÉVISIONS CÔTIÈRES SeaDataNet mid-term review – Bologna – September 2008, 19
Common data index : gate to data access
online searchable directory where?, when?, what?, …
tool for managing user’s “Shopping Basket”, that is :
o
To prepare a list of distributed datasets matching user’s request
o o
To ask concerned data centres to serve the user, To display the status of user’s request
o
To route the requested data from the distributed data centres to the user
9
www.seadatanet.org
OBSERVATIONS & PR ÉVISIONS CÔTIÈRES SeaDataNet mid-term review – Bologna – September 2008, 19
SeaDataNet data policy and data licence
inspired by INSPIRE Directive to facilitate the data flow takes into account the local rules and the SeaDataNet users needs
Metadata
o
free and open access, no registration required
o each data centre should provide the meta-data in standardized format to populate the catalogue services
Data and products
o o o
visualisation freely available data download, in general, free for academic purposes with mandatory registration
a “
SeaDataNet role”
(partner, academic, commercial etc.) is attributed to individual user using the Authentication, Authorization and Administration (AAA) Service • Each NODC attributes the roles to the users of its of country • Out of the partnership, the roles are assigned by SeaDataNet user-desk o o
a common data licence has been adopted and users must agree on when registering
each data centre node delivers data according to the user’s role and its local regulation o each data centre should provide the data sets necessary to develop the common products 10
www.seadatanet.org
OBSERVATIONS & PR ÉVISIONS CÔTIÈRES SeaDataNet mid-term review – Bologna – September 2008, 19
QC procedures and tools
SeaDataNet protocol :
o Coherent automatic and visual checks are performed on the data by each NODC o A unique quality flag scale has been adopted
The automatic checks and the visualization could be performed using either by Ocean Data View (ODV) or existing partner software
(if the same protocol and checks are applied)
Example: Using derivatives for spike & discontinuity detection in ODV www.seadatanet.org
11
OBSERVATIONS & PR ÉVISIONS CÔTIÈRES SeaDataNet mid-term review – Bologna – September 2008, 19
Data products
SeaDataNet develops added-value products at regional scales:
o Mediterranean Sea o Black Sea o Baltic Sea o Arctic Seas o Atlantic and global ocean
The preparation of these products allows the checking of the compatibility of the QCs performed on the data in each NODC.
First products => gridded climatologies and trends
The comparability of the different regional products is assessed by using a new version of DIVA (Data Inverse Variational Analysis) developed (University of Liege) www.seadatanet.org
12
OBSERVATIONS & PR ÉVISIONS CÔTIÈRES SeaDataNet mid-term review – Bologna – September 2008, 19
Capacity Building is a main issue
Training program main objectives
to
ensure that management procedures are coherent
within the consortium to transfer expertise and to train
IT experts
of the SeaDataNet data centers in the installation and operation of the SeaDataNet
technical components
13
www.seadatanet.org
OBSERVATIONS & PR ÉVISIONS CÔTIÈRES SeaDataNet mid-term review – Bologna – September 2008, 19
www.seadatanet.org
Please contact : [email protected]
www.seadatanet.org
14
www.seadatanet.org
SeaDataNet mid-term review – Bologna – September 2008, 19 OBSERVATIONS & PR ÉVISIONS CÔTIÈRES
SeaDataNet
Project management
OBSERVATIONS & PR ÉVISIONS CÔTIÈRES SeaDataNet mid-term review – Bologna – September 2008, 19
Role of partners (1/4)
Coordination
o
Ifremer
: project coordinator o
Maris
: technical coordinator animator of the Technical Task Group with the help of : • •
NERC/ BODC
: standard development
AWI :
data quality control and desktop viewing services o
University of Liege
: scientific coordinator in charge of the regional products development with the help of : • • • • •
JRC
: Product harmonization
INGV
: coordinator of Mediterranean products
METU
: coordinator of Black Sea products
NERI
: coordinator of Baltic products
IMR & VLIZ
: coordinators of Artic & North Seas products •
Ifremer
: coordinators of Atlantic & Global Ocean products 16
www.seadatanet.org
OBSERVATIONS & PR ÉVISIONS CÔTIÈRES SeaDataNet mid-term review – Bologna – September 2008, 19
Role of partners (2/4)
Coordination (cont.)
o
HCMR : Coordinator of the networking activities i.e.
metadata and data management, system monitoring with the help for coordinating the distributed network of
•
Maris
: management of Common Data Index (CDI) and of Directories of Marine Projects (EDMERP) and Marine Organizations (EDMO) •
NERC/ BODC
: management of Directories of Marine Environmental Datasets (EDMED) and of Ocean Observing System (EDIOS) •
BSH :
management of Cruise summary reports •
Ifremer
: Management of SeaDataNet users 17
www.seadatanet.org
OBSERVATIONS & PR ÉVISIONS CÔTIÈRES SeaDataNet mid-term review – Bologna – September 2008, 19
Role of partners (3/4)
Technical task team (11 partners : Western Europe + Russia)
o Technical specifications, o Check of the defined standards o Integration and system setup, test and validation of technical components o Running of System V1
Capacity building and training
o
UNESCO/ IOC Project office
with the help of Technical Task Team
Information and communication
o
ENEA
18
www.seadatanet.org
OBSERVATIONS & PR ÉVISIONS CÔTIÈRES SeaDataNet mid-term review – Bologna – September 2008, 19
Role of partners (4/4)
All partners which are mostly NODC’s (except international organizations) have to
o generate metadata : organizations, projects, cruises, observing systems, data sets o manage data from their country : quality, format, … according to SeaDataNet agreed standards o provide access to their data repository, automatically using SeaDataNet system or manually as a first stage o manage users of their respective countries by providing information and help 19
www.seadatanet.org
OBSERVATIONS & PR ÉVISIONS CÔTIÈRES SeaDataNet mid-term review – Bologna – September 2008, 19
External relationships / Collaborations
Advisory board
o
MEDS Canada
(represents IODE/WMO JCOMM, GTSPP and Argo international programs) o
US – NODC
(represents North America initiatives like DMAC) o
EuroGOOS
(represents Operational Oceanography community)
Standard summit (last January on behalf of COI)
o North America, Europe, Japan
“Memorandum of understanding” with other consortiums
o Unidata, Mersea-MyOcean (GMES), EuroBis-Marbef (FP6), GENESI (FP7)
International conference on Marine Data and Information Systems (IMDIS)
o April 2008, more than 200 participants from about 50 countries.
20
www.seadatanet.org
OBSERVATIONS & PR ÉVISIONS CÔTIÈRES SeaDataNet mid-term review – Bologna – September 2008, 19
Project management tools
SeaDataNet web site
www.seadatanet.org
o Distributed management, based on a Content Management system (EZPublish) o General information : Description of Work, Meetings, Conferences, Education (several languages) o
Project services (discovery, data access, …)
o Finalised documents (applicable version of standards) and software tools
Mailing lists
o Based on Sympa software o One list per group (technical, …) owned by the chair person o + user desk (SeaDataNet users point of contact)
Document management
o Based on BSCW software
(Basic Support for Cooperative Work - Germany)
o Cooperative work on documents (non finalised versions) and software
Software management (versionning tool)
o Based on
GForge
21
www.seadatanet.org
OBSERVATIONS & PR ÉVISIONS CÔTIÈRES SeaDataNet mid-term review – Bologna – September 2008, 19
System monitoring
focused on system “availability”
Availability of SeaDataNet user services
o availability of web pages and web services general information, metadata browsing (EDMED, CSR, CDI), data selection users’ interfaces, web services o availability of web data access services data centre download services o Service
“available” if monitoring system get an answer
•
Search of keywords within web pages,
•
Web services can be reached : WSDL returned
Monitoring
o done by IFREMER (coordinator) and HCMR in order to avoid biases due to network troubles o Using
NAGIOS
Open Source software • Polling of services (web pages, web services) 22
www.seadatanet.org
OBSERVATIONS & PR ÉVISIONS CÔTIÈRES SeaDataNet mid-term review – Bologna – September 2008, 19
Polling of system components
BODC ENEA HCMR Ifremer OGS BSH Maris Type(*) Name Address (URL)
Web page (static) Web page (dynamic content) Web page (dynamic content) Web page (dynamic content) BODC home page http://www.bodc.ac.uk/ http://www.bodc.ac.uk/cgi-bin/dbcheck http://www.bodc.ac.uk/data/online_delivery/gebco/ http://www.bodc.ac.uk/maps/ http User Interface MOON-VOS Oceanographic database http://moon.santateresa.enea.it
http://hnodc-data.hcmr.gr
Web page + Nautilus User interface Web page +User interface Coriolis http://www.ifremer.fr/nautilus http://www.coriolis.eu.org
http://nodc.ogs.trieste.it/ Web page (static) NODC portal Home page DOD Home page http://www.bsh.de/de/Meeresdaten/Beobachtungen/DOD Datenzentrum/index.jsp
Web site (static pages) Sea-Search portal http://www.sea-search.net
Comment (**)
Apache web server check (HEAD) Oracle database availability check (via Perl) (GET) Tomcat availability check (HEAD) Map server availability check (HEAD) Will be integrated into SeaDataNet portal soon/ count only on static pages 23
www.seadatanet.org
OBSERVATIONS & PR ÉVISIONS CÔTIÈRES SeaDataNet mid-term review – Bologna – September 2008, 19
Monitoring board
Example of one-month report for Ifremer http access
Electrical failure during 6 hours 24
www.seadatanet.org
SeaDataNet mid-term review – Bologna – September 2008, 19 OBSERVATIONS & PR ÉVISIONS CÔTIÈRES
SeaDataNet
Administrative and financial aspects
www.seadatanet.org
OBSERVATIONS & PR ÉVISIONS CÔTIÈRES SeaDataNet mid-term review – Bologna – September 2008, 19
Known issues
IOC Project Office partner
o In charge of training courses and capacity building o Report at 0 up to now even if work done (3 training courses organised yet) no agreement on US $ / € change rate between UNESCO and EC) o IOC is an « Additional Cost partner » not well adapted to training course organization (additional personnel)
Transnational activities report
o Partners with very small web site and/or very little audience o Audience measurment : KiloPageViews unit difficult to report accurately o 80% of partners report less than expected for TA
www.seadatanet.org
26
OBSERVATIONS & PR ÉVISIONS CÔTIÈRES SeaDataNet mid-term review – Bologna – September 2008, 19
Transnational Activities amendment
Change reporting unit from KiloPageViews to KiloHits
o To take into account all hits on online services : web pages (for any usages), web services, ftp servers, … o But, cost / unit must decrease (Project Officer comment)
However, most audiences will remain far less than expected
o Cost are not linearly proportional to the audience « Entry costs » to set up a web site funds transfer to JRA (web site maintenance) o Who? How much?
All must remain in the same budget envelope
27
www.seadatanet.org
www.seadatanet.org
SeaDataNet mid-term review – Bologna – September 2008, 19 OBSERVATIONS & PR ÉVISIONS CÔTIÈRES
SeaDataNet
Planning
OBSERVATIONS & PR ÉVISIONS CÔTIÈRES SeaDataNet mid-term review – Bologna – September 2008, 19
Infrastructure versions & planning
Version 0 – 2006-2007
Continuation and maintenance of existing Sea-Search system :
the data access needs several different requests to each data centres and the data sets are delivered in different formats o
Operated at the start of the project Version 1 – 2008-2010 Setup of the integrated online data services to users :
networking of 10 “interoperable” data centres of the Technical Task Team
unique request
to the interconnected data centres and the data sets are delivered with a
unique format
o
Presently under test and progressive integration of 10 data centres during 2008
Version 2 – 2010-2011 Improved version of the system (ease of use, data visualisation, data access) and by step, all data centres connected for providing online trans-national access to the data, metadata and products
o
Operational by 2010 www.seadatanet.org
29
OBSERVATIONS & PR ÉVISIONS CÔTIÈRES SeaDataNet mid-term review – Bologna – September 2008, 19
Project planning and budget
High level of effort during the first three years
(80% of the budget)
o Common standard adoption, set up and training, o Development / Adaptation of common tools o
System specifications, development and setup (V1 and then V2)
o
Development of first release of regional products
and initial harmonization
Lower level of effort during the last two years
o Continuous data management o
Exploitation
of the system, system maintenance and upgrade o
Upgrade
of regional products
www.seadatanet.org
30
OBSERVATIONS & PR ÉVISIONS CÔTIÈRES SeaDataNet mid-term review – Bologna – September 2008, 19
More information about budget
Funds breakdown / year / partner after second report
o Funds breakdown :
https://www.ifremer.fr/bscw/bscw.cgi/0/110042
Transnational access audience
o
Evolution for TA- Seadatanet.xls
31
www.seadatanet.org