SeaDataNet tools NEMO, MIKADO, VALIDATOR

Download Report

Transcript SeaDataNet tools NEMO, MIKADO, VALIDATOR

Training course – Ostende – 2-6 July 2012

SeaDataNet tools NEMO, MIKADO, VALIDATOR

M. Fichaut, IFREMER, France

Training course – Ostende – 2-6 July 2012

SDN tools directory

• On your desktop, SDN tools directory :

\presentations\SDN_tools.pps

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

Overview

• • • • SeaDataNet tools in their context

NEMO

: Reformatting tool –

Practical session : reformatting a set of stations

NEMO

and SDN

Download Manager and MIKADO – CDI summary file

–

coupling table, NEMO

–

Practical session : generation of a CDI summary file and of a coupling table

MIKADO

between : XML files generator,

Web validation service,

Link

MIKADO

and SDN

Download Manager

–

coupling table

–

Practical session : 1. Automatic creation of XML files for CDI, Web validation of these files, creation of the corresponding coupling file 2. Manual input for EDIOS program, series and platform

New metadata New data

Training course – Ostende – 2-6 July 2012

XML validator SeaDataNet Vocabulary Metadata in Excel files MIKADO Data in database Metadata In database MIKADO Coupling table Meta data at XML ISO19155 CSR EDIOS EDMED S E A D A T A N E T Data in ASCII file Format X NEMO Collection of ASCII files at SDN Format EDMERP CDI P O R T A L Local copy of data for downloading

[email protected] – www.seadatanet.org

Data centres Data request Downloading AAA RSM European portal

Training course – Ostende – 2-6 July 2012

Data Centre Data in database Data Collection of ASCII files at SDN format SQL Select Download Local ID + manager SQL Select CSR EDIOS EDMED S E A D A T A N E T User Central ID Zip file Data request AAA EDMERP P O R T A CDI Central ID + L ID local + EDMO code Local copy of data for downloading

[email protected]

Zip file Manager (RSM) ODV DIVA

Training course – Ostende – 2-6 July 2012

Survey about the tools



29 answers Do you know the tool?

Yes NEMO MIKADO

4 16

25 13 • • • NEMO : question about a Cruise name field Questions on the coupling table : how to generate it, using which tool?

Those who know, please help those who don’t know

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

NEMO – Reformating software

New metadata New data

Training course – Ostende – 2-6 July 2012

[email protected] – www.seadatanet.org

Data centres : SeaDataNet nodes Data request Downloading AAA RSM European portal

Training course – Ostende – 2-6 July 2012

Objectives



NEMO is a reformatting software used to

generate ASCII files at MEDATLAS or ODV formats which are defined as SeaDataNet formats for data exchange between SeaDataNet partners. Can be downloaded from SeaDataNet Web site :  http://www.seadatanet.org/standards_software/software/nemo It was created in IFREMER for internal purposes, and adapted to take into account the SeaDataNet formats and vocabularies.

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

Technical characteristics

• • • • • • Release : 1.4.4

Written in

Java

Language (Version >= 1.6) Bilingual (French, English) Available under multiple environments : – Microsoft : Windows 2000, WINDOWS 7, XP, VISTA , APPLE – Unix - Solaris – Linux Interactive and batch modes available Use of

SeaDataNet common vocabularies

web services – to update lists of values of the SeaDataNet common vocabularies • need network connections in order to have up to date lists of values.

Training course – Ostende – 2-6 July 2012

NEMO main features

• • Reformat ASCII text file of vertical profiles, time-series or trajectories to a SeaDataNet ASCII format (ODV, MEDATLAS).

•

The input ASCII files can be:

•

one file per station for vertical profiles or time series

•

one file for one cruise for vertical profiles, time series or trajectories

Interact with Mikado, to be able to generate ISO-19115 XML descriptions of the data.

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

NEMO principles

• • • NEMO must be able to read almost

any ASCII format

to translate it to MEDATLAS, ODV Users of NEMO describe the entry files format so that NEMO is able to find the information which is necessary in the SeaDataNet formats One

mandatory

pre-requirement is that in the set of entry files the information about the stations must –

be located at the same position

: same line in the file, same position on the line or same column if CSV format – –

be in the same format

For example : for all the stations the latitude is

• • •

on line 3 on the station header, from character 21 to character 27, or 3 rd column in CSV the format is +DD.ddd

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

Running NEMO

• • 4 or 5 steps to convert the entry files to a SeaDataNet format

Describe the type of file(s) [Describe the cruise, if the files are related to 1 cruise]

NEMO is able to read XML CSR generated with MIKADO

Describe the station information Describe the measured parameters Convert the file

• One more step to

Save the description of the format (Model) [email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

NEMO - Description of entry files (1)

• Where are the files? • Is it a cruise?

– Is it one file per cruise or one file per station ?

• Is it a collection of station files ?

– grouped by cruises or not?

• Are the files with separators?

– Tabulations? Semicolon? Comma ?

• Are they vertical profiles, time series or trajectories data?

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

NEMO - Description of entry files (2)

File to convert is displayed in NEMO window The principle is to describe the input file by selecting the information in the file window

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

Station header : constant number of lines

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

Station header : number of lines not constant

[email protected] – www.seadatanet.org

Cruise description

Can be: Manually input Imported from a SeaDataNet XML CSR Imported from a database (IFREMER only)

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

Station description

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

Station description

Select the date

in the file, in the left part of the screen

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

Station description

Input the date format [email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

Station description

Click on the Set button

to tell NEMO the start and end position of the date

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

Station description

Click on the Test button

to verify your input

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

Station description

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

Data description

Choose the parameter list [email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

Data description

Choose the parameter list Give the code, unit, position , format and input default value

of each parameter

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

Data description

Select « Parameter list »

in the context menu of the measured parameters table (right click to open the menu)

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

Data description

Add all the measured parameters and their unit

(P061) using the context menu

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

Data description

Select the value of the measured parameter

in the file, in the upper left part of the screen

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

Data description

Select « Set start /end »

in the context menu of the measured parameters table

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

Data description

Start and End position

of depth are added in the table

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

Data description

Describe the format and default value of the parameters

(output default value only if output format is MEDATLAS)

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

Data description

Check your input : Select « Update test »

in the context menu of the measured parameters table

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

Data description

The values read by NEMO are displayed in the Test column

of the table. User is able to check that his inputs are correct

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

Data description

Tell NEMO which parameter is the vertical reference [email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

File conversion

Output

• •

formats MEDATLAS

– Multi-station

ODV

– Mono-station – Multi-station

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

Non numeric parameters

For sediment description in Sediment Core Compatible with ODV

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

NEMO in batch mode

• MIKADO can be run in batch mode, from NEMO install directory, using existing models • Several arguments can be added on the command line • Log file in NEMO install directory

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

• • •

Files which can’t be reformatted by NEMO

Binary format : like EXEL, WORD … – File have to be first reformatted to text file Files which do not respect NEMO pre-requirements –

be located at the same position

: same line in the file, same position on the line or same column if CSV format –

be in the same format

Files which are not : –

Vertical profiles

(with depth or pressure as vertical reference) –

Time series

(with time as reference) –

Trajectories

(with latitude, longitude and time as reference)

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

1201 310219590650370-029-330100 2 19660719140058999000000000000080161 013 30029800000258+ + + + 201214 42010 28 0000 6000000036043261501461904 6000090030943320801460604 6000240039043409401465404 6000470054343452101472604 6000710047843449101470404 6000940052043472301472704 6001180059643491301476404 6001410065743509201479404 6001650060343506601477704 6001880051943491601474504 6002120052343496101475104 6002340046543491801473104 1201 31025246064008000060530000 2 19660806190058999000000000000170161 013 30030900000298+ + + + 201214 40213 0000 6000000147043307601503504 6000100133643417301500604 Like for example old historical tape formats 6000250119043467401496504

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

NEMO – User manual

• User manual is provided : File :

sdn_Nemo_UserManual_V1.4.4.pdf

• Detailed explanation for NEMO installation and use, lots of snapshots

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

• • •

NEMO next releases

V1.4.5 (soon) – With correction of some know bugs V1.5.0 (scheduled in autumn) – BODC vocabulary, version 2 • Pre-requirement : Importation tools must manage BODC V2 vocabulary – Include a Cruise name field that can be read in the input file V1.5.1 (not scheduled yet) – Generation of files at NetCDF format – Pre-requirement • NetCDF format final definition

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

[email protected] – www.seadatanet.org

Training Workshop – Ostende – 1-4 March 2010

Training course – Ostende – 2-6 July 2012

Practical work on NEMO

Converting files

M. Fichaut

Training course – Ostende – 2-6 July 2012

From ASCII data to XML files

Collection of ASCII files ODV files XML CDI files Coupling table Export Coupling table summary_CDI_NEMO.xml

Delivered with NEMO CDI summary CSV file

[email protected] – www.seadatanet.org

MIKADO SeaDataNet CDI

Training course – Ostende – 2-6 July 2012

NEMO installation

• Open the install_nemo_1.4.4_windows.zip file (on the desktop) • Unzip it • Double click on launcher_nemo.bat

• Follow the installation procedure • Don’t forget to add the NEMO shortcut on your desktop

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

Collection of ASCII files

•

Training data set

– 6 CTDs raw files under

SDN tools/Practical work/data/input files

– ASCII files with measured parameters in column – Measured parameters = Depth, Temperature, Salinity, Fluorescence and Dissolved oxygen

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

First step : conversion to ODV with NEMO

•

Run NEMO

to convert the files to ODV format.

1. Modify the settings of NEMO

• Default directories for data set to

SDN tools/Practical work/data

– Default directories for data set to

SDN tools/Practical work/models

– Enter your EDMO ID in the field below using the search function (right click)

2. Proceed the 4 steps of NEMO [email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

NEMO – File description

• • • • • • • •

Choose the type of file (cruise directory, cruise file, …) Browse your PC to input the data files directory

Once the file is open in NEMO,

have a look to its content Choose file type :

Profile

Choose the output format :

ODV, one unique file for all stations

One file per station

Enter the Station header information

– there is a specific character string at the end of each Station header :

*END*

The Data termination indicator is

EOF (End of file)

Validate the step when description is OK [email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

• • • • • • • • •

NEMO – Station description

Station number is given on line “**Station” Data type is ‘CTD stations’ Time is U.T.

Latitude format is

NDD MM.mm

Longitude format is

NDDD.MM.mm

Bottom Depth is not available

Input all the mandatory fields :

station number, time, date, latitude, longitude

Use the Test function

to verify your input

Validate the step

when description is OK

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

NEMO – Data description (1)

• •

Select the parameter list to use :

P011 via P021

Add the parameters

: P021 codes are – Depth :

AHGT

Temperature :

TEMP

, Salinity :

PSAL

, Fluorescence :

FVLT,

Dissolved oxygen :

DOXY,

Input the units of the parameters (

for salinity, use Dimensionless

)

–

Add a text label

to each parameter •

Set Start and End

positions for the parameters • Tell NEMO which code is the

vertical reference [email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

NEMO – Data description (2)

• • • •

Format is mandatory

– Describe the output format that you want – %6.1f : means 6 digits total (decimal point included), 1 digit for decimal, f for float (decimal number) – Use the contextual menu ‘Select the format’ for predefined format

Use the test fields to validate your input Validate the step when description is OK

Save you model in the model directory

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

File conversion

• You will be asked to give : • A dataset reference (can be a cruise name, for example) • A

name for the output file

(all the stations in the same file) or

name for the output directory

(one station per file) in the

SDN tools\Practical work\output

directory • Have a look to the output files • Drag and drop the ODV files in ODV4 to test their conformity

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

NEMO and MIKADO interaction – CDI summary file NEMO and SDN Download Manager – coupling table

Training course – Ostende – 2-6 July 2012

While converting …

• • • NEMO is able to

Generate a SeaDataNet CDI Summary

– Text file containing the minimum mandatory information needed in the CDI ISO-19115 description – This file can be converted to an Excel file – Which can be read by MIKADO to generate the XML CDI files

Generate a coupling table

that will be used by the Download manager of SeaDataNet – It is the link between the LOCAL_CDI_ID and the file

[email protected] – www.seadatanet.org

CDI summary

• Further information must be added in NEMO settings (menu Options of NEMO)

Training course – Ostende – 2-6 July 2012

[email protected] – www.seadatanet.org

CDI SUMMARY File

Training course – Ostende – 2-6 July 2012

[email protected] – www.seadatanet.org

Interaction with MIKADO

Training course – Ostende – 2-6 July 2012

Collection of ASCII files ODV files XML CDI files summary_CDI_NEMO.xml

CDI summary CSV file

[email protected] – www.seadatanet.org

MIKADO SeaDataNet CDI

Training course – Ostende – 2-6 July 2012

Coupling table for Download Manager

• Further information must be added in NEMO settings (menu Options of NEMO) – The coupling table makes the link between each LOCAL_CDI_ID and the file(s) which contains it

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

Coupling table

• If the “Generate mapping” option is clicked : • NEMO will insert a record in the coupling table for each vertical profile, time series or trajectory that it converts.

• The coupling table is managed by a local database imbedded in NEMO • The coupling table can be : – Edited (for modification, insertion or deletion) – exported (to be used by the download manager) – imported (from previous version of NEMO, for example)

[email protected] – www.seadatanet.org

Coupling table content (1)

Training course – Ostende – 2-6 July 2012

LOCAL_CDI_ID Modus 1 : mono-station 3 multi-station Filename Date of creation

[email protected] – www.seadatanet.org

Coupling table content (2)

Training course – Ostende – 2-6 July 2012

• • One unique LOCAL_CDI_ID in one mono-station ODV file The same LOCAL_CDI_ID in 2 different files : – One multi-station MEDATLAS file – One mono-station ODV file

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

Practical work on NEMO

Generating CDI summary and coupling table

M. Fichaut

Training course – Ostende – 2-6 July 2012

From ASCII data to XML files

Collection of ASCII files ODV files XML CDI files Coupling table Export Coupling table summary_CDI_NEMO.xml

Delivered with NEMO CDI summary CSV file

[email protected] – www.seadatanet.org

MIKADO SeaDataNet CDI

Training course – Ostende – 2-6 July 2012

•

Creation of the CDI-SUMMARY and the coupling table

Input the information in NEMO SETTINGS – Mapping •

Click

in the box

Generate mapping for SeaDataNet download Manager

– SeaDataNET CDI summary •

Unfold

the SeaDataNet CDI Summary block • •

Click Fulfill

in the box

Generate SeaDataNet CDI summary

all the mandatory fields (in red), using right click for the Search function

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

Run the file conversion

• • • • • • Same file than in the previous exercise

You will be asked to give

: A

dataset reference

(can be a cruise name, for example) A name for the

CDI summary file

name for the output file

(all the stations in the same file) or

name for the output directory

(one station per file) An

output directory prefix

(for the coupling table). This prefix will be subtracted from the file name in the mapping table.

For example if

–

output file name = C:\username\NEMO\cruise_name\file_name

–

and output directory prefix = C:\username\NEMO

–

file name in the mapping table will be : cruise_name\file_name

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

CDI summary file

• Convert it to an EXCEL file (for future use by MIKADO) –

Open the CDI summary text file

delimited by tabulations) with EXCEL (field – –

Rename the spreadsheet into Save it under

XLS or XLSX

STATION

format, it will be used later on during the practical work on MIKADO

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

Coupling table

• •

Have a look

to the coupling table – Menu :

Coupling_table>Edit

– It gives the link between the LOCAL_CDI_ID and the file in which the LOCAL_CDI_ID will be found by the download manager of SeaDataNet – MODUS = 1 if the LOCAL_CDI_ID is in a mono-station file – MODUS = 3 if the LOCAL_CDI_ID is in a multi-station file – One LOCAL_CDI_ID and one format is unique in the coupling table – Add and delete records

Export

the coupling table in a flat file – Menu :

Coupling_table>Export

– Have a look at the flat coupling.txt file

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

MIKADO

Tool for the generation of XML descriptions of SeaDanaNet catalogue records

New metadata New data

Training course – Ostende – 2-6 July 2012

[email protected] – www.seadatanet.org

Data centres: SeaDataNet nodes Data request Downloading AAA RSM European portal

Training course – Ostende – 2-6 July 2012

Objective

    

MIKADO

XML is used to generate XML catalogue descriptions, it creates

ISO-19115

files using

SDN common vocabularies

for metadata   exchange of 

CSR

- Cruise Summary Reports 

EDMED CDI

- Marine Environmental Data sets - Common Data Index

EDMERP

- Marine Environmental Research Projects

EDIOS

– Permanent Ocean-observing System

SensorML and O&M for seismic data Planned Cruise Report (not used in SeaDataNet)

Is freely available on SeaDataNet Web site http://www.seadatanet.org/standards_software/software/mikado

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

Technical characteristics

• • • • • Release 2.3

Written in

Java

Language (Version >= 1.6) Available under multiple environments : – Microsoft : Windows 2000, WINDOWS7, XP, VISTA , APPLE – Unix - Solaris – Linux.

Interactive and batch modes available Use of the

SeaDataNet common vocabularies

web services to update lists of values – needs network connection in order to have up to date lists of values.

Training course – Ostende – 2-6 July 2012

MIKADO main features (1)

• •

MIKADO can be used in 2 different ways

– One manual way, to input manually information for the catalogues and CDI in order to generate XML files.

– One automatic way, to generate XML descriptions automatically, from information catalogued in a relational database or in an Excel file. Automatic way is needed for those who have many entries referenced in a relational database

Only one interface for all catalogues

– Same look – Same principles

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

MIKADO main features (2)

MIKADO Java code Manual Automatic XML files for SeaDataNet catalogues CSR EDMED EDMERP CDI EDIOS DATABASE JDBC Java DataBase Connectivity Native Drivers MYSQL ORACLE POSTGRES SYBASE MSServer Bridge Drivers

using Microsoft ODBC

ACCESS, EXCEL, SQL SERVER, Open Office Other Drivers

Downloaded from ad hoc Websites

EXCEL File

Training course – Ostende – 2-6 July 2012

MIKADO and SDN vocabulary lists (2)

List code C16 C77 C174 C320 C371 C381 L05 L021 L031 EDMERP EDMO List Name List code List Name SeaDataNet Sea Areas ICES ROSCOP data types SeaDataNet CSR ship metadata ISO countries Ten-degree Marsden Squares Ports Gazetteer SeaDataNet device categories SeaDataNet Geospatial Feature Types SeaDataNet Measurement Periodicity Classes European marine projects European marine organisation L061 L071 L081 L101 L111 L181 L231 L241 P021 P081 SeaDataNet Platform Classes SeaDataNet data access mechanisms SeaDataNet Data Access Restriction Policies SeaDataNet geographic co-ordinate reference frames Height and Depth Vertical Co-ordinate Reference Datum ROSCOP sample quantification units SeaDataNet metadata entities SeaDataNet data transport formats BODC Parameter Discovery Vocabulary SeaDataNet Parameter Disciplines [email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

MIKADO and SDN vocabulary lists (2)

Automatic check of the version of the vocabulary lists : once when MIKADO starts – If “On” is clicked in the Vocabulary Update Menu – MIKADO downloads locally the latest version of each list Possible to enable-disable the automatic check – If “Off” is clicked Manual check – Update once now

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

• • • •

MIKADO – Manual input

Available for 5 catalogues : EDMED, CSR, CDI, EDMERP, EDIOS (Programme, series and platform) Each input generates an XML file that can be sent to the central catalogue For EDMERP and CSR : EDMERP CMS and CSR online can also be used, but MIKADO is useful – if you have problems with the NETWORK connection – if you want to keep locally an XML description of your catalogues For EDMED, EDIOS and CDI, there is no online input tools.

[email protected] – www.seadatanet.org

MIKADO – Manual input

Training course – Ostende – 2-6 July 2012

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

MIKADO manual : LOCAL Identifier

• • • This LOCAL ID exists for all the catalogues and is

under the responsibility of the data centre who generates the XML descriptions

The LOCAL ID must be unique for a catalogue and a data centre (EDMO code) The

LOCAL Identifier is vital

because it is kept in the central catalogue and is the entry point to know if the record is new or if it is an update.

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

MIKADO manual – vocabulary lists (1)

• •

Common vocabulary BODC, list C320 (Country ISO codes) [email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

MIKADO manual – vocabulary lists (2)

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

• • • •

MIKADO – Automatic XML generation

Principle

Read the information about CSR, EDMED, EDMERP, EDIOS or CDI in a database or in an Excel file MIKADO has predefined variables which correspond to the XML tags definition for each catalogues MIKADO helps user to write the SQL orders to fulfill these variables with the information available in the database or in the Excel file

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

• • • • •

MIKADO – Automatic XML generation

4 STEPS

Connect to a database or an Excel file and test the connection Write the queries to retrieve information in the database or in the Excel file, test the queries Save the queries in a “Configuration file” Generate the XML files using the “Configuration file”

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

MIKADO – Automatic XML generation

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

MIKADO automatic - connection

• • • Help for the connection to the database Pre-filled information for some databases Check of the connection

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

MIKADO automatic – connection OK

• Green message in the Check box

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

MIKADO – automatic – connection KO

• Red message in the Check box

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

MIKADO automatic – queries

Expendable trees Main query Return the LOCAL ID Single subqueries Return 1 row Multiple subqueries Return 1 to n rows Single an multiple queries related to each LOCAL ID returned by the main query.

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

MIKADO automatic – queries

Write the queries

SQL syntax (for Oracle, Excel, MySQL, …) and SQL variables must be adapted to your own data base

Check the Queries

Green OK Red KO : read the error message

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

MIKADO automatic – single queries

All the XML variables are listed in the expendable tree

• In bold : mandatory fields • 1 to n single subquery can be written • Green ticks: fields already fulfilled • Add or delete variables in a query • Delete a full query • Check the query

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

MIKADO automatic – multiple queries

All the XML variables are listed in the expendable tree

• Number of queries is pre-defined • The list of variables for each of these multiple queries is also pre-defined • In bold : mandatory field • Green ticks : fields already fulfilled

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

MIKADO automatic – multiple queries

• • In a group of variables (same XML block, same pre defined set of variable in a query), the non mandatory variables can be left to null Check all function

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

MIKADO automatic - Save the queries

• • When all the queries are written Saved in an XML file (configuration file) to be re used later on

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

MIKADO automatic - generate the XML files

1. Select the catalogue you want to generate 2. Open the corresponding configuration file 3. Choose the output directory 4. Choose the type of export files 5. Export the XML files • Progress bar • Cancel allowed

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

MIKADO automatic - local mapping

• • • • While generating the XML files for all the catalogues Each time that MIKADO does not recognized a value (entrykey or entryterm) which should come from the common vocabulary, it asks the user for mapping MIKADO manages a demand-driven continuous (incremental) extension of a local mapping : mapping of the local database to the common vocabulary Mapping tables can be modified – Delete rows – Modify the LOCAL value

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

MIKADO automatic - local mapping

Example CSR generation • Mapping of the platform type

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

MIKADO automatic - local mapping

Modification of the local mapping If wrong entries have been input, it is possible to: • Delete one entry • Delete all the entries • Change the local code

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

MIKADO in batch mode

• MIKADO can be run in batch mode using existing configuration files • Several arguments can be added on the command line •

Java –Djava.endorsed.dirs=”dist/lib” –jar dist/Mikado.jar

mikado home=[path] argument2= … argumentn=

• Log file to register the errors

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

• •

Coupling file for Download manager

MIKADO is able to generate this coupling file The coupling file is used by SeaDataNet download manager to make the mapping between a LOCAL_CDI_ID (one profile, one time-series or one trajectory) and – the name of the file containing this LOCAL_CDI_ID (MODUS1 and 3) : if the metadata is in a data base and the data in files or – the SQL Query to retrieve the meta data and the data of this LOCAL_CDI_ID in the local database (MODUS 2)

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

Coupling file for Download manager (2)

• The principle to create this coupling file is the same than to create XML files for catalogue descriptions • User has – to create a configuration file that will be used for the generation of the coupling file.

– to write the queries to retrieve the filename or the data for each LOCAl_CDI_ID • Retrieve the filename: if the data are stored as files • Retrieve the data: if the data are stored in a database

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

Coupling file generation

•

3 STEPS

• Create a configuration file • Connect to a database or an Excel file and test the connection • Write the queries to retrieve information in the database or in the Excel file, test the queries • Save the queries in a “Configuration file” • Generate the coupling table using the “Configuration file”

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

• •

Configuration file for coupling

Create a new one – From nothing – Using a MIKADO CDI configuration file Open an existing one

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

• •

Configuration file for coupling

Connection to the database or excel file Main query to retrieve all LOCAL_CDI_IDs

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

• • • •

Configuration file for coupling: MODUS 2

Data are stored in a database Write the query to retrieve the data in the database – Temperature – Salinity – Oxygene … Test your query Give the name of the mapping file used by DM to generate ODV files

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

• • • • • •

Configuration file for coupling: MODUS 1 or 3

Data are stored in files – Modus 1: on the shelf – Modus 3: to be converted by the Splitter Choose the file format Write the query to retrieve The adress of the file Test your query Preview the coupling file Save your configuration file

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

•

Generation of the coupling table

Using the configuration file for coupling – Select the configuration file – Select the output directory – Select the write mode • Append • Overwrite

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

Generation of the coupling table

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

MIKADO – User manual

• User manual is provided : • File :

SDN_MIKADO_UserManual_V2.3.pdf

• Detailed explanation for MIKADO use, lots of snapshots

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

MIKADO next release: version 3

• ISO 19139 for CDI and CSR – CDI available, to be finalised • BODC vocabulary, version 2 – New services to be included – Backwards compatibility with old XML files : Read of old URN and conversion to new ones • Pre-requirements – Importation tools must be ready for ISO-19139 catalogues and must manage BODC V2 vocabulary

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

Coupling table : NEMO or MIKADO?

• Depending of your data organisation – If data in data base • MIKADO – If metadata in database, and address of the file in a specific table • MIKADO is recommended – If no database, data in flat files generated using NEMO • NEMO

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

[email protected] – www.seadatanet.org

Training Workshop – Ostende – 1-4 March 2010

Training course – Ostende – 2-6 July 2012

XML validation tool

Web service for validation of the XML files generated by MIKADO

New metadata New data

Training course – Ostende – 2-6 July 2012

XML validator SeaDataNet Vocabulary Metadata in Excel files Metadata In MIKADO database Data in database MIKADO Coupling table Meta data at XML ISO19155 CSR EDIOS EDMED S E A D A T A N E T Data in ASCII file Format X NEMO Collection of ASCII files at SDN Format EDMERP CDI P O R T A L Local copy of data for downloading

[email protected] – www.seadatanet.org

Data centre Data request Downloading AAA RSM European portal

• Select the catalogue you want to validate

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

• • • View all CDI schema versions View statistics about detected errors Run the validation with the last version of XML schema

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

• • Upload the file you want to validate Validation works only file per file

[email protected] – www.seadatanet.org

• The file is displayed on the screen • Press Validate

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

• Errors and warning are displayed • Warning are not obstacles for XML delivery • Errors must be corrected

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

VALIDATION of XML files

• XML validation Services have been developed in the frame of SeaDataNet by the Russian NODC • It is a Web validation Service available at

http://www.seadatanet.org/validator

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

[email protected] – www.seadatanet.org

Training Workshop – Ostende – 1-4 March 2010

Training course – Ostende – 2-6 July 2012

Practical work on MIKADO

Creation of XML files for CDI

M. Fichaut

Training course – Ostende – 2-6 July 2012

Exercise 1

a.CDI Automatic generation from the CDI summary file created this morning with NEMO – Validation of the XML files b.Creation

of the corresponding coupling file

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

Collection of ASCII files

From ASCII data to XML files

ODV files a b Coupling file for DM XML CDI files VALIDATOR Coupling table Export Coupling table summary_CDI_NEMO.xml

Delivered with NEMO CDI summary CSV file

[email protected] – www.seadatanet.org

MIKADO SeaDataNet CDI

Training course – Ostende – 2-6 July 2012

• • • •

a) Configuration file for CDI XML

•

Run MIKADO

•

Open the configuration file delivered with NEMO :

–

MIKADO =>

Automatic/Open/CDI

the configuration file is in the HOME_DIRECTORY of NEMO, but for the training it is under

\Practical work\config

file

summary_CDI_NEMO.xml

•

Make the connection with the Excel file

that you have created with NEMO ( the Excel file must be closed to avoid conflict access

) Check the connection

(you must have a green message)

Go to the Queries tag and have a look at the queries Check them

(green message after check all)

Save your configuration file [email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

a) XML file generation

• • Run Automatic/Generate/CDI – Open the configuration file

\Practical work\config\summary_CDI_NEMO.xml

• Choose the export directory (where CDI XML files will be written) – Choose the type of export file you want : • XML files • Zip file containing the XML files • Both (XML and Zip) Open one XML export file to have a look at it with MIKADO

Manual>Open>File>CDI

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

a) Web validation service: Validator

• • • • • • Check the XML file with the Web validation Service at : http://www.seadatanet.org/validator Select the CDI catalogue Validate file with CDI Xml schema Upload one of the XML file that has been generated Validate the file Look at the result of the validation :

you must have no error

, you can have some warnings depending on the vocabulary versions.

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

• • • • • • • •

b) Coupling configuration file

Run MIKADO Open the configuration file delivered with NEMO :

–

MIKADO =>

Tools/Coupling table for download manager/ Import CDI configuration

the configuration file is the same than in a) ) in the

config

b) Coupling file generation

Run the generation of the coupling file

–

MIKADO =>

Tools/Coupling table for download manager/ generate/ Modus 1 or Modus 3 depending on your ODV file(s)

Select the configuration file for coupling that you just create Select the output directory Choose the mode for writing (Append or Overwrite) Generate the coupling file Open it with a text editor [email protected] – www.seadatanet.org

Exercise 2

• EDIOS manual input of

– 1 Program – 1 Series – 1 Platform

Training course – Ostende – 2-6 July 2012

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

• • •

Under SDN_tools\practical_work

Under \EDIOS__POSEIDON_Example\EDIOS_Data Open the EDIOS_POSEIDON.xls file describing the EDIOS program, series and platforms ( 1 program, 15 series,15 platforms ) • Input the program information into MIKADO and create the corresponding XML file • Input 1 series information into MIKADO and create the corresponding XML file • Input the platform related to the previous series in MIKADO and create the corresponding XML file Why can’t you first enter a platform, then a series then a program?

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

Exercise 3

• EDIOS Automatic generation of XML files for:

– Program – Series – Platforms

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

• • • • •

Under SDN_tools\practical_work

\EDIOS__POSEIDON_Example\EDIOS_Data Open the EDIOS_POSEIDON.xls file describing the EDIOS program, series and platforms ( 1 program,15 series, 15 platforms) Run MIKADO to create the automatic query to retrieve the program information – Or open the already written XML configuration for MIKADO for EDIOS program : \EDIOS__POSEIDON_Example\EDIOS_Data\queries\poseidon_edios_program.xml

Run MIKADO to create the automatic query to retrieve the series information – Or open the already written XML configuration for MIKADO for EDIOS program : \EDIOS__POSEIDON_Example\EDIOS_Data\queries\poseidon_edios_series.xml

Run MIKADO to create the automatic query to retrieve the program information – Or open the already written XML configuration for MIKADO for EDIOS platform : \EDIOS__POSEIDON_Example\EDIOS_Data\queries\poseidon_edios_platform.xml

[email protected] – www.seadatanet.org

Training course – Ostende – 2-6 July 2012

Generate the XML EDIOS files

• EDIOS PROGRAM – Run MIKADO : Automatic>Generate>EDIOS Program • Choose the \queries\poseidon_edios_program.xml

• Choose the export directory \results\program • EDIOS SERIES – Run MIKADO : Automatic>Generate>EDIOS Series • Choose the \queries\poseidon_edios_series.xml

• Choose the export directory \results\series • EDIOS PLATFORMS – Run MIKADO : Automatic>Generate>EDIOS platforms • Choose the \queries\poseidon_edios_platforms.xml

• Choose the export directory \results\platforms