Transcript MIKADO tool - Seadatanet2

Training Workshop – Ostende – 26-29 October 2009
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
REFORMATTING TOOLS
Med2MedSDN and NEMO
M. Fichaut
www.seadatanet.org
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
SeaDatanet
Vocabulary
Metadata
in
Database
Metadata
Input
Metadata
In
Excel files
XML Validator
MIKADO
XML
Metadata
Files
MIKADO
CSR
Coupling
table
Data in
Database
Data Input
EDMED
Collection
Collection
NEMO
of
of
ASCII files
ASCII files
Format SDN
Format X Med2MedSDN
EDMERP
CDI
ODV
www.seadatanet.org
S
E
A
D
A
T
A
N
E
T
Local copy
of data
to download
Partner system : pilot data centre
Data request
Data download
P
O
R
T
A
L
European portal
2
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 Overview
• Med2MedSDN
• Objectives and main features
• NEMO
• Objectives, main features and principles
• Technical characteristics
• Description of the different steps to follow to be able to
reformat ASCII files to SeaDataNet format
• Link between NEMO and MIKADO
• Link between NEMO and SDN Download Manager – coupling
table
www.seadatanet.org
3
Training Workshop – Ostende – 26-29 October 2009
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Med2MedSDN
For MEDATLAS files
www.seadatanet.org
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 Objectives of Med2MedSDN
• Translate MEDATLAS files to SeaDataNet MEDATLAS
files - available under Windows
• Version 1.1.02 and user manual available at :
http://www.seadatanet.org/standards_software/software/Med2MedSDN
www.seadatanet.org
5
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 Med2MedSDN main features
• Reformats MEDATLAS files to MEDATLAS SeaDataNet format
• Java tool, bilingual (French, English)
• Adds the additional SeaDataNet information : mapping for
parameters and LOCAL_CDI_ID and EDMO_CODE
• Can reformat one file or a large number of files (in one directory)
• Generates a coupling file for the SeaDataNet download manager
• Linked to SeaDataNet vocabularies through Web services for
parameters mapping and for list of EDMO codes
• Need of internet connexion while updating lists
www.seadatanet.org
6
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 Med2MedSDN
Med2MedSDN vocabulary update
www.seadatanet.org
7
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 Med2MedSDN main screen
www.seadatanet.org
8
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 Med2MedSDN log files
INFO
Informative messages for starting of the conversion or successful conversion
ERROR
For conversion errors : conversion is cancelled on the current file but
continues on the other files
FATAL
For conversion errors which stop the processing of the files
• Errors (managed by Log4j) are registered in a log file in Med2MedSDN
installation directory.
• One line in the log file is composed as following:
• Date, Name of the Software, Error severity level, Error message
www.seadatanet.org
9
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 Med2MedSDN - User manual
• User manual is provided :
File : sdn_Med2MedSDN_UserManual_V1.0.pdf
• Detailed explanation for Med2MedSDN installation
and use, lots of snapshots
www.seadatanet.org
10
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
ANY QUESTION?
www.seadatanet.org
11
Training Workshop – Ostende – 26-29 October 2009
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
NEMO
For ASCII files on any type
www.seadatanet.org
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 Objectives
 NEMO is a reformatting software used to
generate ASCII files at MEDATLAS or ODV formats which are
defined as SeaDataNet formats for data exchange between
SeaDataNet partners. Can be downloaded from SeaDataNet
Web site :
http://www.seadatanet.org/standards_software/software/nemo
 It was created in IFREMER for internal purposes, and adapted
to take into account the SeaDataNet formats and vocabularies.
www.seadatanet.org
13
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 Technical characteristics
• Written in Java Language (Version >= 1.6)
• Bilingual (French, English)
• Available under multiple environments :
• Microsoft : Windows 2000, XP, VISTA , APPLE
• Unix - Solaris
• Linux
• Use of SeaDataNet common vocabularies web services
• to update lists of values of the SeaDataNet common vocabularies
• need network connections in order to have up to date lists of
values.
• But NEMO works offline once the lists are up-to-date
www.seadatanet.org
14
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 NEMO main features
• Reformat any ASCII file of vertical profiles, time-series
or trajectories to a SeaDataNet ASCII format (ODV,
MEDATLAS).
• The input ASCII files can be :
• one file per station for vertical profiles or time
series
• one file for one cruise for vertical profiles, time
series or trajectories
• Interact with Mikado, to be able to generate ISO-19115
XML descriptions of the data.
www.seadatanet.org
15
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 NEMO principles
• NEMO must be able to read almost any ASCII format to
translate it to MEDATLAS, ODV
• Users of NEMO describe the entry files format so that NEMO
is able to find the information which is necessary in the
SeaDataNet formats
• One mandatory pre-requirement is that in the set of entry files
the information about the stations must
• be located at the same position : same line in the file,
same position on the line or same column if CSV format
• be in the same format
• For example : for all the stations the latitude is
• on line 3 on the station header,
• from character 21 to character 27, or 3rd column in CSV
• the format is +DD.ddd
www.seadatanet.org
16
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 Running NEMO
• 4 or 5 steps to convert the entry files to a SeaDataNet format
1. Describe the type of file(s)
2. [Describe the cruise, if the files are related to 1 cruise]
•
NEMO is able to read XML CSR generated with MIKADO
3. Describe the station information
4. Describe the measured parameters
5. Convert the file
• One more step to
• Save the description of the format (Model)
www.seadatanet.org
17
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 NEMO - Description of entry files (1)
• Where are the files?
• Is it a cruise?
•
Is it one file per cruise or
one file per station ?
• Is it a collection of station files ?
• grouped by cruises or not?
• Are the files with separators?
• Tabulations? Semicolon?
Comma?
• Are they vertical profiles, time
series or trajectories data?
www.seadatanet.org
18
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 NEMO - Description of entry files (2)
• File to convert
is displayed in
NEMO window
• The principle is
to describe the
input file by
selecting the
information in
the file window
www.seadatanet.org
19
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 Station header : constant number of lines
www.seadatanet.org
20
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 Station header : number of lines not constant
www.seadatanet.org
21
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 Cruise description
Can be
• Manually input
• Imported from a
SeaDataNet
XML CSR
• Imported from a
database
(IFREMER
only)
www.seadatanet.org
22
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 Station
description
www.seadatanet.org
23
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 Station
description
• Select the
date in the
file, in the
left part of
the screen
www.seadatanet.org
24
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 Station
description
• Input the
date
format
www.seadatanet.org
25
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 Station
description
• Click on
the Set
button to
tell NEMO
the start
and end
position of
the date
www.seadatanet.org
26
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 Station
description
• Click on
the Test
button to
verify your
input
www.seadatanet.org
27
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 Station
description
www.seadatanet.org
28
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 Data
description
• Give the
position ,
the default
value, the
format of
each
parameter
• Pressure is
mandatory for vertical
profiles (MEDATLAS
requirement) and
cannot be deleted.
• QC flags
can also
be
described
www.seadatanet.org
29
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 Data
description
• Select
« Parameter
list » in the
context
menu of the
measured
parameters
table (right
click to open
the menu)
www.seadatanet.org
30
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 Data
description
www.seadatanet.org
31
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 Data
description
• Select the
value of the
measured
parameter
in the file, in
the upper
left part of
the screen
www.seadatanet.org
32
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 Data
description
• Select « Set
start /end »
in the
context
menu of the
measured
parameters
table
www.seadatanet.org
33
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 Data
description
• Start and
End
position of
temperature
are added in
the table
www.seadatanet.org
34
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 Data
description
• Select
« Update
test » in the
context
menu of the
measured
parameters
table
www.seadatanet.org
35
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 Data
description
• The values
read by
NEMO are
displayed
in the Test
column of
the table.
User is able
to check
that his
input is
correct
www.seadatanet.org
36
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 File
conversion
Output formats
• MEDATLAS
• Multi-station
• ODV
• Mono-station
• Multi-station
www.seadatanet.org
37
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 While converting …
NEMO is able to
• Generate a SeaDataNet CDI Summary
• Text file containing the minimum mandatory information
needed in the CDI ISO-19115 description
• This file can be converted to an Excel file
• Which can be read by MIKADO to generate the XML CDI
files
• Generate a coupling table that will be used by the
Download manager of SeaDataNet
• It is the link between the LOCAL_CDI_ID and the file
www.seadatanet.org
38
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 CDI summary
• Further information must be added in
NEMO settings (menu Options of
NEMO)
www.seadatanet.org
39
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 CDI SUMMARY File
Text file
• Open it with Excel
• Rename the sheet : STATION
• Save the file as an EXCEL file
www.seadatanet.org
40
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 Interaction with MIKADO
Collection
of
ASCII files
ODV
files
XML
CDI files
summary_CDI_NEMO.xml
CDI summary
CSV file
www.seadatanet.org
SeaDataNet
CDI
MIKADO
41
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 Coupling table for Download Manager
• Further information must be added in NEMO
settings (menu Options of NEMO)
• The coupling table makes the link
between each LOCAL_CDI_ID and the
file(s) which contains it
www.seadatanet.org
42
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 Coupling table
If the “Generate mapping” option is clicked :
• NEMO will insert a record in the coupling table for each vertical
profile, time series or trajectory that it converts.
• The coupling table is managed by a local database imbedded in
NEMO
• The coupling table can be :
• Edited (for modification, insertion or deletion)
• exported (to be used by the download manager)
• imported (from previous version of NEMO, for example)
www.seadatanet.org
43
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 Coupling table content (1)
LOCAL_CDI_ID
Modus
• 1 : mono-station
• 3 multi-station
Filename
Date of creation
www.seadatanet.org
44
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 Coupling table content (2)
• One unique LOCAL_CDI_ID in one mono-station ODV file
• The same LOCAL_CDI_ID in 2 different files :
• One multi-station MEDATLAS file
• One mono-station ODV file
www.seadatanet.org
45
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 Downloading the data
CSR
EDMED
Coupling
table
Collection
of
ASCII files
Format SDN
EDMERP
Download
Manager
List of
LOCAL_CDI_IDs
CDI
Local copy
of data
to download
Data request
S
E
A
D
A
T
A
N
E
T
P
O
R
T
A
L
Data download
www.seadatanet.org
SeaDataNet Data Centre
46
European portal
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 NEMO – User manual
• User manual is provided :
File : sdn_Nemo_UserManual_V1.2.pdf
• Detailed explanation for NEMO installation and
use, lots of snapshots
www.seadatanet.org
47
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
 And don’t forget that
• All software freely available at :
http://www.seadatanet.org/standards_software/software
• FAQ on the web site for MIKADO and NEMO
• Don’t forget to read the user manuals which are very detailed
• Any problems using the software
Mail to [email protected]
www.seadatanet.org
48
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3rd SeaDataNet training course – Ostende – 27 October 2009
ANY QUESTION?
www.seadatanet.org
49