Transcript Storing EM data with HDF5

SEG Technical Standards Committee – EM subcommittee status update
EAGE 2011, Vienna
May 24, 2011
Friedrich Roth (subcommittee chairman)
Status update
Motivation for the formation of a TSC subcommittee
At last year’s TSC meeting at the SEG Annual Meeting in Denver, several attendees
(EMGS, Interaction, OHM-RSI, Statoil) expressed interest in self-describing binary data
formats to facilitate the exchange of marine EM field data.
Therefore the TSC is forming a subcommittee with this focus.
What has happened since then?
• Telephone/ web meeting on Feb 16 to inform a number of interested parties about the
initiative.
• Announcement in April version of TLE and the EM Network on LinkedIn (593 members)
 Limited feedback
• E-mail invitation for telephone/web meeting on June 13 to solicit input from appropriate
organizations and individuals.
Individuals/organizations that have showed interest to support or engage in
this initiative
•
•
•
•
•
•
•
•
•
•
•
•
•
Mikhail Boulaenko, Rocksource, Saint Petersburg, Russia
Hung-Wen Tseng; OHM-RSI, Houston, USA
Antony Price, Total, Pau, France
Øyvind Andersen, Interaction, Stavanger, Norway
Rune Halvorsen, Interaction, Stavanger, Norway
Andreas Becht, Statoil, Stavanger, Norway
Vidar Markhus, EMGS, Trondheim, Noway
Friedrich Roth, EMGS, Houston, USA
Charles Oden, Earth Science Systems, Golden, USA
Joonsang Park, Norwegian Geotechnical Institute, Oslo, Noway
Gitta Zaalberg, Fugro, Leidschendam, The Netherlands
Anna Kelbert, MT group at Oregon State University, USA
Allen McCay, PGS, Norway
Things to do
• Establish complete subcommittee member list
• Inform other standardization organizations about planned effort (OGP)
• Evaluate legal foundation for HDF5 based SEG standard
• Work on draft for standard (June-Sept 2011)
• SEG TSC meeting and workshop (San Antonio, Sept 18-23)
Previous work
H5EM-TS: Self-describing data exchange format for field
EM data
Main contributors
EMGS: Vidar Markhus, Friedrich Roth
Statoil: Andreas Becht, Arne Johannes Kaaijk Jenssen
Interaction: Øyvind Andersen
Details:
ftp://fileformats.emgs.com
• Documentation
• Example files
EM survey data
Currently supported by H5EM-TS storage model
H5EM-TS extension
Types of EM surveys
• Controlled-source electromagnetics (CSEM)  Magnetotellurics (MT)
• Marine  Onshore
• Ocean bottom node (OBN)  Streamer
Types of data
• Electric and magnetic fields
• Source current (CSEM)
• Navigation
• Environmental data
• Hardware description
• Calibration information
• Quality indicators (e.g. noise estimates)
• Survey information
Data states
• Field data (raw data or calibrated)
 time series
• Processed data
 Magnitude & phase data (frequency-domain CSEM)
 Traces (transient CSEM)
 Impedance tensor (MT)
Source: http://marineemlab.ucsd.edu
Types of data formats
Recording format
• Raw data written to disk by
data acquisition units
• Hardware dependent data
formats
Primary purpose of H5EM-TS
Secondary objective of H5EM-TS
Data exchange format
• Preprocessed data containing
all information relevant for
further processing, analysis
and interpretation
Processing format
• Data stored as required for
application of a specific
processing task
• In-house data formats
• Standard data formats
(preferred)
HDF5
Self-describing binary data format
• Versatile high-level data model
• APIs for C, C++, Fortran, Java, Python, Matlab
• Platform independent storage
Developed and maintained by The HDF Group, Inc.; www.hdfgroup.org
• Non-profit organization
• Mission-driven
Open source technology
Users
• Argonne National Laboratory, US Department of Energy, Deutsche Bank, Lawrence
Berkeley National Laboratory, Lockheed Martin, Los Alamos National Laboratory,
NASA, Raytheon, US Naval Research Lab, and many more
Key features
• Groups, datasets, attributes
• Rich set of data types (integer, floating point, string, …)
• Shared dimensions
• Compression and chunking (optional)
• Very efficient readers/writers exist
File
Groups
Attributes
Dimensions
Datasets
H5EM-TS design concepts
Intuitive data hierarchy
• Defines how the EM survey data is organized in the file and provides well-defined locations for each
piece of information stored.
Standard data objects (datasets and groups)
• Define classes (i.e. templates) for storing the data.
Standard types
• Categorize data objects within their respective class, e.g. the type of receiver, the type of channel, etc
Naming convention
• Describes rules for naming groups, datasets and their attributes.
Standard units
• Describe acronyms for specifying a quantity’s physical unit, e.g. “m” for meter.
Flexible sampling modes
• Support for (block-)regular and irregular sampled quantities.
• Support for (block-)constant quantities.
H5EM-TS design concepts (cont’d)
High-precision timestamp
• Based on POSIX time and describes a 64bit unsigned integer counting the number of nanoseconds
since midnight Coordinate Universal Time (UTC) of a January 1, 1970, not counting leap seconds.
• Human-readable UTC timestamp string for first sample of each stored time series.
Geodetic definitions
• In accordance with the standards of the OGP Surveying & Positioning Committee (www.epsg.org), in
particular the P2/94 Exchange Format for Raw Marine Positioning Data.
Hardware descriptions
• Support for horizontal electric dipole sources (HED) and ocean bottom node receivers (OBN).
• More hardware descriptions planned in future format versions.
Scalability
• Equal possibility for storing one or more complete EM surveys in one file as it does for storing data
from just one receiver unit or source unit.
H5EM-TS data hierarchy
1 ..*
survey
0 ..*
0 ..*
receiver_deployment
source_deployment
receiver
OBN_description
coordinate_reference_system
1 ..*
1 ..*
1
1
source
1 ..*
channel_collection
0 ..*
1
: group
HED_description
: dataset
1 ..*
channel
0 ..*
regular_timeseries
0 ..*
blockregular_timeseries
1
indeces
0 ..*
0 ..*
constant_timeseries
1
blockconstant_timeseries
1
time_intervals
0 ..*
irregular_timeseries
1
timestamps
H5EM-TS data hierarchy
survey
1 ..*
: group
1
coordinate_reference_system
1
1
geodetic_datum
vertical_datum
0,1
transverse_mercator
0,1
mercator
0,1
lambert
0,1
seven_parameter_cartesian_datum_shift
H5EM-TS examples
H5EM-TS benefits
• Self-describing
• Easy access to all relevant data
• Cross-platform portability
• Freely available libraries for data reading/writing in multiple programming languages
• Built-in compression (optional)
• A naming convention and standard names for quantities typical to EM data
• Forward compatibility supports future EM systems
• Can serve equally as data exchange format, archiving format and processing format
• Support for marine CSEM and MT data
• Full support for the vector nature of EM fields and sources
• Support for stationary and moving receivers/sources
• Designed for data acquired by continuous recording systems
• Geodetic definitions in accordance with P2/94