Transcript Data Management System to Collect, Quality Control

Data Management System to
Collect, Quality Control,
Distribute, and Archive Near
Real-time Marine Data
Jacob T. Rettig, Mark A. Bourassa, Ph.D., Jiangyi Hu, Ph.D.,
E. Michael McDonald, Jeremy J. Rolph, and Shawn R. Smith
Introduction
• The application of the Data Management System
•
to collect data from platforms in the Gulf of
Mexico will be described.
Shipboard Automated Meteorological and
Oceanographic System (SAMOS) initiative
– Developed in 2004 to support underway data
collection of participating research vessels
• Northern Gulf of Mexico Institute (NGI)
– Updated in 2008 to provide data stewardship for
instrumented Air Force towers in the northern Gulf of
Mexico
SAMOS Initiative
• Manages navigational, surface
•
•
•
•
oceanographic, and
meteorological observations
From 19 oceanographic
research vessels
Validation studies for satellite
sensors and numerical model
products
Air-sea interaction research
Developing satellite data
retrieval algorithms
NGI
• Observations are
managed for one Air Force
Tower currently
• Contributes to physical
•
N7 tower
oceanographic, air-sea
interaction, and
ecosystem research
Real-time validation for
offshore forecast from
the National Weather
Service
The Data Management System
• Combines automated processing programs, a database, and web
tools
• Prior to processing, the database must be populated with
platform- and instrument-specific metadata
The Database: File Tracking,
Version Control, & QC
Data Flow
• SAMOS initiative and NGI data flows
– differ in data acquisition and final product
– identical in automated Data QC processing
Preliminary Data Processing
• Check & Verify - Incoming ASCII files are verified if they
originated from a recruited platform and properly
formatted.
• Quality Control
– Verify that observations are over water and within realistic
physical limits.
– Poor quality data values are not removed from data set, instead
they are flagged.
Data Merge and secondary QC
Visual QC - (SAMOS only)
Data Archiving – (SAMOS only)
• The Archive Contains:
–
–
–
–
–
–
–
–
Original file
Preliminary NetCDF file
Merged NetCDF file
Visual QC NetCDF file
File mapping
parameter mapping
File list with MD5s
MD5 of file list
• Monthly archive
agreement with NODC
Distribution (http://samos.coaps.fsu.edu)
• Vessel and instrumentation
metadata
• Imagery and schematics
Distribution
(http://www.coaps.fsu.edu/ngi)
• Raw data at 10 minute
intervals
• NetCDF downloads in
development
Acknowledgements, Links, and Q&A
• COAPS – http://www.coaps.fsu.edu/
• SAMOS – http://samos.coaps.fsu.edu/
• NGI – http://www.coaps.fsu.edu/ngi/
SAMOS is supported by the NOAA Office of Climate
Observation and NSF Oceanographic Technical
Services. The NGI is supported by NOAA Office
of Oceanic and Atmospheric Research.
Questions?