Transcript Data Management and Product Delivery in the BLUElink> Era

BLUElink>: Operational Issues and
Product Delivery
Graham Warren
National Meteorological and Oceanographic Centre
Central Operations Branch
Bureau of Meteorology
Overview
• NMOC
– Role
– Scheduling
• Observational Data Requirements and Management
• Status of Current Systems
• Product Delivery
– Internal Users
– External Users
– Research Users
Second BLUElink> Users Forum
17/07/2015
2
NMOC Overview
• Most forecasts/warnings are generated manually in the
Bureau’s 7 Regional Offices, but all depend on
centralized guidance from NMOC.
• NMOC is the 8th 24x7 office and is responsible for:
–
–
–
–
–
Running of the NWP suite
ITC Help Desk
Aviation forecasts
Tsunami Warnings
Implementation & support of NWP systems
Second BLUElink> Users Forum
17/07/2015
3
National Meteorological and
Oceanographic Centre (NMOC)
• 60 staff
• Central Operations (24x7):
–
–
–
–
Southern Hemisphere Analysis
Tsunami Warning Centre
Aviation Desk
Help Desk (IT and Comms)
• Operational Development
– NWP model implementation & support
• Oceanographic Systems
– Ocean model implementation and support
• Software Systems
– Real-time Database and Systems Administration
• National Weather Graphics Unit
Second BLUElink> Users Forum
17/07/2015
4
Oceanography
• Support for Marine Meteorology
– Wave models (and surface winds)
• Support for Climate
– SST analysis
– Sub-surface analysis
– Seasonal prediction model (POAMA)
• BLUElink> Ocean Forecasting
• Marine Hazards
– Tsunami
– Storm surge
Second BLUElink> Users Forum
17/07/2015
5
The Operational Forecast System
OceanMAPS
Data Flows – Global Models
e)
OceanMAPS
Global
SST Analysis
O.I
Ocean Model
POAMA
Australian Region
MOF
PME
Manual
Intervention
(MSLP)
GASP
Global Assimilation
And Prognosis
1DVAR+GenSi+Spectral
UV
OCF
(Sites
Gridded)
Sea State
Model
WAM
EER
HYSPLIT47
f)
Boundary Conditions
To TXLAPS & LAPS
GASP_EPS
Ensemble Prediction
System
• Most other models rely on GASP model output.
• Boundary conditions from GASP feed into TXLAPS and LAPS
• Model outputs from some systems above get stored in RTDB
and MARS
Verification
Scheduling – Supervisor Monitor Scheduler
(from ECMWF)
OceanMAPS schedule
Data Requirements
• In Situ data
– Drifting Buoys
– Moored Buoys
– Ship of Opportunity
(XBT)
– ARGO profiling floats
– Sea-level gauge data
Second BLUElink> Users Forum
17/07/2015
10
Data Requirements
• Satellite data
– SST
• AVHRR
– Local retrievals
– Global
• AATSR
• Microwave (AMSR-E)
– Altimetry
– Scatterometer
Second BLUElink> Users Forum
17/07/2015
11
Data Monitoring (Quantity)
Data Monitoring (Coverage)
Second BLUElink> Users Forum
17/07/2015
13
Data Monitoring (Timeliness)
Meteorological Forcing
• Need to have access to surface fields from the
meteorological models used for forcing
– GASP
– LAPS
• Wind Stress
• Heat Fluxes
• Useful to have integrated DM system
Second BLUElink> Users Forum
17/07/2015
15
Meteorological Archive and
Retrieval System
• Developed and Operated by ECMWF
• Suited to handling large data sets and operational
model analyses and forecast products
• Provides seamless access to real-time and
historical data
• Allows sub-setting in time and space of large
datasets
Second BLUElink> Users Forum
17/07/2015
16
Ocean Analysis
and
Modelling Systems
Internal
Data Viewer
In situ data
(ARGO, SOOP,
Buoys,
SST Data
Mass storage
Database
File delivery
(AVHRR, ATSR,
AMSR-E,…)
MARS
Altimeter Data
(Jason-1, ENVISAT,
GFO)
Internet Access
NWP Output
(GASP, LAPS,…)
P1 Data and Product Management
Status of BLUElink> Systems
• High resolution SST analysis
– Regional (1/12 deg resolution - operational)
• Run daily ~ 0100 UTC
– Global (1/4 deg resolution – testing)
• To be run daily ~ 0400 UTC
• OceanMAPS – Ocean Forecast System
– Operational 7 day forecast run twice weekly (Monday and
Thursday)
– One major failure fixed (15/18 Jan)
– Some data drop-outs cause problems
Second BLUElink> Users Forum
17/07/2015
18
Product Delivery
• Internal Users
– Diagnostic products
• Data monitoring
• System performance
– Graphical Viewer
– Intranet
• Tailored products
Second BLUElink> Users Forum
17/07/2015
19
Product Delivery
• External Users
– Basic Products (freely available)
• Static Web images
– Registered users (cost of access)
• File delivery via ftp server
• Research Partners
– OpenDAP
Second BLUElink> Users Forum
17/07/2015
22
Graphical Products
Second BLUElink> Users Forum
17/07/2015
23
OceanMAPS Data Products
• Best daily ocean in-situ profile observations
• Remotely sensed sea surface height (Jason, Envisat,
GFO)
• Remotely sensed sea surface temperature (GHRSST)
• Surface flux products (NWP GASP/LAPS)
• Basic model products
– sea surface height (SSH)
– ocean currents (U,V,W) and states (T,S,Rho)
– mixed layer depths
– surface momentum, heat, and fresh water fluxes
• Model restart files
Second BLUElink> Users Forum
17/07/2015
24
Data and Product Storage
• Behind real-time analysis
– Run once per day
–
4GB per day
–
32GB per week
– 138GB per month
– 1680GB per year
• 192 hour forecast
– Run twice per week
–
92GB per week
– 368GB per month
– 4788GB per year
OceanMAPS file/data formats
• File Formats
- NetCDF v3 file format (all data products)
- Climate and Forecast (CF-1.0) conventions
- COARDS conventions
•
Products
- Best daily Observations (Argo netCDF format)
- Sea surface temperature products
- GHRSST L2P, L3P, and L4 formats (netCDF, CF)
– Bureau NWP GASP and LAPS surface fluxes (netCDF, COARDS)
– MOM4 model products (netCDF, COARDS, CF-1.0 coming)
• Conversion to GRIB/BUFR (WMO Standards)?
Second BLUElink> Users Forum
17/07/2015
26
OceanMAPS data/file size
• Basic product data array size per time point
– 2-D array, single precision is 4.4 MB
– 3-D array, single precision is 206 MB
• Restart data array size
– 2-D array, double precision is 8.8 MB
– 3-D array, double precision is 412 MB
• Basic data product (daily mean, forcing 3hrly)
– U,V,T,S,eta,forcing is ~1129 MB
• Basic data product (3 hourly mean)
– U,V,T,S,eta,forcing is ~7229 MB
Second BLUElink> Users Forum
17/07/2015
27
Data Transport Protocols
• FTP - file transfer protocol
– Time tested and robust
– But…user downloads entire file
• HTTP - hyper transport protocol
– Works thru most firewalls
– Wrapped in nice interface for user
– But…user downloads entire file
• OPeNDAP - Open Network Data Access Protocol
(formerly
DODS)
–
–
–
–
Catalogs of data for users and crawlers
User interface for viewing file metadata
Virtual data sets and aggregation
User downloads selected spatial and temporal indices, as well as variables
Second BLUElink> Users Forum
17/07/2015
28
Products Available via ftp
(to Registered Users)
• Fixed Domains (as for graphics)
• NetCDF format (GRIB coming?)
– Temperature (1-10 Mbytes/time slice)
– Salinity
– Zonal Wind
– Meridional Wind
– Sea Level Anomaly (20-200kBytes)
Second BLUElink> Users Forum
17/07/2015
29
Distributed Oceanographic Data System (DODS)
or OpenDAP
 Conceived in 1993 at a workshop held at URI.
 Objectives were:
– to facilitate access to PI held data as well as data held in national archives and
– to allow the data user to analyze data using the application package with which he
or she is the most familiar.
 Basic system designed and implemented in 1993-1995 by Gallagher and Flierl
with NASA and NOAA funding.
 From 1994 to present it has been extended with NASA, NOPP, NSF and NOAA
funding.
Second BLUElink> Users Forum
17/07/2015
30
Considerations with regard to the
development of OPeNDAP
 Many data providers
 Many data formats
 Many different client types
 Many different semantic representations of the data
 Many different security requirements
Our (BoM) History of OPeNDAP
• 2004 - First DODS server installed for TPAC digital library to serve nearreal time SST and MSLP data sets (~ 20 GB )
• 2005 - Trialed enhanced service to serve MARS data thru DODS
(closed due to lack of security)
• 2006 - Upgrade to OPeNDAP server 3 and THREDDS server to deliver
last 7 days of NWP and ocean forecasts (~ 1.0 TB)
• 2007 - Added Tsunami scenario database (~ 5.0 TB)
• 2008 - To add second phase of Tsunami scenario database (~55 TB)
Second BLUElink> Users Forum
17/07/2015
32
OPeNDAP
Bureau
netCDF
CSIRO
HDF
Matlab
Ferret
VisAD
GrADS
IDV
Access
IDL
Excel
ncBrowse
Second BLUElink> Users Forum
17/07/2015
33
MARS
GRIB
OPeNDAP Clients
• Network addresses based on URLs
– url = 'http://www.opendap.org/opendap/nph-dods/data/fnoc1.nc?u,v’
– url_path = 'http://www.opendap.org/opendap/nph-dods/data/
• MATLAB
– Command: “loaddap url”
– Structure is loaded into memory with file metadata names
• IDL
– Command: “stat = OPENDAP_GET(url, data)”
– Structure is loaded into memory with file metadata names
• NCO
– Command: ncks -o ./output -O -C -q -p $url_path fnoc1.nc?u,v
• Open Source: www.opendap.org
Second BLUElink> Users Forum
17/07/2015
34
GODAE Services
LAS
WCS HTTP
TDS
OPeNDAP gateway
LAS ------ Live Access Server
TDS ------ THREDDS Data Server
WCS ----- Web Coverage Service for GIS
MARS ---- Met. Archival Retrieval System
SIMS ------ Services Information Mgmt System
SIMS
Disk Cache
BoM Firewall / Router
TDS
SQL
TDS
MARS
OPeNDAP gateway:
User Authentication:
yes
DataSet Authorization: yes
Resource Mgmt:
yes
TDS
Global File
System
SAN
Disk
Array
Disk Array
Tape
Silo
Fast Disk Array
Fast Disk Array
SATA Disk Array
SATA Disk Array
GODAE Service Architecture
Virtualized Storage - data aging
Tape Archive
Conclusion
• OceanMAPS has proved to be robust and reliable
• Accuracy dependant on input data
– In situ
– altimetry
• Product availability
– Fixed Images (freely available)
– Fixed NetCDF files (available to Registered Users)
– OpenDAP Access (available to research users)
Second BLUElink> Users Forum
17/07/2015
36