Transcript hycom data service
HYCOM Data Service
www.hycom.org/dataserver
An overview Ashwanth Srinivasan, (FSU) Steve Hankin (NOAA/PMEL) Contributors: Jon Callahan (Mazama Consulting) Roland Schweitzer (Weathertop Consulting), Ansley Manke (NOAA/PMEL) Jeremy Malczyk (UW/JISAO) Peter Cornillon (URI) 1
HYCOM Data Service: History and Current Status Data Service was launched in 2002 in Miami and served HYCOM Outputs via FTP, LAS and OPeNDAP servers Started Serving Near Real Time Atlantic data in 2003 Hosted about 3 TB of Data until Dec, 2006 in Miami.
Jan -2007 – service moved to FSU and hosted on a 100 TB SAN Currently serving approximately 60 TB of data 2
New Hardware
100 TB Fibre Channel SAN Three 8 CPU machines 32 GB RAM/machine
Software
Red Hat Linux Red Hat cluster suite and Global File System Apache Web Server THREDDS LAS Server Vsftpd server 3
Operational Enhancements
All components (LAS,OPENDAP,FTP) co-located in the same physical machine.
Subset of files are uncompressed for faster access In memory, decompression by using RAM scratch disks High-availability architecture – minimal downtime during updates 4
• • • • •
Currently Available Datasets
Daily outputs from the real time Global HYCOM+NCODA analysis (Apr 2007 – current) Global HYCOM+NCODA analysis (2004-2005 available now; 2005-2007 will be available shortly) Four years of Global Non-Assimilative Simulation (2003-2007) Near real-time 1/12 prediction system output (June 2003 – Present) Atlantic Ocean Monthly mean 1/12 Pacific Ocean simulation output (1978-2003) 5
Global HYCOM Data Serving Challenges
Very large data/file sizes – each day of output is 12-19 G – consists of 26 (15+9) 2d and 3d variables (4500 x 3298 x 32 pts) Curvilinear, staggered and hybrid vertical coordinates makes analysis and visualization on large files resource intensive and time consuming Need to handle model products in non standard native HYCOM format and in standard format like NetCDF (B.Cs are needed on native grids and in both NetCDF and HYCOM IO formats) 6
HYCOM Ocean Prediction System Data Flow
Global run at NAVO daily – 5 day hindcast and 5 day forecast in curvilinear and hybrid coordinates 50 G pulled to FSU ~ 2-3 hrs Conversion to NetCDF and re-gridding to Z levels *Approximately 4-5 hrs of data processing daily THREDDS and LAS Update 7
Means of Data Access
THREDDS Provides a dataset catalog and OPeNDAP access to all datasets.
Live Access Server V isualization and downloads in different formats, batch access etc.
OPeNDAP enabled clients Choice of common applications or user written programs to access data FTP 8 NetCDF files for each variable
HYCOM LAS : http://hycom.coaps.fsu.edu/las/ Provides NetCDF data in both native HYCOM (staggered) and regular (non-staggered) grids On-the-fly conversion to fixed Z levels (but limited to pre-configured Z levels) tested HYCOM format IO as a product –but needs better integration Full resolution global outputs not available currently via LAS 9
Access to native coordinates and regridded fields 10
Hybrid-Z regridding
fixed-Z native-Z 11
HYDAE model intercomparison 12
13
Synthetic Data Used in the Experiments
14
Truth and Forecasts on Day 50
15
RMS Error in 1-Day Forecasts
16
SSH 1-Day Forecast Error Distribution
17
HYCOM specific Matlab tools from the OPeNDAP Developers The Matlab GUI developed for the 1/12 o North Atlantic Model Runs has been significantly modified.
Access the GUI from: http://opendap.org/download/ml-toolbox.html
The OPeNDAP-IDL client is now robust and can be used to access HYCOM data (no GUI yet).
Many modifications to the netCDF-OPeNDAP libraries.
18 Server4 - supporting GridFTP is now available.
The HYCOM Matlab GUI
19
Access the OPeNDAP Matlab GUIs from the OPeNDAP download webpage http://opendap.org/download Or go directly to the GUI web page http://opendap.org/download/m l-toolbox.html
20
FTP – Access
FTP://www.hycom.org/datasets 21
Access Details for June 2007
23
Community Collaborations
Working with John Caron at Unidata on testing the Forecast Aggregation server with large operational datasets Building a HYCOM .[ab] format IO service provider (ISOP) to be incorporated into the JAVA NetCDF 2.2 library Partnership with IOOS data management Caucus 24
Other Addition/Improvements
Forcing data Reference observational data sets A completely redesigned website is being planned along the lines of GODAE server Searchable mailing list, FAQ’s Links to other data repositories 25