Transcript www.godae.org

The USGODAE Monterey
Data Server
13 March 2007
http://www.usgodae.org
Phil Sharfstein
GODAE Project Manager
[email protected]
The Global Ocean Data Assimilation
Experiment (GODAE)
•
•
•
Make ocean monitoring and prediction a
routine activity in a manner similar to
weather forecasting.
GODAE will contribute to an information
system for the global ocean that will serve
interests from climate and climate change to
ship routing and fisheries.
The USGODAE Server is developed and
operated as a prototypical node for this
global information system.
The International GODAE Project Time Line
•
1998-1999 Concept
•
•
2000-2002 Development
•
•
•
Server brought on-line as a 24x7 operational system
Funding provided by ONR for the duration of GODAE
2003-2006 Operational Demonstration
•
•
•
•
USGODAE Server begins initial development and deployment
USGODAE Server sees steady growth in usage
Increased involvement in International GODAE Projects
Increased breadth of data holdings
2007-2008 Consolidation and Transition
•
•
•
ONR funding ends 2008
Server Decommissioned (If no additional funding is provided)
Goal: Seek alternative funding source to ensure that the
system remains available to support both DoD operations and
research, and the current broad range of R&D users.
USGODAE Server Schedule
• 2006-2007: Full Support
• ONR has committed to fund server operation through the
end of the GODAE Project.
• FNMOC will continue to provide full support for all server
operations and data management tasks.
• 2008: Server Decommissioned (If no additional funding is
provided)
• Transition Argo GDAC to other funded resource (if
available).
• Provide PKI data access for all DoD researchers currently
using USGODAE.
• Provide data distribution for FNMOC data sets only served
by USGODAE that are used for DoD research
(Computational Grid NOGAPS, Ocean QC, High-Resolution
Computational Grid Ocean Model/Analysis Products).
USGODAE FTP Activity Through the
Operational Demonstration Phase
Monthly Downloads
Monthly Sessions
GODAE Server
• Hosted in the FNMOC operational data center
• Provides 24x7 system support.
• Development through a partnership of FNMOC and NOAA's
Pacific Marine Environmental Lab (PMEL)
• The server hosts:
• In-situ oceanographic data
• Atmospheric forcing fields suitable for driving ocean models
• Unique GODAE data sets, including demonstration ocean
model products
• Primary Server users include:
• GODAE participants
• IOOS community
• DoD activities
• General earth and ocean science research community
• The GODAE Server has played a significant role in the
international GODAE program and the ongoing development of
ocean models
USGODAE Hardware
•
•
•
•
ONR support for the final phase of GODAE included
funding for a complete hardware upgrade.
High reliability/high availability architecture (ATOS2)
Scaled to meet the increasing server usage
Provides the resources to run the latest data serving
applications
CISCO 6509 switches
CISCOSYSTEMS
Private BackEnd Network
Front-End
Network
IBM x-346
usgodae1
IBM x-346
usgodae2
Load Balanced/Failover
Web and application
servers
IBM x-346
usgodae3
Adaptec Snap
18000 6TB
godae-snap1
Adaptec Snap
18000 6TB
godae-snap2
Development/
Mirrored 6TB Network Attached
Utillity
Storage (NAS) - Expandable to 30TB
USGODAE Data Serving Goals
• DoD Users:
• Reliable, easy access to real-time and short-term
archives of FNMOC public data sets
• Operational access to unique GODAE data sets for
NAVO and FNMOC
• General Users:
• Flexible access to data: users can quickly find and
start using the data they need.
• Provide well documented, standard Web interfaces by
adhering to community data serving standards:
• National Virtual Ocean Data System (NVODS)
• Integrated Ocean Observing System Data Management
and Communications (IOOS DMAC)
USGODAE Data Server Applications
• All Server data are available through anonymous FTP
and HTTP download
• Custom Web Applications
• Data Catalog – access and documentation links
• Preview Generator – non-gridded data
• THREDDS - Data discovery and catalog for NVODS
• Local Data Manager (LDM) - Data push for large model
output
• OPeNDAP - A Protocol that facilitates access to data and
data products via the Internet
• Including GrADS/Data, Aggregation, and CGI servers
• Live Access Server (LAS) - Human-intelligible, georeferenced browsing for all NVODS data
USGODAE Data Discovery and Catalog
NASA Global Change
Master Directory
Unidata
THREDDS
Catalog
Custom USGODAE
Data Discovery
Application
USGODAE Data Serving Example:
Argo Global Data Assembly Center (GDAC)
• Transition Goal: Continue to provide the complete archive of
quality-controlled Argo data on a Navy hosted server
• Allow operational access for NAVO and FNMOC to one of the most
important in-situ oceanographic data sets
• Argo is a major project under GODAE and the Global Ocean
Observing System (GOOS)
• Will have a global array of ~3,000 drifting temp./salinity profiling
floats sending data in real-time
• USGODAE is one of two Argo GDACs
• Hosts the complete set of quality controlled T/S profiles
• Over 300,000 profiles on-line from 9 national/regional centers
• Delayed Mode (Scientific QC’d data) on-line
• Currently 90,000 delayed mode profiles
• USGODAE Argo Data served through LAS and OPeNDAP
• Provides complete integration of Argo data with NVODS and IOOS
Monthly Argo Coverage (Aug. 2005)
Image from USGODAE Argo Data Browser
Constrain By: Time, Region,
DAC, Float ID, Delayed Mode
Data Browser provides fast
access to data previews
and easy download for
Argo data files
Argo GDAC Web Interface Output
File Retrieval and Preview
Options
Argo Data Served Through
OPeNDAP (A.K.A DODS)
• A Protocol that facilitates access to
oceanographic data and data
products via the Internet
• Regardless of:
•
•
•
•
•
data type
where the data are stored
the format in which they are stored
the user's visualization package
the user's level of expertise.
OPeNDAP Server for Argo Data
•
•
All NetCDF files on the
GDAC are available
through OPeNDAP
OPeNDAP functionality
•
•
•
•
•
Export to binary or ASCII
Subset
Subsample
Browse
Standard Interfaces and
APIs
OPeNDAP Aggregation Server for Argo
Data (and THREDDS Catalog)
•
Create virtual
collections of Argo
data without hosting
multiple copies of each
profile.
•
FUTURE: THREDDS
will provide an
OpenGIS compliant
interface to Argo data
Argo Data Served Through
Live Access Server
“human-intelligible, geo-referenced browsing
for all NVODS data”
• Basic graphics
• Well annotated
• All possible viewing planes and axes
• Readable tables of values
• Formatted subsets of data
Data
Products
• NetCDF, tab-delimited, GIS formats, others…
• Comparisons
• Overlays
• Side-by-side plots
• Difference plots (with regridding)
• and basic transformations
Analysis
Tools
LAS Interface for Argo Data
•
Select profile data by:
• Time Period
• Region
• Depth Range
•
Additionally
constrain by
• Float ID
• DAC
• Real-Time/Delayed
Mode
• QC Values
Standard LAS Output Options for Argo Data
Gaussian
Pie
Metadata
Comma Separated
HTML Text
Waterfall
Custom LAS Argo Data Products
Color coded legends list location, time, and float/profile ID
Profile
T/S Composite
Track/Location
LAS Single/Multiple Profile Selection
Select product
and variables
Select one or
more profiles
Unique USGODAE Data Sets:
FNMOC Model Forcing
•
Transition Goal: Provide public access to FNMOC data sets for
non-operational Navy/DoD use
•
•
Reduces work load on FNMOC staff fulfilling custom archived data requests
and new account requests from non DoD users
Academic and industry partners have instant access to data needed for their
projects
•
Data available through FTP, HTTP, LDM, LAS, and OPeNDAP
•
Global NOGAPS
•
•
•
•
•
•
Data on 1° application grids sufficient to force the regional COAMPS model
(~5500 files/watch)
Surface forcing fields, boundary conditions and fluxes on ½° Gaussian
computational grids custom tailored to force ocean models
Regional COAMPS – Three Regions Surrounding CONUS
WW3 – Complete Wave Watch 3 144 Hr. forecast
NCODA – Global 3D Ocean Temperature, Salinity, and Surface Height
Analysis
GHRSST – High Resolution 6 hr. SST/Sea Ice Analysis
NOGAPS Surface Pressure Data
through LAS
GRIB
→ GDS → LAS
Unique USGODAE Data Sets: Near
Real-Time In-Situ
•
Transition Goal: Provide DoD and public access to data not generally
distributed through the operational data feeds
•
Quality controlled internal model initialization data
•
Used by NRL, NPS, and NASA researchers
•
Observational data from the FNMOC Ocean QC
•
Contains all non-classified data used to initialize NOGAPS
•
Data files contain all background fields used in QC process
•
PROFILE: Fixed/drifting buoy, bathy, and PALACE float data, including
real-time Argo profiles
•
SURFACE: Ship, fixed/drifting buoy, and CMAN in-situ SST
•
ALTIMETRY: Level 2B TOPEX, GFO, and ERS altimeter SSH
•
MCSST: Level 2B AVHRR SST retrievals
•
SSM/I: Level 2B DMSP sea ice concentration retrievals
•
Meteorological (MET): Three week archive of the latest 4x daily MET
•
NAVOCEANO Level 2B Satellite Data
•
SST and SSH: Including high resolution LAC SST
In-Situ Data Access
Thumbnail Generator
(From Files)
LAS: Custom
Database
The USGODAE Server User
Communities
•GODAE
•DoD
•Non-DoD
•IOOS
Research/Commercial
Server User Community: GODAE
The Server acts as a focal point for US and international
GODAE projects, data, and information.
•Data for GODAE
• Custom tailored model forcing fields
• Near real-time In-situ atmospheric and oceanographic data
• Operational high-resolution model and analysis fields for
GHRSST
•GODAE
Project Hosting
• Ocean QC Intercomparison, Equatorial Intercomparison, Model
Forcing Intercomparison, LAS GODAE Modelers Output Site.
•GODAE
Information
• Meeting agendas and presentations, project documentation
DoD Usage of the USGODAE Server
The DoD community is one of the largest server users.
•
•
•
•
56.5% of HTTP Downloads (5,213,704 files/year)
10.8% of FTP Downloads (3,690,638 files/year)
Top DoD Users:
1.
2.
3.
4.
Navy Operational Centers: Naval Pacific Meteorology and
Oceanography Center, NAS Point Mugu, Sasebo Naval
Base, US Pacific Command, Naval Oceanographic Office
Navy R&D: NRL Monterey, NRL Stennis, Naval
Postgraduate School
Other DoD: Air Force Weather Agency, US Coast Guard,
Army Research Labs
DoD Industry Partners: Applied Science Associates (Coast
Guard), Lockheed Martin (Air Force), Barrodale
Consulting Services (Navy), SAIC (Army)
Server User Community: General
R&D
•
The USGODAE Server is the primary source of
FNMOC model and observational data for a broad
range of research projects and institutions.
•
Many funded research projects that will continue
beyond the GODAE transition period rely on
USGODAE as their principal data source.
•
The Server provides a framework and data archive
for DoD R&D that will need to be replicated at
each new (or existing) project.
Non-DoD R&D and Commercial Users
.gov
•
•
•
NOAA: FSL, NODC, NDBC, AOML, PMEL, GFDL
NASA: JPL, GSFC
.org
•
•
UNESCO, Alaska Ocean Observing System (AOOS), Southeastern
Universities Research Association Coastal Ocean Observing and
Prediction Project (SURA/SCOOP), Gulf of Maine Ocean Observing
System (GOMOOS), Columbia Scientific Balloon Facility,
Seaturtle.org
.edu
•
•
•
Several hundred institutions retrieving over 150,000,000 files each year
through FTP, HTTP, and LDM.
Partnered with Unidata/UCAR to provide data for educational
research.
.com (Excluding DoD Partners)
•
•
Orbimage, Accuweather, Weather3000, Surfline, Ocens, Weathernews
Server User Community: IOOS
The bridge between GODAE and IOOS
•
The USGODAE Server currently provides a wide range
of real-time data, and oceanic/atmospheric models that
are essential to the success of several IOOS projects.
•
The Server is one of the primary real-time data servers
for global in-situ oceanographic data.
•
The USGODAE Server is designed and operated to
adhere to the data serving standards specified by the
IOOS/DMAC.
•
As a partner in IOOS, FNMOC can make many of its
higher-resolution operational model fields covering the
waters surrounding CONUS available to the IOOS
community.
IOOS Participants Served by the
USGODAE LDM Server
(Each institution served over 400,000 files/day)
SCOOP
•
•
Bedford Institute of Oceanography, Louisiana State University, University of
Florida, University of Miami, University of North Carolina, Virginia Institute
of Marine Sciences, University of Alabama at Huntsville
SEACOOS
•
•
North Carolina State University,University of North Carolina Charlotte,
University of North Carolina Chapel Hill, University of Miami, Florida State
University, University of South Florida (FTP/HTTP only)
GoMOOS
•
•
Bedford Institute of Oceanography, University of Maine
GCOOS
•
•
Texas A&M University
AOOS
•
•
AOOS LDM/Data Server, University of Alaska Fairbanks
IOOS Products from USGODAE Server Data
AOOS imagery from
USGODAE satellite data
collection
Wave models for GoMOOS,
SCOOP, and SEACOOS forced by
COAMPS from USGODAE LDM
FNMOC Data and Models Currently
Provided for IOOS
Global Wave Watch 3
•
•
1 degree resolution, forecast to 144 hours
COAMPS Application Grids
•
•
•
Data for the three regions surrounding CONUS
Currently used to force several IOOS wave models
Global NOGAPS Application and Computational Grid
•
•
Computational grids provide the most accurate fields for
forcing models in the coastal zone.
Daily High Resolution Global Ocean Analysis
•
•
2161x1051 34 Levels (0-5000m)
High Resolution Global SST/Sea Ice Analysis
•
•
3241x2441 6 hourly
FNMOC Models Available for IOOS
(all pending FNMOC approval for public release)
High Resolution Regional Wave Watch 3
•
•
•
•
Covers East Pacific, West Atlantic, and Central America
(Gulf of Mexico)
Forced by FNMOC Regional COAMPS Models available
on Server
27 km resolution, forecast to 48 hours
COAMPS Computational Grids
•
•
•
Data served on the original computational grids for three
regions surrounding CONUS
Provides the most accurate fields for forcing ocean models
in the coastal zone
USGODAE COAMPS and WW3
Coverage for IOOS
Regional COAMPS and
WW3 are available for
the three regions
surrounding CONUS:
EPAC, CENTAM, and
WATL
FNMOC Regional Atmospheric and
Ocean Models for IOOS
EPAC Regional WW3
COAMPS Comp. Grid
Lambert Projection for EPAC
Summary
•
Thanks to ~8 years of funding from ONR, the USGODAE
Monterey Data Server has been developed into an
invaluable resource for both the R&D community and the
Navy operational community.
•
As planned from the outset, ONR funding for the Server
will end at the beginning of FY08.
•
To ensure continued USGODAE Server availability we
must enhance its role as an R&D system, supported under
the projects that have made the Server a key part of their
data serving infrastructure.
•
If no additional funding is provided, the USGODAE Server
will be decommissioned in 2008.