Transcript Distribution of Real-Time and Retrospective NCEP Model Data Sets RT-NOMADS

Distribution of
Real-Time and Retrospective NCEP
Model Data Sets
RT-NOMADS
NOAA Operational Model Archive
and Distribution System
Jordan C. Alpert [email protected]
Jun Wang
[email protected]
Glenn Rutledge [email protected]
Alpert 9/5/03
where the nation’s climate and weather services begin
NOMADS is an agreement between
agencies who participate to adhere to
a set of common …




Data and observation distribution software, format
independent and description methodology,
Documentation and organization framework,
Forum to plan and organize
Funding avenue to direct $.
… for university, federal agencies and organizations, a
mechanism to obtain support for the dissemination
of their data sets.
Participants
U.S.NOAA, DOE, NASA,
NCAR, Unidata, COLA,
OPeNDAP Project,
& Several Universities
InternationalCEOS
NERC
ESG
UK BADC
BOM /
Dept. of State via CAP
NOMADS
and the NCEP Real Time Component
A solution for Recommendation 5 of “Fair Weather:
Effective Partnership in Weather and Climate Services”




Wider information dissemination through web pages and
other avenues.
Framework to create specialized and value added products
needs access to weather data.
Facilitate communication among commercial, public,
research community and federal agencies.
NOMADS is a system for participants to interact with
others using common protocols and formats. Existing
differences in details are mollified by software.
A NOMADS Goal:
Specify 24/7 servers for NOAA Operational Products.



The server technology employed by NOMADS (OPeNDAP or DODS) provides clients the ability to see all or
part of the data base as if it were local to them.
The distribution of NOAA’s National Digital Forecast Data
Base could benefit from this technology as in the NVODS
project.
NOMADS is a service that encourages NOAA projects and
Operations to distribute and document data records for
science and the public.
RT-NOMADS at NCEP
Consists of




Distributed Ocean Data Server - OPEN_DAP (DODS)
based on metadata file descriptors and XML.
Web based ftp, the NCEP ftp2u client and “Great
Displays” (pdisp), a grid analysis and display client.
Model initialization ingredients, “The minimum set”
for model rerun/retrospective analysis ready
observations & restart files.
The Real Time NCEP Operational model run history.
RT-NOMADS is a service for science,
development and public access.
To distribute NCEP’s Operational data set to
researchers and the public.
To provide server software to NOAA projects who
need to serve their data through web services.
 NCEP NOMADS servers are the intermediary
between NCEP operational computers and NCDC
NOMADS archive servers.
Tools for Users on the server
Pare down large file sizes of high resolution
initialization and model results.
 (re-) Group different data sets to create
needed products – such as initialization files
for model development.
 Subset the data sets in parameter space
 Subset the data sets in physical space
(subset the grid in 4 dimensions)

NCEP Real-Time NOMADS Server








Real time.
Operational Suite of Model products.
Forecast information for model comparison.
Operational initialization and observations.
Maintenance of codes.
Utilities for data manipulation and grid
transformation & sub-setting.
Documentation.
Serves as transfer point to NCDC for NOMADS
archive of reruns and retrospectives.
The Server – Client distributed relationship:
A Common Sense Approach to Data Distribution
Examples of Clients :



ftp2u, Great Displays, and GDS allow sub-setting
of data by parameter, by 3D region, and by time.
Allows for results with low band width! Eg.,
NASA/GSFC takes 1 hour to ftp entire GFS
(AVN) forecasts or 13 seconds to download their
desired fields (ftp2u).
GDS: NOMADS participants serve up their data
sets through a client-server relationship, that is,
the data sets are internet ready and the display is
done by the client.
Extending NOMADS to Real Time
The A in NOMADS is for Archive.
 NCEP historical cases for rerun/retrospective case
studies of ocean, atmosphere and climate.
 NCEP Real Time model output in the NOMADS
framework … so there is a seamless access to
these important data sets at NCEP and at NCDC!
 Use of benchmarks to characterize system loads in
order to determine costs and give NOAA
administration the information it needs to triage
data resources.
RT-NOMADS provides the framework for long term
support to be available to keep up with modeling
advances.

RT-NOMADS is a prototype
Goals:



Provide proof of the concept that distributed systems
can serve data sets across agencies for modeling
science, the university community and .
Specify an Operational system for NOAA/TOC,
Provide current state of the art operational capability
for serving observations and data to Forecast Offices,
Data Application Protocol (DAP) using…
… GrADS client …. commands
Sdfopen http://nomad2:9090/dods/reanalysis-2/pgb/hgt
t=1
'set gxout shaded'
while (t <= 1068)
'set t 't
'q time'
date=subwrd(result,3)
'draw title hgtprs 'date' 't
'd hgtprs'
'/home/wd23ja/grads/lib/cbarn.gs'
'swap'
t=t+1
endwhile
Getting data without GrADS or a commercial
client:
http://140.90.198.156:9090/dods/eta/eta20030527/eta_00z.ascii?tmpprs[0:
0][0:0][30:130] [290:290]
#the order of the square bracked values is:
# [time][level][lat][lon]
# where lat is measured from the south pole (0)
# to equator (90) to NP (180)
(Use a “wget” and the URL with the ascii response in cron or a cgi-bin
script provides the needed number(s) when and where needed.)
NOMADS clients at NCEP
“Great Displays” (pdisp) and ftp2u
by Ebisuzaki and modified by Wang







Real Time Operational model data sets from GFS, and
ETA from present to one month ago.
Attention to accumulation variable differences and high
and low resolution.
Analysis pgb, restart files.
Recent (“real time”) climate runs, ie., RSM.
Observations (soon).
Reanalysis-2 (and 1)
AMIP type runs (Suru’s finest runs eg., 28 and 64 levels
1978-2002), sudden warming, ….
MATLAB: A Commerical Client
server=‘nomad2:9090’;
directory=‘/dods/gdas/rotating/’; fext=‘ascii’;
model=‘gdas’; datestr=‘YYYYMMDDHH’; varname=‘hgt’;
varlevs=‘1’;
Arrc=[‘[‘,t1,’:’,t2,’][‘,nlev1,’:’,nlev2,’][‘,ilat1,’:’,ilat2,’][‘,ilon1,’:’,ilon2,’]’];
urlstring=[http://’,server,directory,model,datestr,’.’,fext,’?’,varname,arrc];
url = java.net.URL(urlstring); % connect to web page
Try
stream = openStream(url)
ireader = java.io.InputStreamReader(stream);
breader = java.io.BufferedReader(ireader);
Line = readLine(breader);
disp(line);
NCEP Data Sets
Archives at NCDC
NOAAPORT Operations.
 Continuation of NCEP real time model
component.
 NCEP model initialization ingredients, “The
minimum set” for model rerun/retrospective
analysis ready observations & restart files.

Model Data Requirements
from CLASS
NOAAPort
Gridded Output
North American
Regional Reanalysis
Current Archive:
NOAAPort:
2TB/yr
GDAS/SSI:
400Gb/yr
NARR
10Tb
Total
12.5Tb
FY04
NCEP Hi-Res:
Total
+10TB/yr*
22.5Tb
* 5 year retention only
HDSS
23TB/yr
NCEP
Global Assimilation
Spectral Statistical
GDAS O/I
FY03
NCEP Hi-Res
Eta/Global
and NARR
Legend
Green- Existing Archive
Blue- Expected FY03/04
Rutledge and Alpert 6/03
Model Data Req’s - NWP and GCM’s
Volume Tb/Yr
Cumulative (Tb)
250
140
120
200
100
80
60
40
GCM
Reanal
NWP*
NWP*
Reanal
GCM
150
100
50
20
0
'04 '05 '06 '07
GCM- Global Climate Model
Reanal- N. Am. Regional Reanalysis
NWP- Numerical Weather Prediction
0
'04
'06
'08
'10
* 5-year rotating NWP archive
RT NOMADS Server at NCEP
Two 1TB RAIDZONE servers
 http://nomad2.ncep.noaa.gov
 http://nomad1.ncep.noaa.gov
(backup)
 Documentation web page under construction
 New disk space on order….

Non-Operational “Projects” can become NOMADS
participants to display their data sets:
NOMADS gives guidance to:
Install scripts and clients, ftp2u and “Great
Displays” (pdisp) for a GDS framework,
 Setup GDS XML file – see example on web
page, for example,…
 Including GDS administration with updates
and new additions,
 “newxml” update scripts – to automatically
add new directories and files.

(RT) NOAMDS data sets ….
One data set (e.g, GRIB) provides the data
for all access applications.
 GraDS control files (grib2ctl) and index
files (gribmap) create unique metadata
description and random access.
 GDS can read HDF, NETCDF* as well as
other formats.

NOMADS
and the NCEP Real Time Component
A solution for Recommendation 5 of “Fair Weather:
Effective Partnership in Weather and Climate Services”


NWS should make its data and products available in
Internet-accessible digital form.
Information held in digital databases should be based on
widely recognized standards, formats, and metadata
descriptions to ensure that data from different observing
platforms, databases and models can be integrated and
used by all interested parties in the weather and climate
enterprise.