Transcript Marine Modeling & Analysis data and Real Time NOMADS (NOAA Operational Model Archive

Marine Modeling & Analysis data
and
Real Time NOMADS
(NOAA Operational Model Archive
and Distribution System)
Jordan C. Alpert [email protected]
Jun Wang
[email protected]
Bhavani Balasubramaniyan [email protected]
MMB 11/08//05
where the nation’s climate and weather services begin
NOMADS – What is it?
For Data producers – e.g., NCEP models and projects…
A service orientated architecture prototype to provide
distributed format neutral access to model data.
 Data distribution software, format independent and
description methodology,
 Documentation and organization framework,
 Forum to plan and organize,
… for university, federal agencies and organizations, a
framework for the dissemination of their data sets.
RT-NOMADS at NCEP (jargon)
Consists of
Distributed Ocean Data Server - OPeNDAP
(DODS) based on metadata file descriptors and
XML under the GrADS Data Server (GDS).
 Now known as the Open-Source Project for a
Network Data Access Protocol.
 NCEP NOMADS servers are the intermediary
between NCEP operational computers and
NCDC NOMADS archive servers.
 3Yrs of operation, many partners, countless
users.
RT-NOMADS is a service for science, development
and public access for data users.
Tools on the server – Client Application Examples:
ftp2u, http(fast ftp) or plain ftp
 Pare down large file sizes of high resolution initialization
and model results.
 (re-) Group different data sets to create needed products –
such as initialization files for model development.
 Subset the data sets in parameter space,
 Subset the data sets in physical space and time.
(subset the grid in 4 dimensions)
The user’s resulting URL address containing user control
variables, can be placed in a suitable script with an
interactive web download command and placed in cron for
custom automatic data flow.
The Server – Client distributed relationship:
A Common Sense Approach to Data Distribution
Tools on the server – Examples of Clients (con’t):


ftp2u, Great Displays, and GDS allow sub-setting
of data by parameter, by 3D region, and by time.
Fast ftp using Curl comands (http) see
http://www.cpc.ncep.noaa.gov/products/wesley/fast_downloading_grib.html

“Great Displays” (pdisp) A display program operating
from the same database as ftp2u to display any part of the
NCEP data base by time, region, vertical level and
variable.

GDS or DODS or OPeNDAP …
ftp server is present for data sets.

Tools on the server –
Client Application Examples:
pdisp (“Great” displays)



Many features: Plot type: map, time series, lat/lon vs time
and Animation.
User selected projection map areas, contours, colors, units,
averages ….
URL address can be copy-pasted into browser or unix
script using non-interactive web download command with
user changes. Create products and embed results in other
web pages.
The NCEP server (nomad5) data entry table
The GrADS-DATA Server (GDS)





NOMADS participants serve their data sets through a
client-server relationship, that is, the data sets are internet
ready and the display is done by the client.
GDS combines both GrADS, a freeware client and DODS
(OPeNDAP) server to exchange data in many formats with
http protocol.
This means that server data can appear to the user or
client application as a local file!
Like Network file system over the web!
DODS requests are made by many freeware and
commercial high level language clients.
Simple http queries to the DODS server can create value
added products.
Data Application Protocol (DAP) using…
…and any browser
… A GrADS client example script that animates 3 years of reanalysis
height fields and places the date in the title and a color bar below. A user
with GrADS freeware can run this script. Note the first line, as directed
by metadata description in next slide….
Sdfopen http://nomad3:9090/dods/reanalysis-2/pgb/hgt
t=1
'set gxout shaded'
while (t <= 1068)
'set t 't
'q time'
date=subwrd(result,3)
'draw title hgtprs 'date' 't
'd hgtprs'
'/home/wd23ja/grads/lib/cbarn.gs'
t=t+1
endwhile
Metadata description includes data set location URL address
and all other information about the file and variables.
A commercial or freeware client is not necessary to access data...
Using http to extract data from the server:
http://nomad3.ncep.noaa.gov:9090/dods/eta/eta20030527/eta_
00z.ascii?tmpprs[0:0][0:0][130:130] [290:290]
the order of the square bracked values is:
[time][level][lat][lon]
where lat is measured from the south pole (0) to NP (180)
In units represented by the metadata descriptor file.
(Use a “wget” and the URL in cron or a cgi-bin script
provides needed values.)
MATLAB: A Commerical Client
server=‘nomad2:9090’;
directory=‘/dods/gdas/rotating/’; fext=‘ascii’;
model=‘gdas’; datestr=‘YYYYMMDDHH’; varname=‘hgt’;
varlevs=‘1’;
Arrc=[‘[‘,t1,’:’,t2,’][‘,nlev1,’:’,nlev2,’][‘,ilat1,’:’,ilat2,’][‘,ilon1,’:’,ilon2,’]’];
urlstring=[http://’,server,directory,model,datestr,’.’,fext,’?’,varname,arrc];
url = java.net.URL(urlstring); % connect to web page
Try
stream = openStream(url)
ireader = java.io.InputStreamReader(stream);
breader = java.io.BufferedReader(ireader);
Line = readLine(breader);
disp(line);
NOMADS Data Sets at NCEP
Metadata descriptor files describe all aspects of the data
sets and are created uniquely from the headers of GRIB
files. Data sets at NCEP:
Wave model forecast
6 hours
RTOFS model forecast daily
Example of pdisp (“Great” displays) on the Wave model Forecast
enp_00z.ctl HTSGWsfc 1 00Z08nov2005
http://nomad5.ncep.noaa.gov/cgibin/pdisp_wave.sh?ctlfile=enp_00z.ctl&ptype=map&povlp=noovlp&psfile=off&v
ar=HTSGWsfc&level=1&op1=none&op2=none&hour=00Z&day=08&month=no
v&year=2005&proj=custom&lon0=170&dlon=60&lat0=0&dlat=60&type=shaded&cint=def&white=def&fwrite=off
&plotsize=800x600&title=&dir=%2Fenp%2Fenp20051108
RT NOMADS Server at NCEP
http://nomad5.ncep.noaa.gov
 http://nomad3.ncep.noaa.gov
 http://nomads6.ncdc.noaa.gov

Demonstration
cat pdisp_script.sh
wget -o result_log -O- "http://nomad5.ncep.noaa.gov/cgibin/pdisp_aofs.sh?ctlfile=rtofs_atl.ctl&ptype=map&povlp=noovlp&psfile=o
ff&var=WTMPsfc&level=1&op1=none&op2=none&hour=00Z&day=03&
month=dec&year=2005&proj=atlantic&lon0=280&dlon=50&lat0=60&dlat=60&type=shaded&cint=def&white=def&fwrite=off&plotsize=800
x600&title=&dir=%2Faofs20051201"
The successful pdisp client/user interaction which produced the
URL address/command in quotes above, is placed
(copy-pasted) into the script file pdisp_script.sh as
the object of an interactive web download command (like wget).
Pdisp returns the display plot
as a web page as shown on
<head>
the previous slide. The html
<title>Display WTMPsfc: NCEP RTOFS(Atlantic) Model</title> Text on this page results from
resending the same URL
</head>
command with the
<body bgcolor="#ffffff">
interactive web download
<h3 align="center">NCEP RTOFS(Atlantic) Model rtofs_atl.ctl</h3> program wget.
./pdisp_script.sh
<h3 align="center">WTMPsfc 1 00Z03dec2005</h3>
<FORM NAME="form1" action="pdisp_aofs.sh">
<center>
<img src="/tmp/ATEST12963.png" alt="plot">
The plot is on the server with a
unique file name given in the
URL address commands:
/tmp/ATEST12963.png
</center>
<pre>
image was generated
We use a second wget to get the
png file (on the next slide).
</pre>
<small><p>&nbsp&nbsp <p align=left>Problems with the datasets, contact:
[email protected],[email protected]<br>
pdisp version v0.9.6.7: [email protected],[email protected]
</body>
#!/bin/sh
set -x
locnam=`/home/wd23ja/info/rtofs/pdisp_script.sh |grep
"\/tmp\/"|cut -d\" -f2`
echo "locnam $locnam "
wget -O out.png -o log1.txt
"http://nomad5.ncep.noaa.gov${locnam}"
First execute the pdisp_script.sh script above finding the
line with the characters “/tmp/” in it and from it cut away
the server directory/filename, /tmp/ATEST12963.png and
send that location in another wget to get the .png picture file
into the file out.png for further viewing.