Transcript A Client Application for Real Time NOMADS to Disseminate NOAA’S

EMC
A Client Application for Real Time
NOMADS to Disseminate NOAA’S
Information Data Base
NCEP/Environmental Modeling Center
Jordan C. Alpert/NCEP/EMC
Jun Wang/SAIC
[email protected]
[email protected]
21st Conf. on IIPS, 17.5, 85 Annual AMS Meeting, 9-13 January 2005, San Diego, California
where the nation’s climate and weather services begin
Motivation for NOAA data services:
Administratively

“Disseminate and deliver NOAA’s information and services.” (FY0711 Baseline Assessment)

“The National Weather Service should make its data and products
available in Internet-accessible digital form” is Recommendation 5 of
the National Research Council’s Fair Weather: Effective Partnerships
in Weather and Climate Services.

The Advanced Weather Interactive Processing System and National
Digital Forecast Database for weather service Operations
modernization, (read AWIPS and NDFD) need a scalable and
modularized data dissemination system.
A framework and working prototype can be found in NOMADS.
For the Project that Needs to Serve Data
In the scientific and modeling community including federal
Agencies.
Serving environmental data sets from NOAA and other organizations
in a way that is more cost effective and versatile:
 In terms of dollars, because NOMADS is scalable and open source
– Inexpensive modular hardware components are used.
 In terms of computer resources, because the right jobs are in the
right place -- the server gets the server jobs and the client (user)
gets the client jobs.
 Best use of available bandwidth by having users get what they
need instead of transmitting entire files.
Participants who embrace the protocols can say “NOMADS inside”
and participate to influence future developments.
For the Project that needs to Serve Data and
for the Users who Need to Receive Data
NOMADS offers guidelines, server software and
client examples to participants.
Real Time NOMADS component at NCEP:
 Real Time NOMADS at NCEP serves operational model
data sets, GFS, Eta, Ensembles, RSM, RUC etc...
 “Slice and Dice” ftp with ftp2u application client,
 Plot (Great) Displays with pdisp, application client,
 GrADS-DODS (OPEN-DAP and GDS) access using open
source and XML, to make our server disk storage appear
as if it is attached your PC!
The GDS framework allows many users to have direct access
to the model data base to make the products users need at
the users location.
Using the NOMADS client example applications:




The pdisp (Great) Displays: plots a part of the data base with standard
(weather) maps, cross sections, animation and hovmuller diagrams
across the selectable 5 dimensions (time, height, longitude, latitude and
parameter space).
The ftp2u script can pare down large high resolution model files
including area subsets, parameter space (variables) physical space
(longitude, latitude, height level), and over time.
The URL (complete web address) produced by this process is available
to users to be re-used in a suitable Unix script with a non-interactive web
based download program (like wget or wwwgrab) to control the data
flow,
Data can be obtained at regular intervals by utilizing these URL
constructs (Unix script) in cron for automated data flow control.
This software is used at NCDC NOMADS.
Users have complete and customized
control of their data flow needs.
Users can gain access to the server data base directly over
network resources.



The client examples presented can be executed on the server with direct access
to the data base or they may run equally well on the users computer (the client)
accessing the OPEN-DAP (DODS) server.
Most use of NOMADS is made through these client applications but the
OPEN-DAP/DODS servers are accessible through a number of commercial
clients: MATLIB, IDL, and freeware clients, e. g., GrADS.
An internet browser can be used as well or a non-interactive web based
download program (like wget or wwwgrab).
To see how OPEN-DAP/DODS works we will use the NCEP global
ensemble model forecasts data on the server and create the
constrained queries to produce event of user selected probabilities.
Ensembles and Event Probabilities





NCEP ensembles are constructed from many (10) independent
model integrations by changing the initial conditions from
adding (p1-p5) and subtracting (n1-n5) the fastest growing
global modes from 6-h forecasts. 11 in all with the control.
Ensembles attempt to span the space of possible forecasts.
Ensemble members are equally probable forecasts – if not, we
apply a suitable normalization.
Probability estimates are defined simply as the percentage of
forecasts of the total number that satisfy a specified weather
event.
We apply this to weather elements like surface temperature,
wind speed or precipitation at a location and future model
forecast time.
Screen shot of the web page containing prompts and user
entered responses for the probability of frost at day 4 ½.
Web page showing the URL queries of the ensemble database. The
(Probability of frost was 37%.)
The OPEN-DAP/DODS Constrained Query
http://nomad5.ncep.noaa.gov:9090/dods…
URL contains the the server address including the port (:9090)
/enshires/archive/ens20041014/ensc0_00z_1x1 …
Location of XML data and metadata descriptions, containing all the
properties digitally describing the ensemble data from GRIB files.
.ascii?tmin2m[18:18][130:130][255:255]
The server unpacks GRIB files, finds the minimum 2 meter temperature,
tmin2m, but sends only data to the client specified by the constraints in
the square brackets for time and location (and level if applicable):
[18:18]
indicates the 4 ½ day forecast (3-hr intervals from IC)
[130:130]
# of lats north of the South Pole on a 1x1 degree grid.
[244:244]
# of lons measured eastward from Greenwich meridian.
An OPEN-DAP/DODS (XML) metadata file description
(the .info page)
GrADS-DODS Server - info for /gfs/rotating/avn_00z : dds das
DODS URL:
http://nomad2.ncep.noaa.gov:9090/dods/gfs/rotating/avn_00z
Description:
AVN fcst starting from 00Z28dec2004, downloaded Dec 28 05:00 UTC
Documentation:
none provided
Longitude:
0°E to 359°E
(360 points, avg. res. 1.0°)
Latitude: -90°N to 90°N
(181 points, avg. res. 1.0°)
Altitude: 1000 to 10
(26 points, avg. res. 39.6)
Time: 00Z28DEC2004 to 12Z04JAN2005
(61 points, avg. res. 3.0 hours)
Variables:
(total of 109)
absv ** absolute vorticity [/s]
no4lftx
** surface best (4-layer) lifted index [k]
no5wavh
** 5-wave geopotential height [gpm]
acpcp
** surface convective precipitation [kg/m^2] (hidden)
apcp
** surface total precipitation [kg/m^2] (hidden)
cape
** surface convective avail. pot. energy [j/kg]
….
Value Added Products




Use wget or wwwgrab non-interactive download programs to send
queries or get information about the datasets.
Aggregate Services from the Distributed Metadata Server (DIMES)
which operates (reads .info pages) on nomad servers and catalogs the
data sets for users (clients) to search by time, location, variable and file
name.
More complicated calculations than event probabilities can be made
with this data on the client, for example, one could obtain the wind
field at CONUS data points and calculate the 3-D divergence and
compare with severe weather – a subject of another report (18.5).
A second ensemble probability example immediately follows to add a
loop over all forecast times. 180-h forecasts on 6-hr intervals for 11
ensembles is 31 X 11 = 341 queries for each scaler variable.
Timing and Efficiency




A few queries ~5 per second, 50 seconds for the 341 queries of one
value.
The time for a File open/close is much longer than I/O such that a
query of a thousand points (numbers) takes the same time as a query of
one.
When the number of values is large, the time for the user to obtain the
data is constrained by the bandwidth. When one value per query is
needed then the I/O overhead is the limiting factor.
The optimal use of server and client is written into the client and server
applications.
Summary





NOMADS servers are a prototype system using the client-server
framework now accepted to disseminate the data base results of a
number of NOAA projects.
NOMADS framework is scalable and can run over public or private
connections as it uses the common software of the internet.
NOMADS provides a framework for efficient use of bandwidth by
utilizing applications to transmit only what is needed and customize
resources use.
Applications using ftp2u, pdisp or OPEN-DAP/DODS can be used by
a commercial or public enterprise or value added retailer to reduce
programming time and making products available in remote locations.
NOMADS framework lends itself well to NDFD and AWIPS
operational systems.
Web Pages
http://nomad1.ncep.noaa.gov
Contain Operational models, GFS, Eta,…, Re-analysis I&II -- Backup to nomad2
and alias to nomad3
http://nomad2.ncep.noaa.gov
Contain Operational models, GFS, Eta,…, Re-analysis I&II
http://nomad3.ncep.noaa.gov
Contain Climate runs from the CFS project
http://nomad5.ncep.noaa.gov
Contain Global Ensembles and for the client applications presented here in:
add cgi-bin/var/ensprob1.pl or cgi-bin/var/ensprob2.pl
Coming soon: nomad 4,6,7 with land surface project data base
Add “/ncep_data” to any of the above to find the “Data Table” with <DODS>,
<pdisp> and <ftp2u> access.
DIMES Aggregate Servers for NCEP NOMADS:
http://spring.scs.gmu.edu:8199/nomad