Transcript WDSS-II Development

Setting up a real-time system
[email protected]
National Severe Storms Laboratory
Norman OK, USA
http://www.wdssii.org/ :: Platforms, Real-time Monitoring
A quick word about hardware…
Running WDSSII is an intensive process:

General hardware/software specs
(from http://cimms.ou.edu/~lakshman/WDSSII/software/platforms.html)
Item
Processor
Minimum Spec
for research
version
One Pentium IV or
better
Random Access 512MB
Memory (RAM)
Regional-scale
(1000 km x 1000
km) algorithm
machines
CONUS-scale
algorithm
machines
Display -only
machine for realtime applications
dual 32-bit Xeon or
Athlon
dual 64-bit Opteron or single Xeon or better
Xeon
2GB per processor
(ideally 4 GB per
processor)
8 GB per processor
(ideally 16 GB per
processor)
2 GB
Storage (I/O)
1GB per data case
RAID 0 (dedicated RAID) RAID 0 (dedicated
-- 4 GB
RAID for each
machine) -- 20 GB
1 GB
Video Card
32MB graphics card
supported by Linux
None
128MB NVidia
GeForce4 for display
machine
21 July 2015
None
2
Generally Speaking…
Get the data


21 July 2015
Process the data
Real-time
Archive
3
Obtaining Data: Archive Data
 Radar:
º
Level II & III from NCDC
http://hurricane.ncdc.noaa.gov/pls/plhas/has.dsselect
º
OKC TDWR*#
 Model:
º
RUC (20km) Hourly Analyses*
http://www.archive.arm.gov/cgi-bin/arm-archive
 Satellite:
º
6 Bands*
 Mesonet:
http://www.mesonet.ou.edu/data/basic/mesonet/mdf/
* Data available “in-house” at NSSL.
# Data only available through NSSL
Contact [email protected]
21 July 2015
4
Obtaining Data: Real-time
 Typically via Local Data Manager (LDM)
º
See your system administrator or LDM documentation regarding
install/setup of LDM
 PQACT.CONF File
Format: feedtype TAB prodIdPat TAB action TAB [arg ...]
(from http://www.unidata.ucar.edu/software/ldm/ldm-current/basics/pqact.conf.html)
 Typical WDSSII PQACT.CONF File:
Data
Source
Name
of file
WSR-88D
(NWS radar
network)
pqact.conf
Entry
NEXRD2 ^L2-([^/]*)/(....)/([0-9][0-9][0-9][0-9][0-1][0-9][0-3][0-9][0-2][0-9][0-5][0-9][0-9][0-9])
PIPE /home/ldm/bin/nexradII_bz
/home/ldm/data/nexradII/\2/\3.raw
EXP ([^/]*)/(OKC)([0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9](-)[0-9][0-9][0-9][0-9][0-9][0-9](-)[0-9]+(.)[0-9][0-9](TDWR (FAA
pqact.conf )[0-9]+(-)[0-9]+(-)[0-9]+(.)(bz2))
weather radar)
FILE /data/realtime/tdwr/Raw\2/\2\3
RUC (FSL
NMC2 [^/]*/RUC2/#252/([0-9][0-9][0-9][0-9][0-1][0-9][0-3][0-9][0-2][0-9][0-5][0-9])/F000/([A-Z]*)/([^/]*)(!) ([0Rapid Update pqact.conf 9][0-9][0-9][0-9][0-9][0-9])
Model)
PIPE /home/wdssii/WDSS2/bin/gribToNetcdf -o /data/realtime/model/ruc2 -P
21 July 2015
5
Data Processing
Raw Data
Raw Data
Processing
Base Data
Base Data
Processing
Processed Data
(Final Output)
21 July 2015
6
Data Processing
Process the “Raw” Data
1)
•
Results in “Base” Data
º
Reflectivity, Velocity, Spectrum Width, Pressure Levels,
Cloudtop Temperature, Wind Direction, etc…
2) Process the “Base” Data
•
Run Algorithms on “Base” Data
º
21 July 2015
“Azimuthal Shear”, “Reflectivity at Freezing Level”, “TVS
Detection Algorithm”, etc…
7
Processing Data
First: rssd

What it is:


rssd is the server process
that provides notification
to your algorithms and
tools.
WDSS-II algorithms use
rssd to provide these
capabilities:


21 July 2015
Read data files on remote
machines.
Let algorithms know when the
data they want is available

Setup:

Set environmental
variable “RMTPORT”
Eg. export RMTPORT=50000

Set rssd.conf

Syntax:
Client: computers allowed access
Path: accessible local paths

Example:
Client: cairo.protect.nssl
Client: 172.19.*.*
Path: /data/radar
Path: /data/raid/satellite
Path: /tmp
8
Processing Raw Data:
Command-line Executables

Note: with any executable found
in the WDSSII bin directory,
running the program with no
arguments will provide a
“usage” commentary:
To Process
 Radar (Level II / TDWR)


Satellite


gribToNetcdf
Mesonet


satIngestor
Model (RUC – GRIB format)


ldm2netcdf
mesonetIngest
There are others – check
Tools listing on wdssii.org
21 July 2015
9
Processing Base Data
Remember: processing base data means “running algorithms”.

Processing base data is syntactically the same as processing raw
data.


Again, if you are unsure of the arguments needed to run a WDSSII
executable, run it with no arguments and the “usage” will be
displayed.
Inputs for processing base data are typically the outputs from the
raw data.



Look for code_index.lb (binary file), or code_index.xml (XML file)
Depending on which file type you use, specify or
“xmllb:/path/to/code_index.lb” or “xml:/path/to/code_index.xml”
The path to the code_index file is usually the same path specified with
the “-o “ argument when processing the raw data
21 July 2015
10
Processing Data: Command-line Steps

To summarize simple data processing:
1.
2.
Obtain raw data
Process raw data
1.
2.
3.
3.
Process base data
1.
2.
3.
4.
21 July 2015
“ldm2netcdf” – radar
gribToNetcdf – model data
etc.
w2qcnn
scit3D
etc.
Display output
11
Processing Data: Distributed Processing

You are not limited to processing data (raw or base) on only
1 computer.


If you have multiple computers with WDSSII installed on them, you can
distribute the processing.
If you are performing WDSSII processing on multiple machines, you can link
them together (so that the output of one process feeds into the input of
another process)

First make sure that your rssd.conf is set up to allow
communication between computers and associated
directory paths
 Add the computer name to the path of your WDSSII
program. I.e.:
filetype:computer:/path/to/code_index
xmllb:wdssiicomputer1:/home/wdssiiuser/KTLX/code_index.lb
21 July 2015
12
Processing Data: w2algrun

Often a user wants/needs to process multiple sources
of data



Processing of raw data from multiple radars
Multiple sources (RUC, radar) needed to run a complex
algorithm like SCIT or HAD
Can be accomplished at the command line by using
quotation marks “”:
scit3D –i “xmllb:/tmp/KTLX/code_index.lb xmllb:/tmp/KINX/code_index.lb
xmllb:/tmp/KICT/code_index.lb xmllb:/tmp/RUC20KM/code_index.lb” –o /tmp/multi


This way can quickly become cumbersome and
confusing
Use w2algrun
21 July 2015
13
Processing Data: w2algrun

What is w2algrun?


A group of scripts (programs) that helps the user to set
up multiple WDSSII processes (for both raw and base
data) and control (start/stop) processes as a group.
The user is generally only concerned about 2 items in
the w2algun directory:



The w2alg.conf file
The w2alg executable
Be sure to check out: http://www.wdssii.org/ and click on “Realtime Monitoring”for more information
21 July 2015
14
Processing Data: w2alg.conf

The w2alg.conf file:


This is a config file that allows the user to set up
multiple processes.
Common parameters:








21 July 2015
PROCESS_EXE: the name of the algorithm e.g: w2qcnn
PROCESS_PARAMS: the command-line parameters to the algorithm.
Type the name of the command (w2qcnn, for example) to find out what
parameters that program takes.
NUM_LOGS: default is 10, after which the logs will be rotated
LOG_SIZE: number of lines per log file
LOG_DIR: directory to write logs
DIR: directory to run from
PROCESS_DWELL: number of seconds to pause before starting the
next algo.
RESTART_SLEEP: number of seconds to wait before restarting dead
processes
15
Processing Data: w2alg.conf

More Parameters:

Declaring “groups”:
# group GLOBAL sets for all groups
# Any 'set' overrides within hierarchy
#
group GLOBAL
group KMHX,KRAX,KAKQ,KLWX,KFCX

Setting “macros”


Macros are named M1-M9
Provides a shorthand substitution
# Set a macro to provide the top left and bottom right corners
of a domain in lat/lon as well as vertical grid spacing
set M1 " -t "40 -82 20" -b "32 -73 0" -s "0.01 0.01 1" "

“Managers”

Label a process command to be called by w2alg as an executable
manager RSSD
manager PROCESS-A2
21 July 2015
16
Processing Data: w2alg.conf

Example
set M1 " -t "40 -82 20" -b "32 -73 0" -s "0.01 0.01 1" "
group CLOUD
manager PROCESS-CLOUDCOVER
set PROCESS_EXE "/home/wdssii/WDSS2/bin/w2cloudcover"
set PROCESS_PARAMS "-i "xmllb:cairo:/data/realtime/satellite/code_index.lb
xmllb:wrwdss:/data/ruc2/products/code_index.lb" -o /data/realtime
–l /data/realtime/radar/multi/code_index.lb -S TMP-HTGL:20km2_analysis
-r -e 5 $M1 -I --verbose"

21 July 2015
Notice how we call the macro “$M1” in the PROCESS_PARAMS
section
17
Processing Data: w2alg.conf

Example
group KMHX,KRAX,KAKQ,KLWX,KFCX
manager PROCESS-A2
set PROCESS_EXE "ldm2netcdf"
set PROCESS_PARAMS "-i /home/ldm/data/nexradII/$GROUP -s $GROUP -o
"/data/realtime/radar/$GROUP /data/realtime/radar/U-$GROUP" -k -D "
manager PROCESS-QCNN
set PROCESS_EXE "w2qcnn"
set PROCESS_PARAMS "-i xmllb:/data/realtime/radar/U$GROUP/code_index.lb -o /data/realtime/radar/$GROUP -r -u"
manager PROCESS-SCIT2D
set PROCESS_EXE "swatScit2D"
set PROCESS_PARAMS "-i xmllb:/data/realtime/radar/U$GROUP/code_index.lb -o /data/realtime/radar/$GROUP -f -r -s $GROUP"
set DIR "/home/wdssii/algconfig/$GROUP/scit"

In this example, we first declare a “group” consisting of several
radars. We then declare three different programs to be called with
their associated parameters.

21 July 2015
Note how instead of writing out the full path for each radar and for each
command line parameter, we simply call “$GROUP”
18
Processing Data: w2algrun

The w2alg executable:


After the w2alg.conf file is set up, the user can control stopping and
starting of algorithms in various ways:
Syntax:
w2alg command group [manager]
 Where:
command: is one of start, stop, status, or tail
group:
one of the members as defined by the “group” parameter in the w2alg.conf
file, or “all” to call every member of the “group” declaration
manager: [optional] – specify an executable as defined by the “manager” declaration
21 July 2015
19
Processing Data: w2algrun

The w2alg executable:

Examples
Command
Action
w2alg start all
Starts all the algorithms
w2alg start ktlx
Starts all the algorithms in the group KTLX
w2alg start ktlx a2
Starts the a2 algorithms in the group KTLX
w2alg stop all
Stops all the algorithms
w2alg stop ktlx
Stops all the algorithms in the group KTLX
w2alg status kltx qcnn
Provides status of the qc algorithm on KTLX
w2alg status kltx
Provides status of all the KTLX algorithms.
w2alg tail kltx qcnn
Tails the log of the qc algorithm on KTLX
w2alg tail mainlog
Tails the mainlog which tells you which algorithms died and were
restarted.
21 July 2015
20
Running an Archived Case

Simple processing (I.e., single input) of an
archived case can be easily run at a command
line. However, this is usually done as quickly
as the computer hardware allows.
 On the other hand, more complex processing,
requiring multiple inputs and/or hierarchical
steps, (usually run with w2algrun) requires
coordinated timing.
 Use w2simulator to simulate a real-time data
stream.
21 July 2015
21
Running an Archived Case

w2simulator reads in processed data (the
code_index files), determines the speed of the
notification updates (based on user settings),
and writes/updates an output code_index-like
file at the determined speed.
 Typical use of w2simulator is to read in the
base data. w2simulator then updates the
notifications accordingly. The user then
connects to the w2simulator output as they
would when running real-time algorithm
processing.
21 July 2015
22
Running an Archived Case
Real-time:
Arrival of
Raw Data
Process
Output Base
Data and
Notify
Process
Algorithm
Output (and
Notify)
Archive:
Available
Raw Data
21 July 2015
Process
Output Base
Data
23
Running an Archived Case
w2simulator:
Available
Base Data
21 July 2015
Read
Output Base
Data and Notify
based on timing
arguments given
to w2simulator
Process
Algorithm
Output (and
Notify)
24
Running an Archived Case

We can set up w2algrun to run with w2simulator:

Make your first “manager” section be the w2simulator.
group KMHX,KRAX,KAKQ,KLWX,KFCX
manager SIMULATION
set PROCESS_EXE “w2simulator”
set PROCESS_PARAMS "-i xmllb:/data/realtime/radar/U-$GROUP/code_index.lb –o
/data/simulation/radar/$GROUP –K”
manager PROCESS-QCNN
set PROCESS_EXE "w2qcnn"
set PROCESS_PARAMS "-i xmllb:/data/simulation/radar/$GROUP/index_0.lb -o /data/simulation/radar/$GROUP
-r –u –l /data/simulation/radar/$GROUP/index_0.lb "
manager PROCESS-SCIT2D
set PROCESS_EXE "swatScit2D"
set PROCESS_PARAMS "-i xmllb:/data/simulation/radar/$GROUP/index_0.lb -o /data/simulation/radar/$GROUP
-f -r -s $GROUP –l /data/simulation/radar/$GROUP/index_0.lb "
set DIR "/home/wdssii/algconfig/$GROUP/scit"




21 July 2015
Notice that we do not call “ldm2netcdf” since the data should already be
processed
Notice the input path of the algorithms is the output path of w2simulator
Notice the .lb filename (both input and output) is not code_index.lb
We place the output of the algorithms into the simulation directory (-o)
25
Creating “Snapshots”

Sometimes it is useful to save images of the
data you are viewing.
 WDSSII provides the user with two ways to
take a “snapshot” of the current view


21 July 2015
“Random” Snapshot – click on the display and
take a snapshot of the current image
Auto-snapshot – Set WDSSII to create an
image of each product whenever there is an
update of the data
26
Creating “Snapshots”

Snapshot preferences:





In the “wg” display
Under the “Options” Menu Item
Click “Edit Preferences”
Select the “Snapshot” tab
Rule to create the default name for
snapshots


Good idea to include product
name (%p)
Include either time (%t) or count
(%c)

Use custom-size if you want
images larger/smaller than the
screen size.
 Can specify png/gif/jpg etc.
21 July 2015
27
Creating “Snapshots”: Random Snapshot
1) “Right Click” on radar image
21 July 2015
2) Select “Take Snapshot”
28
Creating “Snapshots”: Auto-Snapshot

Turn on “AutoSnapshot”
21 July 2015

Turn on “AutoUpdate”
29
Question

How can you create a sequence of
images from a 10-hour data case?

Answer on next slide.

Hint: how would you capture 10-hours of imagery in real-time?
21 July 2015
30
Answer

Don’t do things manually!



Think about using the WDSS-II tools in combination.
The display is just another algorithm
Use w2simulator and the display





Simulate the 10-hours of data you have
Use the display to view the simulation LB
Switch on auto-update to display data as they become
“available” in the simulation LB
Switch on auto-snapshot to create images from the displayed
data.
Let the w2simulator and wg run until the case is exhausted

21 July 2015
Run at 10x real-time speed to finish the job in 1 hour
31
Mirroring Data: w2mirror

Sometimes it is helpful to process the data on
one machine, but serve the data on another.

For example, it may be helpful to split data
processing among many machines, but serve
them on one machine.



Less confusing for users to locate data
Fewer potential security/firewall issues
Use w2mirror

21 July 2015
Simply copies data from “source” computer to
“mirror” computer and adjusts all relative links.
32
“Housekeeping”

Universal WDSS-II command-line
options
* --verbose prints out additional information that may be useful in locating
problems
* --configFile=myConfigFile.xml allows the user to run the application with a set
of options that are specified by the input file. This is very useful (and even
required in some cases) when you have a long command line. For
example
<filter path="w2merger">
<option letter='i' value="xmllb:anubis:/data/NSE/code_index.lb
xmllb:isis:/data/realtime/radar/KABR/code_index.lb ..... (long list of LBs
here)">
<option letter='o' value='/data/realtime/radar/multi'/>
<option letter='t' value='51 -127 20'/>
<option letter='b' value='21 -67 0'/>
<option letter='s' value='0.01 0.01 1'/>
<option letter='C' value='5'/>
<option letter='a' value=""/>
<option letter='V' value='false'/>
<option letter='e' value='60'/>
<option letter='I' value='ReflectivityQC'/>
<option letter='p' value='0.5'/>
<option letter='r' value='true'/>
</filter>
* --heartbeat=myOutputLB.lb,XX will write out a "heartbeat" message every XX
seconds. This is useful for determining whether or not an algorithm is
running at all, or has just stopped producing output for some other reason.

Tools for investigating a linear buffer
(I.e., code_index.lb):
lb_cat
lb_cat is a way to examine the contents of a linear buffer or
to write to it from the command line or a script.
lb_create
Allows you to create a linear buffer from a script. Make sure
to use the -n option to size the linear buffer to about 32000.
lb_info
lb_info gives you basic information about the messages in a
Linear Buffer.
lb_nt
lb_nt allows you to troubleshoot event notification.
lb_rm
Removes a linear buffer.

Cleaning up:
w2scour
scours (deletes) data, similar to unix “scour”
Found in src/scripts directory
Usually run via cron
Configure with scour.conf
21 July 2015
33
Troubleshooting
When running WDSSII fails, there are several areas to
examine:


Is RSSD running?
Are the raw data arriving?

Check LDM setup and watch
input flow
 “ldmadmin watch”

Is there any available disk
space left on the hard drive?
 Have we scoured the raw
data?
 Have we scoured nonneeded processed data?
21 July 2015

Are the raw data being
processed?
 Check for code_index.lb
 Is code_index.lb in the
correct directory path?
 Is there data being written
to code_index.lb?

“lb_cat code_index.lb”
34
For More Information…
Check out the following websites:

http://www.wdssii.org
An overview of the entire WDSSII system
http://forum.nssl.noaa.gov
A great Q&A site for WDSSII development and
administration.



End of
“WDSSII: Setting up a real-time
system”
21 July 2015
35