Transcript WDSS-II Data formats Valliappa Lakshmanan National Severe Storms Laboratory & University of Oklahoma

WDSS-II Data formats
Valliappa Lakshmanan
National Severe Storms Laboratory &
University of Oklahoma
[email protected]
5/27/2016
http://www.wdssii.org/
1
Data types


WDSS-II is based on types of products, not the
products themselves.
These are the most common WDSS-II data types:










5/27/2016
RadialSet
PolarGrid
CartesianGrid2D
CartesianGrid3D
LatLonGrid
LatLonHeightGrid
DataTable
ContourData
WindFieldStruct<LatLonGrid>
WindFieldStruct<CartesianGrid2D>
http://www.wdssii.org/
2
RadialSet

A set of Radials






A Radial




In any order
Not necessarily indexed
At a constant “target” elevation angle to the ground
Can cover more/less than 360 degrees
(0,0) is the radar location
A 1D array of gates
Constant beam-width and gate-width
Defined by starting azimuth and beam width
The data values are scalars (“float”)

5/27/2016
Reflectivity and Velocity are two separate RadialSet
http://www.wdssii.org/
3
PolarGrid

A more convenient form of a RadialSet



Covers 360 degrees (with missing data if needed)
At a constant “target” elevation angle to the ground
Can be a projection of a RadialSet to the ground



Can be indexed to start at defined azimuths
Is a 2D array of scalars (float)





Simplifies vertical computations
First dimension is the radials
Second dimension is the gates
Goes clockwise from due-north
(0,0) is the radar location
Polar data are stored as RadialSets only



5/27/2016
Algorithms convert a RadialSet on the fly to a PolarGrid
Very fast and much more flexible
The algorithm can project/index/sub-sample as required.
http://www.wdssii.org/
4
CartesianGrid2D

Not a recommended structure



Recreates “flat-earth” structure of many NEXRAD
single-radar algorithms
Polar data projected to a plane tangent to the
earth’s surface at the radar
A 2D array of scalars




5/27/2016
SpaceTimeRef is the radar location
(0,0) is the north-west corner
First dimension is the row number
Second dimension is the column number
http://www.wdssii.org/
5
CartesianGrid3D

Enables vertical computations on layers
of CartesianGrid2D products

5/27/2016
Constant height above the radar.
http://www.wdssii.org/
6
LatLonGrid

A 2D array of scalars






Cylindrical equidistant projection
(0,0) is the north-west corner
First dimension is latitude
Second dimension is longitude
Lat-long spacing is constant throughout grid
Only earth-relative system supported by
algorithms


5/27/2016
Read/write this format only
Ingest routines will need to convert from Lambert,
Mercator, etc. (classes exist to simplify this).
http://www.wdssii.org/
7
LatLonHeightGrid

A stack of LatLonGrid
Each LatLonGrid is at constant height
 The LatLonHeightGrid may have either
uniform heights, or non-uniform heights
 (0,0,0) is the lowest north-west corner
 First dimension is height
 Second dimension is latitude
 Third dimension is longitude

5/27/2016
http://www.wdssii.org/
8
Attributes

The gridded products only have scalars.
How do you store the VCP of a RadialSet?
 The expiry time of a product?


Store optional things as “attributes”

In source code:

setAttributeValue(“vcp”, 21);
 getAttributeValue(“vcp”) returns a DataCell
Since attributes are optional, algorithms should be
able to proceed if an attribute is not present in the
data.

5/27/2016
http://www.wdssii.org/
9
SparseGrid

Storing complete grids can be wasteful
For example, a running average or sum
 Need to have 50+ grids in memory


Can use SparseGrid to do this optimally
SparseGrid2D<LatLonGrid> or
SparseGrid2D<CartesianGrid2D>
 Will only store non-”background” values
 Useful when the grids are sparse

5/27/2016
http://www.wdssii.org/
10
SpareGrid netcdf files

The WDSS-II netcdf writer will try to optimize
your write




Choosing to write only non-missing values
Or the entire grid
Based on which will save you space
Can turn this off using environment variable
W2_SPARSE_GRID_THRESH



5/27/2016
-2 to always write in sparse-grid format
2 to always write gridded values
Default value is 0.33
http://www.wdssii.org/
11
DataTable

Used to store tabular data


Column-by-column
The values and their units only



The colors are determined by display configuration
The units of storage should be SI, not English
Special columns


RowName to identify unique names that will be matched
across time
Latitude, Longitude

5/27/2016
If no columns with this name exist, all rows are at the same
location.
http://www.wdssii.org/
12
WindFieldStruct

A data product that consists of 2 or 3
LatLonGrid or CartesianGrid2D
Used to store vector quantities
 u,v,w

5/27/2016
http://www.wdssii.org/
13
ContourData

A product that consists of a sequence of
lat-lon locations organized as polygons.
5/27/2016
http://www.wdssii.org/
14
What output format?

The output format of WDSS-II algorithms
is determined by
w2config/misc/dataformat
The default file uses XML for DataTable
 Uses gz-compressed netcdf for all others
 Can change this to any format


5/27/2016
No need to change algorithms’ source code.
http://www.wdssii.org/
15
How to add an output format

If some one wants you to write data in a different
format

Algorithm can write out in netcdf




You can write a DataEncoder subclass to natively write data in
the other format




5/27/2016
Use converter to convert the data to new format
Recommended solution if the other format is not complete or
properly geo-referenced.
Or if you will still need the netcdf format for other purposes.
Change w2config/misc/dataformat to write other format
The algorithm source code doesn’t change
Recommended solution when deploying the application in other
scenarios.
Note that if you need to read this data back, you also need to
implement a Builder and Extension Loader
http://www.wdssii.org/
16
Netcdf and XML data format

The netcdf and XML data formats that
WDSS-II algorithms use are documented
online
See http://www.wdssii.org/
 And click on “Data format”

5/27/2016
http://www.wdssii.org/
17
LatLonGrid netcdf format
dimensions:
Lat = 650 ;
Lon = 700 ;
variables:
float SHI(Lat, Lon) ;
SHI:Units = "dimensionless" ;
// global attributes:
:TypeName = "SHI" ;
:DataType = "LatLonGrid" ;
:Latitude = 37. ;
:Longitude = -100. ;
:Height = 1000. ;
:Time = 990402843 ;
:FractionalTime = 0.475000000005821 ;
:attributes = "" ;
:LatGridSpacing = 0.01 ;
:LonGridSpacing = 0.01 ;
:MissingData = -99900.f ;
:RangeFolded = -99901.f ;
5/27/2016
http://www.wdssii.org/
18
Index Record

Whenever you write out a data file



You need to also write out a Index Record
This notifies waiting algorithms about your new product
The record has three parts

Time of product (UTC seconds since 1970)


Name of product (“selections”)




YYYYMMDD-HHMMSS InfraredTemperature band2
Time TypeName SubType
In the display, the time is listed last.
How to read the product



5/27/2016
Integer + fractional part
FormatName format-specific-parameters
Eg: netcdf machine:directory filename.gz
Eg: w2algs xmldata machine:directory xmlldata
http://www.wdssii.org/
19
makeIndex

Can automatically create XML/netcdf index
records from a set of files



Run makeIndex.pl or makeNetcdfIndex
makeIndex.pl assumes file naming convention
makeNetcdfIndex reads the data file




Non-WDSS-II programs (C or Perl) can use lb_cat



5/27/2016
Very expensive
Better to have writing program create record also
Use DataEncoder when writing a WDSS-II program
Open a pipe (popen) to lb_cat
Write out XML string corresponding to record
Close the pipe
http://www.wdssii.org/
20