Transcript Document 7215277

AWIPS Continuous Technology Refresh (CTR)
AWIPS Software CTR
TO8: Outbrief
February 7, 2008
This document includes data that shall not be duplicated, used, or disclosed – in whole or in part – outside the Government
for any purpose other than to the extent provided in contract DG133W-05-CQ-1067. However, the Government shall have the
right to duplicate, use, or disclose the data to the extent provided in the contract. This restriction does not limit the
Government’s right to use information contained in this data if it is obtained from another source without restriction. The data
subject to this restriction are contained in all sheets.
T08 Outbrief Introduction
Intro
AWIPS-II T08 Summary
 AWIPS-II Architecture Update
 AWIPS-II CAVE Summary
 AWIPS-II Workstation (CAVE) Capabilities
 AWIPS-II SOA services
 AWIPS-II Plug Ins
 AWIPS-II Library
 AWIPS-II Data
 AWIPS-II TO8 Testing

5/22/2016
Page 2
Weather System Distributed User Needs:
Increasing Data Interactions
Drives Architecture Towards Advanced Solutions
First Person
Shooter Video
Game
Weather
Edge User
Visualization
Summary
Totally Non-Propriety
Solution (Open Source)
Quickly Adaptable to New
Science & Data Types
Accelerate Innovation &
Speed of Deployment
Sat Com
Google Earth
Situational
Awareness
Visualization
CAD
Systems
Enabling of Con-Ops
Flexibility: 164 Centers
Increasing Data Rates
High Data Rates With Gaming Style Interaction
5/22/2016
Page 3
TO8 Task Order

Summary
Primary task is to migrate D2D workstation
capability to the new AWIPS-II developed
architecture
– Performed workstation capabilities migration
– Performed server side migration for ingest and storage of
data
Perform risk reduction for GFE
 Perform risk reduction for Hydro

5/22/2016
Page 4
AWIPS-II Central Architectural Concept
ESB SOA Plug In Adaptable
T08
Future
SOA Data Type Plug Ins
AWIPS-II Services
Staging
Ingest
Purging
Index
VTEC
Archive
Notify
Adapter
Utility
AutoBld Product
Proxy
Airep
Binlightening
Bufrua
Gfe
Gfe
Grib
Obs
Pirep
Radar
Recco
Satellite
Sfcobs
Shef
Shef
Taf
Text
Architecture
Eclipse RCP Visualization
Plug Ins
14 Core
8 User Interface
22 Library
14 Application
CAVE
Thick /Light
/Light
Client Mode
Enterprise Service Bus
5/22/2016
Page 5
Mule ESB + SPRING Container
Services decoupled from transport mechanisms
Architecture
MULE ESB
Mule End Point Provider
Message
Receiver
Data
Data
Inbound Router
SEDA
Load Balanced Queue
Connector Transformers
Mule Interceptor Pattern
Logger
Mule End Point Provider
Transformers Connector
• JMS
• Files
• HTTP
• FTP
• E-Mail
• ***
Data
Data
Outbound Router
Profiler
Message
Receiver
Security Check
EDEX
EDEXSOA
SOA
Service
Service
EDEX SOA services are Dependency
Injected Components
• Configured in the XML Mule files
5/22/2016
Page 6
Plug Ins Pattern Refinements
Dynamically Extends to New Data & Transforms
Plug In
(i.e)plugin-airep
ESB
Endpoint
Config
Architecture
Auto Generated HDF5
schema for Blob data
Metadata DB
HDF5
Lighting
Radar
Satellite
Grid
GFE
Auto Generates Schema
Creates Data URI
Specific
Data Type
Decoding
Custom Data
Transforms if
necessary
Custom
Data
Access
Plug In
Registry Table
5/22/2016
Page 7
Data Repository Maintenance Refinement
Based on a circular cycle of tables & HDF5 files
1 hr Wake Up
Table holds retention
period /2 of data (I.e. 12hr
of data for 24 hr of
retention time)
Architecture
PurgeSrv
DbManager
HDF5 directory tree
Plugin
Metadata
delet
e
30
day & hour
1
Metadata_1
Metadata_2
2
Metadata_3
new
Delete Referenced HDF5 files
Metadata_4
Purge creates new table and
deletes oldest
5/22/2016
Page 8
Clustered Data Ingest Flow Refinement:
Pass References and MetaData between Services
../sbn/…
../../processing
StagingSrv
singleton
EDEX
file reference
clustered queue
jms://cp/…
read & delete
data
record
IngestSrv
f(plugin)
HDF5
vm://…
metadata
persist
IndexSrv
f(plugin)
List records i.e. MetaData
RDBMS
Architecture
Airep
Binlightening
Bufrua
Gfe
Grib
Obs
Pirep
Radar
Recco
Satellite
Sfcobs
Shef
Taf
Text
vm://…
Alert Topic
Subscriptions
NotifySrv
threaded
5/22/2016
Page 9
CAVE Data Notification + Subscription
Data Matched to Menus by Optimized Decision Tree
Architecture
Data Item
CAVE
DecisionTree
CAVE Startup
Quinlan ID3
Algorithm
AlertJobStarter
EDEX
IngestSrv
Build Data URI
Menu dataURIs
URICatalog
IndexSrv
Message Received
AlertManagerJob
Data URI List
quartz Timer
ingested dataURIs
Data URI
Aggregator
EDEX
AutoBldSrv
Script Ids
NotifySrv
topic:edex.alerts
postGreSQL
topic:edex.subscription
Table: Subscription
Table: scripts
Check scripts for URIs
5/22/2016
Page 10
Workstation CAVE Architecture Overview
Built from Eclipse RCP Plugins + Extensible
CAVE-Summary
Eclipse 3.3 XML User
Customizable Menus & Tool bar
GIS Display defined by map
projection and set of layers
(history recallable, procedures)
High Performance Image Rendering
by GPU Shader Language Controls of
(Projection,Color, …)
Large raster data set rendering
through Quad Tree Tilling and
HDF5 chunking
Active Raster and Vector Data
Interrogation
Dynamic map re-projection using
GeoTools Transforms& GPU Solid Shape
Warping
Auto updating Animations of
all data types in all panes
Enterprise Service Bus Data Connections
ESB Connection for thin/thick
client access to data
5/22/2016
Page 11
CAVE leverages Eclipse Perspectives
Enables Multiple User Interfaces
Weather Workstation
Hydrology Workstation
CAVE-Summary
Graphical Forecast Editing
• Perspectives leverages entire set of CAVE eclipse plug ins
• Each perspective can have unique menus, toolbars, and dialogs
• User can actively switch back and forth between perspectives
5/22/2016
Page 12
Map Reprojection: Texture Warping
GPU Fragment Shader 50 times Faster than CPU
• Map Reprojection at Interaction Speeds
• Interactive Color Map
Eclipse RCP Visualization
Mapping Vertex Array
HDF5 Repository
HDF5 Library
IMAGING_GLSL_PROGRAM
CAVE-Summary
Chunked
Float Array
Color Map 1D Texture
OpenGL
API
JOGL
RGB Color Map
XML RGBA
quads
Control
• Raw data sampling enabled by the use of raw data
CPU-GPU Boundary
Graphical Processing Unit (GPU)
GPU Commands & Data Streams
IMAGING_GLSL_PROGRAM
5/22/2016
Page 13
Workstation Menus / Toolbars / Rendering
XML Configurable Menus + XML Bundles
Workstation
.ui.personalitles.awips
plugin.xml
.site. (from localization)
plugin.xml
XML Display Resource Bundles
Commands
~19 Map
plugin.xml
Commands
Parameters
Menu labels
setScale
~11 Plots
RadarDisplayControls
menuRetrieval
productURI
productName
variableList
1 Skew-T
~30 Display
Resources
8 Radar
11 Models
5/22/2016
Page 14
Workstation Configuration
XML used for most all configuration
Workstation
CAVE configuration files under
cave/etc
bundles
colormaps 121 RGBA list unlimited
plotmodels 17 svg, 12 spi
volumebrowser XML Menus
warngen Templates + config
radarInfo
contourStyleRules
imageryStyleRules
graphStyleRules
5/22/2016
Page 15
Workstation Modes
Implemented Using Eclipse Preferences
Workstation
VTEC
Product class code
Test phrases in
Body of Warning
5/22/2016
Page 16
Workstation (CAVE) Localization Pattern
Uses UtilitySrv with a HTTP interface
ESB Sync Protocal
Log In User Name
Determines User Context
Site Through Perferences
Workstation
<<SOA Service>>
UtilitySrv
CAVE Config
At CAVE
Start Up
CAVE Menus
Localization
Hierarchy
Base
Local Persistence
Of Current
Active Localization
Color Maps
Site
User
5/22/2016
Page 17
Workstation History and Procedures
D2D Style Con-Ops (except Alter Bundle)
Workstation
Bundle Components
Map Scale Definition
Zoom Level
Center Point Coord.
Map Projection
<XML>
Bundle
Grid Range
Load Mode
Layer Definitions
Metadata Fields /
File Name
Menus Also Use
Bundles
• Easy Customization of Menus
• Bundles are reuseable through the
passing in of arguments
Color Map
Need to add XY
Components
Display Controls
5/22/2016
Page 18
GIS Map Display Defined by XML Bundle
Enables reuse in D2D style Menus
Workstation
Map Scale Definition
Zoom Level
Center Point Coord.
Map Projection
<XML>
Bundle
Grid Range
Load Mode
Layer Definitions
Display Bundle is
automatically built as
data is selected
Metadata Fields /
File Name
Time matching
control
Color Map
Display Controls
Map Bundles
Available Under Maps
Menu Pull Down
5/22/2016
Page 19
Workstation Volume Browser
Retrieves Views of Data From Grid Repository
Workstation
TimeMatcher
PlanViewDisplay
RequestJob
Loader
{…}Resource
HDF5
Grid
Repository
TimeMatcher
VarHeightDisplay
Loader
SkewTDisplay
{…}Resource
TimeHeightDisplay
Point | Line
Direct Port of
D2D Time
Matching
CrossSection…Display
Slice
Data
Cube
5/22/2016
Page 20
Workstation Volume Browser Menus
KOAX menu & Basic included in installation
Workstation
XML Menu Config
cave/etc/volumebrowser
KOAX Localization
ADE 1.0 Volume Browser
Under CAVE menu
Basic User Manual
Rename Files To
Create Basic Version
BrowserPlanViewPlanes -> BrowserTimePlanes
BrowserSpaceFieldMenu -> BrowserSpaceFields,
BrowserCrossSectionSpaceFields,
BrowserCrossSectionTimeFields,
BrowserHeightFields
BrowserTimeFieldMenu -> BrowserTimeFields
BrowserSourceConus -> BrowserAllSources
5/22/2016
Page 21
Workstation Plot Model and Maintenance
D2D Plot Vector Library Converted to SVG (XML)
Metars
Workstation
Obs MetaData &
Parsed data fields
{…} Plug In
Job
Buoys
PlotResource
PlotModel
Generator
Progressive Disclosure
SVG Plot
SVG Plot
SVG Plot
Model
Model
Models
UA plot 925MB
Entire D2D present
Wx symbol set
converted to SVG
5/22/2016
Page 22
Workstation Color Map Editor
Replicates D2D with transparency control
Workstation
Color Maps Stored on Server
Eclipse RCP Plug In
.viz.ui
Also, set background color dialog
SaveColorMapDialog
UtilitySrv
../cave/etc/colormaps/..
Local Storage
Initialized During Localization
Color Map XML Format
(unlimited number of colors)
<colorMap>
../data/utility/cave_static/base/colormaps
<r>..</r>
./site/colormaps/koax
<g>..</g>
./user/colormaps
<b>..</b>
<a>..</a>
</colorMap>
5/22/2016
Page 23
Workstation Text WS and Editor
Implemented as 2 RCP plug ins in CAVE
Workstation
.texteditor
.textworkstation
TextDB
Workstation Interacts with TextDB
EDEX
text Plug In
Hibernate Access Objects
{tablename}Dao
../opt/data/sbn/test
Text Ingest Endpoint
5/22/2016
Page 24
Workstation Warning Generation
Workstation
T09+ Delivered
T08 Delivered
WarnGen GUI
Life Cycle Rules
Dissemination
VTEC Database
tornado.vm
severethunderstorm.vm
Templates 3..19
TextDB
GIS Capabilities
Warning by PolyGon
UPDATE LIST
Severe Weather Statement
Store
Set VTEC
Generate Text
Send to TextWS
Restart
Text Workstation Editor
5/22/2016
Page 25
Workstation Radar Display
T08 Delivered
Workstation
T09+ Delivered
Radar Plug In
Radar HDF5 Storage
Render Radial
Graphic Decode
Render Raster
Radial Decode
Render Graphic
Text Decode
Raster Decode
Store Radar Radial Products
Store Radar Raster Products
Store Radar Graphic Products
TextDB
Post Radar Text Products
5/22/2016
Page 26
Workstation Skew-T Display
Displays both UA and Model data
Workstation
CAVE RCP Plug In
.viz.skewT
Meteo Library
SkewTDisplay
SkewTResource
UAObsAdapter
GribAdapter
Datacube
EDEX
bufrua Plug In
Dependency on Fortran
function with problem in
JAVA interface
Meta
DATA
Grib Plug In
HDF5
5/22/2016
Page 27
SOA Services
Utility, Maps, Radar Message Handler
SRTM Plus from UCSD
• SRTM30 products from USGS EROS
• Ocean data from LDEO project
• 30sec resolution (~1km)
./CAVE/topo
TopoQuery
+getHeight()
+getTopoPoint()
+getTopoLine()
+getTopoArea()
SOA
Stored in decimated
blobs to support Quad
Titling
./CAVE/basemaps
Map Overlays Imported:
• BCD Files
• GeoTIFFs
• LPI Files
• Shapefiles
• SPI Files
METARs
Used to support disclosure in
Plot models.
5/22/2016
Page 28
T08 Radar Implementation
Reuse of AWIPS I
SOA
NEW AWIPS II
SBN
NCF
Level 2
Filtered Level 3 Products
DXn Server
ORPG
Level 3 Products
ORPGCommsMgr
../radar/*
ESB
Container
(EDEX)
RDBMS
MetaData
Radar Plug In
HDF5
5/22/2016
Page 29
SOA Data Type Plug Ins






Plug Ins
Bin lighting (plugin-binlightning)
satellite.h5
GINI Satellite (plugin-satellite)
Grib 1 and 2 (plugin-grib)
BUFR with RAOB (plugin-bufrua)
Aircraft (plugin-airep, plugin-pirep, plugin-recco)
Synoptic (plugin-sfcobs)
binlighting.h5
grib.h5
– Buoy, Metar, CMAN, Land, MAROB, Mobile, SHIPS




Radar (plugin-radar)
TAF (plugin-taf)
METAR (part of Synoptic)
Text (plugin-text)
radar.h5
– Collectives and StdText
5/22/2016
Page 30
Library Capabilities



Library
VTEC management
Meteo Library
Text Generation (XML)
– Switched from ANTLR parsing of existing templates to creating new
Velocity based templates
– Output text remains the same

Generalized Time
5/22/2016
Page 31
VTEC Management as a Service
Initial User is WarnGen Text Generation
New SOA Service
VtecSrv
Hibernate Access Objects
{tablename}Dao
VTEC Database
http
Vtec API
WarngenAction
+updateVtec
+runTemplate
VtecFactory
Library
WarnGen GUI
VtecEventRetrieval.js
VtecEventUpdate.js
VtecObjectRetrieval.js
VtecRetreival.js
Warning Templates
tornado.vm,cfg
Severethunderstorm.vm,cfg
Text WorkStation
Text Data Base
5/22/2016
Page 32
D2D Meteo Library
Library
T08 Component Users
ContourSupport
fortconfbuf()
hgt2pres()
TopographyGraph
SkewtDisplay
calcrh()
Mixrat()
uvcomp()
Controller
+add_array()
**************
+ztopsa()
GlueGen Java
JNI Interface Code
libmeteoLib.so
127 FORTRAN Modules
17 C Modules
5/22/2016
Page 33
Data Management

Data
Data access refinement (4D data cube)
– SPRING Data Base Connection Pooling (T06 ran with 1 connection)
– DaoPool Data access object pooling for improved performance

Data repository refinement
– Added automatic index creation on sub tables


Hydro IHFS repository + Object to Relational Mapping
(Hibernate)
Text Data Base + Object to Relational Mapping
Data Access Objects
Dao Classes
Dao Classes
Dao Classes
Direct Port of
AWIPS-I TextDB
5/22/2016
Page 34
Data Access Improved Performance
By managing data access objects
Singleton
Developed mid T08 to
improve ingest
performance where
queries need to be
performed before inserts
Data
Example Use
DaoPool
Dao Classes
Dao Classes
Dao Classes
Pirep Plug In
PirepDecoder
PIREP MetaData
5/22/2016
Page 35
Hydro IHFS OB8.1 Data base tables
Middlegen used to auto
Generate relational to
object mapping files
Data
IHFS
180 Tables OB8.1
Hibernate
Data Access Objects
Dao Classes
Dao Classes
Dao Classes
One for each table
5/22/2016
Page 36
T08 Omaha Test Environment
Development and Test Hardware Resources
Testing
Omaha Raytheon Server Room
CP Processor
Dell 2550
Raytheon
Firewall
awipscm
Dell Server
Unidata
LDM-6.6.5
LDM-6.6.5
Auto Nightly
2 Week Stability Testing
Fiber Line
awips-int1
2950 Server
awips-int2
2950 Server
LDM-6.6.5
filters
EDEX Cluster
EDEX BldXX
EDEX BldXX
awips-dev1
2950 Server
LDM-6.6.5
filters
EDEX Nightly
DVB-S
Orion Network
awips-db
2950 Server
postgreSql
awips-nas
2950 Server
Shared
NFS
5/22/2016
Page 37
T08 Integration Test Bed Metrics
Testing
Cluster Balance
5/22/2016
Page 38
T08 AWIPS-II Cumulative Memory Profiles
Small Leak Noted: Working with profiling tools
Example Daily
Profile
Testing
JMX cpu Useage
0.41% Day Average
5/22/2016
Page 39