Transcript GOES-R Data Distribution Thomas Renkevens (NOAA/NESDIS/OSD) 4

GOES-R Data Distribution
Thomas Renkevens
(NOAA/NESDIS/OSD)
4th GOES-R Users’ Conference
Broomfield, CO
Tuesday, May 2, 2006
1
Outline
• Requirements
• Instrument Summary and Data Evolution
• PDRR Contract
– Trade Studies
•
•
•
•
•
•
Users and CLASS
Data Distribution Issues
Examples of Current Distribution Systems
GVAR vs GRB
GOES-West Location
Summary
2
GOES-R Observational Requirements
Aerosol Detection
Aerosol Particle Size
Suspended Matter
Volcanic Ash
Aircraft Icing Threat
Cloud Imagery
Cloud & Moisture Imagery
Cloud Base Height
Cloud Layers / Heights & Thickness
Cloud Ice Water Path
Cloud Liquid Water
Cloud Optical Depth
Cloud Particle Size Distribution
Cloud Top Phase
Cloud Top Height
Cloud Top Pressure
Cloud Top Temperature
Cloud Type
Convection Initiation
Enhanced "V"/Overshooting Top Detection
Hurricane Intensity
Imagery: All-Weather / Day - Night
Surface Albedo
Surface Emissivity
Vegetation Fraction
Vegetation Index
Atmospheric Vertical Moisture Profile
Atmospheric Vertical Temperature Profile
Capping Inversion Information
Currents
Ocean Color
Ocean Optical Properties
Ocean Turbidity
Sea & Lake Ice / Displacement & Direction
Sea & Lake Ice / Age
Sea & Lake Ice / Concentration
Sea & Lake Ice / Extent & Characterization
Derived Stability Indices
Moisture Flux
Pressure Profile
Total Precipitable Water
Total Water Content
Clear Sky Masks
Radiances
Absorbed Shortwave Radiation
Downward Longwave Radiation
Flood / Standing Water
Land Surface (Skin) Temperature
Solar Imagery: X-Ray
Microburst Wind Speed Potential
Fire / Hot Spot Imagery
HES – Hyperspectral
Environmental Suite
Sea & Lake Ice / Extent & Edge
Sea & Lake Ice / Surface Temp
Sea & Lake Ice / Motion
Sea & Lake Ice / Thickness
Ice Cover / Landlocked
Snow Cover
Snow Depth
Sea Surface Temps
Energetic Heavy Ions
Mag Electrons & Protons: Low Energy
Mag Electrons & Protons: Med & High Energy
Solar & Galactic Protons
Solar Flux: EUV
Solar Flux: X-Ray
Downward Solar Insolation
Reflected Solar Insolation
Upward Longwave Radiation
CO Concentration
Ozone Total
SO2 Detection
Derived Motion Winds
Lightning Detection
Low Cloud & Fog
Turbulence
Visibility
Geomagnetic Field
ABI – Advanced
Baseline Imager
Dust/Aerosol
Probability of Rainfall
Rainfall Potential
Rainfall Rate
SEISS – Space Env.
In-Situ Suite
SIS – Solar
Instrument Suite
GLM – GOES
Lightning Mapper
3
Magnetometer
GOES-R Baseline Instruments to Meet
User Requirements
• Advanced Baseline Imager (ABI)
– Monitors and tracks severe weather
– Images clouds to support forecasts
• Hyperspectral Environmental Suite (HES)
– Provides atmospheric moisture and temperature profiles to support
forecasts and climate monitoring
– Monitors coastal regions for ecosystem health, water quality, coastal
erosion, harmful algal blooms
• Solar Imaging Suite (SIS) and Space Environmental In-Situ Suite
(SEISS)
– Images the sun and measures solar output to monitor solar storms (SIS)
– Measures magnetic fields and charged particles (SEISS)
– Enables early warnings for satellite and power grid operations, telecom
services, astronauts, and airlines
• Geostationary Lightning Mapper (GLM)
– Detects lightning strikes as an indicator of severe storms
4
GOES Evolution
I-P Combined
Performance Capability
Imaging
Resolution - Visible
Resolution - IR
Full Disk Coverage Rate
# of Channels
Atmospheric Soundings
Resolution
Hourly Coverage
Severe Weather Rapid Scan
# of Channels
Coastal Water Monitoring
Solar Monitoring (SXI)
Lightning Detection
Operate through Eclipse
Ground System Backup
Archive and Access
I-M
N-P
R-Series
GOES I-M
GOES N-P
GOES R
1 km
4 km
30 min
5
1 km
4 km
30 min
5
0.5 km
2 km
5 min
16
10 km
CONUS
No
19
No
10 km
CONUS
No
19
No
10 km / 4 km
Full Disk @ 10 km resolution
Yes - 4km resolution CONUS
~1500
Yes - 300 m resolution
GOES-M only
No
No
Limited
Limited
Yes
No
Yes
Limited
Limited
Yes
Yes
Yes
Full
Yes
5
GOES-R PDRR Contracts and Trades
KDP-B
Internal government
plus
12 industry-led
System
Architecture
Studies
KDP-C/D
3 System Program Definition
& Risk Reduction Contracts
2005
Single System Prime
Acquisition & Operations Contract
2007
Long lead instrument development
initiated early
Launch
Readiness
2012
• Three Program Definition and Risk Reduction (PDRR) Phase
contracts awarded October 22, 2005
• As part of PDRR Contracts, following Trades are under study:
– Data Distribution (GFUL and GRB)
• Distribution methods and content
– Infrastructure Architecture and Interface
• Interface with NESDIS facilities
6
Data Users
MRD ID: 6487
“Discussion: There are also two classes of data usage - operational and
retrospective. Most users performing operational data usage requiring
real-time data delivery for forecast and warning services will receive
their GOES-R data directly from the PD Grouping or from the GRB.
There are other users performing near-real-time operational processing who
could query the GOES-R database to “pull” data in different temporal,
spatial, or spectral forms depending on a particular immediate needs…”
MRD ID: 6491
“The GOES-R system shall make data available to the users portals.
Discussion: Making data available to the users portals may occur by a
number of means including the rebroadcast of the partial data set of
the GRB as described in section 2.10.8.3.1 and options for GFUL
(described in section 3.5.7.4 on GOES-R Series Level 1b Data) ranging
from push and pull capability to GFUL distribution. Push implies
sending data to the user on a subscription basis, typically delivered to the
user on a regular basis. Pull means users request data, not necessarily on a
regular basis.”
7
Key Data Users and CLASS
MRD ID: 5233
“Users shall include, at a minimum, NOAA's National Weather
Service - including: NCEP Units of TPC in Miami, SPC in Norman,
AWC in Kansas City, OPC, HPC, CPC, EMC in Camp Springs, SEC
in Boulder, NCEP Modeling Centers in Fairmont West Virginia,
NWSTG in Silver Spring and its backup at TBS, DoD in AFWA in
Omaha, FNMOC in Monterey, NESDIS in Camp Springs and
Suitland; other portions of NOAA; Academia”
MRD ID: 5171
“CLASS will distribute data and products to a wide variety of
users ranging from scientists doing weather and environmental
research to school children doing homework to other interested
parties wanting a satellite image of a recent storm in their area.
Users will access the CLASS site via the Internet using a
standard web browser. The system will enable Users to search for
the data of interest based on source, instrument, time, and location
and will provide browse images as a search aid. Future
enhancement to CLASS will potentially enable more advanced
search capabilities such as natural language processing, browse
animations, and category searches, e.g., “tornados” or “clear days”.”
8
Data Distribution Issues
• From CONOPS (March 22, 2005) Section 3
– “The difference in processing and communications requirements between
GOES-R and earlier GOES series necessitates a substantial increase in
bandwidth, processing capability, and dissemination approaches.”
– “The large increase of GOES-R data may prevent the entire set of Level 1b
data, called GFUL, from being rebroadcast. A subset of these data, GRB,
will be transmitted to the GOES-R satellite for rebroadcast to all users. Key
users will receive GFUL data by TBD method.
Table 3-1 Comparison of Processing and Storage of GOES-R Series to GOES I –P (TBR)
9
Data Distribution Issues (continued)
• Data distribution considerations under study:
– Distribution methods for GFUL
– Development of a product generation (PG) system that will meet
production processing timelines and product latencies.
• Contents of GRB under study:
– Level 1b and/or Products
• Support of various data formats (MRD ID#6494)
– “The PD shall process product formats including, at a minimum
(TBR), GIF, Text, BUFR, GRIB, Binary, JPEG, NetCDF, and
McIDAS files or their replacement file formats.”
• Other potential formats are under discussion
• Some current distribution methods include
– GVAR Direct Readout, NOAAPORT, McIDAS, Unidata IDD
10
Current GVAR Direct Readout System
Satellite
“Downlink”
(2.6 Mbps)
“GVAR”
(2.11 Mbps)
Any GVAR site
Users
Wallops CDAS
The GVAR is a combination of (mostly) imager and sounder (scaled)
11
radiances (no products). No data compression is used.
Known GVAR Sites
More information on the GVAR system:
http://www.osd.noaa.gov/gvar/gvardownload.htm
12
NOAAPORT Distribution
• Satellite Broadcast
Network (SBN)
– Provides broadcast and
reliable multicast data
transmission to field sites.
• Transmitted data
includes: Centrally
collected radar data,
GOES imagery, NCEP
model data, field
observations, and
watches and warnings
– DVB-S
• Single channel solution.
• Linearly scalable up to
43 Mbps, on demand
13
NOAAPORT Evolution
• Current Architecture challenged to meet increasing data
volumes,
• AWIPS moving to Service Oriented Architecture (SOA)
– Raytheon to implement J2EE Enterprise Service Bus
• SOA will provide a more flexible and robust infrastructure
for AWIPS
• Data delivery and information architecture
– Introduce a more flexible data retrieval paradigm
• Move to “push-pull” data delivery paradigm
– Expanding AWIPS beyond push capability (SBN) only
– Exploring use of OpenDap as a technology to enable a push-pull
paradigm
From Tuell et. al, 2006
AWIPS Then and Now
The Evolution of AWIPS
22nd International Conference on Interactive Information Processing Systems for 14
Meteorology, Oceanography, and Hydrology, Atlanta, Amer. Meteor. Soc.
McIDAS Evolution
McIDAS Background
• McIDAS is a meteorologically oriented analysis and display package, originally developed
by the University of Wisconsin's Space Science and Engineering Center, that emphasizes
comprehensive capabilities for image processing of data from satellite-borne sensor
systems, and that supports ADDE client/server access to remote data
• A collection of user programs and libraries for visualizing and analyzing data
• Support for all international environmental satellites, numerous other observational data
and NWP output
Reasons for Change
• New, advanced instruments require more powerful functionality
• NPOESS (VIIRS, CrIS); GOES-R (ABI, HES)
–
–
–
Imagers: fast, multiple channel combinations
Sounder: convolution of 2000+ hyperspectral channels
Both: co-registration with full radiance information and metadata
Candidate Transition System
• IDV: Integrated Data Viewer
–
–
–
–
Based on SSEC’s VisAD package
Has a data model that is inherently capable of working with standard data types both now and in
the future
Developed for education community by Unidata
Designed to replace McIDAS at Universities
15
What is the IDV?
• Unidata’s newest scientific
analysis and visualization tool
• Freely available Java™
framework and reference
application
• Provides 2- and 3-D displays
of geo-scientific data (plus, of
course, animations)
• Stand-alone or networked
application
• Built on VisAD library
The Integrated Data Viewer (IDV) is a meteorologically oriented, platform-independent
application for visualization and analysis. Developed using Java, VisAD and other
component libraries, the IDV emphasizes interactive 3D visualization and integration of
16
diverse data types.
What is VisAD?
•
•
Open-source, Java library for building interactive and collaborative
visualization and analysis tools
Features include:
– Powerful mathematical data model that embraces virtually any numerical
data set
– General display model that supports 2- and 3-D displays, multiple data
views, direct manipulation
– Adapters for multiple data formats (netCDF, HDF-5, FITS, HDF-EOS,
McIDAS, Vis5D…) and access to remote data servers through HTTP,
FTP, DODS/OPeNDAP, and Open ADDE protocols
– Support for data sharing and real-time collaboration among
geographically distributed users
ADDE Servers (Abstract Data Distribution Environment)
•
•
•
•
•
In use for nearly 10 years
Sectorizing of geographic coverage and bands
Compressed transmission
TCP/IP protocol through ports 112
Variety of clients can access the servers
McIDAS Information From Dave Santek, Tom Whittaker, and Thomas Achtor. Space Science
and Engineering Center, University of Wisconsin in Email correspondence April 7, 2006
17
McIDAS-X vs McIDAS –V Example
McIDAS-X display of an
operational derived
product: Cloud Top
Pressure.
Image is directly
imported into
McIDAS-V, along
with navigation,
graphics, and color
table.
18
Unidata IDD
(Internet Data Distribution)
•
•
•
The Unidata community of over 150 universities is a system for disseminating
near real-time earth observations via the Internet… Unidata IDD is designed so
a university can request that certain data sets be delivered to computers at their
site as soon as they are available from the observing system...
Satellite, radar, and derived product imagery are available to Unidata core sites
in different datastreams delivered in the Unidata IDD:
GOES-East and -West sectors for selected wavelength bands, created at the
SSEC/University of Wisconsin-Madison, and GOES derived products created
at the CIMSS/SSEC are available in the IDD UNIWISC (Unidata-Wisconsin)
datastream
19
Unidata IDD Evolution
•
•
•
•
•
The IDD has grown to become the leading Internet2 advanced-application and one of the
top bandwidth users (http://netflow.internet2.edu/weekly/ ), currently delivering about 20
terabytes (TB) of data per week in the aggregate to participating institutions.
The Unidata IDD has expanded from a US-centric delivery system to one that includes 13
countries on 5 continents.
Most recently, the implementation of a four-node Linux cluster as a top-level IDD relay
demonstrated the ability to relay significant amounts of data to downstream sites (Yoksas,
et al, 2005). Live stress testing (testing conducted on an “operational” system already
feeding data to 220 downstream connections) showed that the cluster was able to relay –
on average – over 500 Mbps (5.4 TB per day) of data to downstream sites during a three
day trial without introduction of product latency.
The limiting factor in this stress test was not the LDM software or cluster node
performance, but, rather, not having more downstream connections. Peak relay data rates
exceeding 900 Mbps showed that the limiting factor in the ability to relay data was the
underlying gigabit network in UCAR.
The successes of the LDM-6 have not deterred investigation of alternate approaches to
data distribution by the Unidata Program Center.
– Network News Transfer Protocol (NNTP) , BitTorrent, design of a new data relay system
–
This ongoing activity will provide the underpinnings of a next-generation IDD that will even better
serve the international Unidata community.
From Yoksas et. al, 2006
The Unidata Internet Data Distribution (IDD) System: A Decade of Development
22nd International Conference on Interactive Information Processing Systems for
Meteorology, Oceanography, and Hydrology, Atlanta, Amer. Meteor. Soc
20
http://www.unidata.ucar.edu/software/idd/
Product Distribution Grouping
• The Mission Requirements Document (MRD) defines the
Product Distribution grouping as follows:
MRD ID: 321
“The Product Distribution (PD) grouping includes distribution of level
1b (GOES full data set), GOES-Rebroadcast data (GRB), and
derived products to user portals while addressing interfaces with
the user for accessing GOES data.”
Note:
GOES-R distribution requirements state all level 1b and derived product
distribution will be through user portals
21
GOES Rebroadcast (GRB)
• GRB system is extension of current GVAR system for GOES-R era
• GRB system described here is a payload service, separate from any
direct service (LRIT, DCPI/R, EMWIN, and SAR)
– GRB is needed to make a large amount of data available to a wide range of
users (geographically and in terms of data use) in a cost efficient manner
• Main differences between GVAR and GRB:
– GRB will be a larger data rate than the GVAR
• 17-24 Mbps vs 2.11 Mbps
– Due to new, large data rates for the instruments currently planned for GOESR, the GRB system will not realistically be able to transmit all level 1b data
without data compression
• With input from PDRR vendors and key users, GRB content must be selected
• Assumptions
– User needs
• All users and applications, both current and future, not known
• If the data are available, user will work to gain access to it
– Users will expect a similar (or higher) level of service
– Communications capabilities will continue to evolve
• Improving capabilities, technologies, and lower cost
22
GOES Rebroadcast (GRB) and GVAR
MRD ID: 4430
“The GOES-Re-Broadcast (GRB) transponder shall support the
rebroadcast of the ground processed weather data from the CDAS
to a wide community of NWS and governmental and academic
research organizations.”
MRD ID: 4431
“Data Content - Ground processed instrument data similar in content
to the current GVAR data (the data content is transparent to the
GRB transponder)”
MRD ID: 6855
“The Ground Segment shall provide a GVAR rebroadcast format of a
selected GOES-R series data subset through the GOES-N series
satellites. Discussion: Current GVAR users that may not be able to
upgrade to the GRB data receivers have expressed an interest in
receipt of a GVAR-like data rebroadcast of GOES-R data in the
23
GOES-R timeframe using only GOES-N satellite series.”
GOES-West view from 135º
24
GOES-West view from 138º
25
Summary
• The great amount of information from the GOES-R series
will offer both a continuation of current products and
services, and provide improved or new capabilities.
• Major improvements in GOES-R means major task in
preparing for the change
• Mission Requirements Document defines product
distribution, users portals, key users, GRB
• Implementation of data distribution and GRB content under
study through PDRR phase
• GOES-R instrument data rates increase by approximately
two orders of magnitude over current instruments
– Some exact instrument designs and data rates still under study
• Need to continue to work with users and PDRR contractors
to shape requirements, define GRB content
26
Alternate Distribution Methods?
27
Links to Additional Information
• NOAA GOES-R Page – Links to CONOPS, GPRD, MRD
– https://osd.goes.noaa.gov/
• NOAA/NESDIS OSD Page
– http://www.osd.noaa.gov/
• NASA Industry Day – Links to Instrument Documentation
– http://goespoes.gsfc.nasa.gov/goesr_industry.htm
• ABI Research at CIMSS
– http://cimss.ssec.wisc.edu/goes/abi/
• HES Research at CIMSS
– http://cimss.ssec.wisc.edu/goes/hes/
• ABI Documentation from NASA:
– http://goespoes.gsfc.nasa.gov/abihome.htm
28