Transcript CERES FM-5 NPP Science Processing Delta Design Review

CERES FM-5
NPP Science Processing
Delta Design Review
GSFC, MD
August 26, 2008
Erika Geier
Jim Closs
Denise Cooper
Sunny Sun-Mack
Lee Bodden
Michael Little
NASA Langley Research Center
CERES FM-5 Delta Design Review
1
Agenda
CERES Introduction
Operations Concept
Design Assumptions
Science Data Processing
Requirements
Instrument Subsystem Design
Clouds Subsystem Design
Testing Strategy
ASDC Implementation
Schedule
Risk Analysis
Issues/Concerns
CERES FM-5 Delta Design Review
2
CERES Mission Ground Systems
• NASA Langley Research Center (LaRC) provides
critical cloud and Earth radiation budget climate data
records to support global climate change research
• Past ERBE and current CERES instruments on
TRMM (1), Aqua (2), and Terra (2), have provided 24
years of sustained radiation budget measurements
• LaRC has collected over 30 Instrument years of
CERES data from TRMM, Terra, and Aqua and has
processed over 90% of that data to date
CERES FM-5 Delta Design Review
3
CERES Current Capabilities
•
Extensive LaRC CERES experience accumulated through the support of
the five CERES instruments currently in orbit
•
Existing scientific expertise for the development, calibration, management
and application of CERES data products
•
Data Center Operations already in place providing ingest, archival,
production, management and distribution of CERES data products
•
Existing Science Computing Facility (SCF) for scientific analysis,
investigations and development of CERES data products
•
Highly-experienced staff already in place supporting:
– Ongoing algorithm investigation, development and enhancements
– Implementation of algorithms into CERES Data Management System
– Enhancements and development to current and new CERES data production systems
– Production and distribution of CERES data products in a configured operations
•
The CERES DM task was successfully appraised at CMMI Capability
Level 3 /Equivalent Maturity Level 2
CERES FM-5 Delta Design Review
4
CERES Climate Analysis Research
System (CARS) Organization
Science
Data
Management
Team
Atmospheric
Sciences
Data Center
• Implements algorithms
• Maintains software
• Verifies data
• Assists in validation
• Provides CM and
documentation support
• Ingests data
• Places operational
software in production
• Produces data sets
• Distributes data sets
• Archives data
• Provides User Services
Algorithm Implementation
Data Production
(DMT)
• Derives & refines
algorithms
• Validates algorithms
• Validates CERES data
sets
• Writes Quality Summary
Algorithm Development
CERES FM-5 Delta Design Review
(ASDC)
5
Required CERES CARS Functionality
• Provide Land PEATE sub-sampling software for aggregated
VIIRS SDRs (Science and DMT)
• Acknowledge data from SDS Land PEATE (ASDC)
• Ingest data from SDS Land PEATE & other sources (ASDC)
• Produce, archive, distribute data products (ASDC)
• Report status of collection (ASDC)
• Support user access to collection (ASDC)
CERES FM-5 Delta Design Review
6
CERES Operations Concept
• CERES instrument raw data transmitted to the C3S and IDPS
• IDPS creates CERES Raw Data Records (RDRs) and provides RDRs
to the SDS SD3E and to ADS/CLASS
• SD3E provides CERES RDRs and VIIRS xDRs to the Land PEATE
• Land PEATE passes CERES RDRs along to ASDC
• Land PEATE runs VIIRS Sub-sampler; provides output to ASDC
• ASDC ingests and archives inputs coming from Land PEATE
• ASDC generates and archives higher-level CERES products
• ASDC distributes products to science researchers and public
CERES FM-5 Delta Design Review
7
CERES Operational Data Flow
NPOESS Systems
Commands, Loads
and requests
CERES
Instrument
Ops Team
Mission Notices
and Data
C3S
Mission Notices
and Data requests
Svalbard
SDS
Data
and
Science
Operations
NOAA Systems
CERES Systems
Existing data,
agreements
Data
Distribution
Data
Providers
Data
Users
CERES RDRs,
VIIRS Sub-samples
& aerosols
RDRs to PST
Cmd
&
Tlm
CERES
CARS
NPP Systems
All RDRs
Mission
Data
RDRs
SDRs
EDRs
IPs
IDPS
All xDRs
SD3E
ADS
CERES FM-5 Delta Design Review
L-PEATE
• Reuse existing systems and interfaces
• System enhancements for NPP CERES
8
CERES Data Processing
Design Assumptions
• CERES CARS data processing development and operations
leverages the existing personnel, procedures, production systems,
and infrastructure already in place supporting CERES instruments
on EOS Terra and EOS Aqua Missions
• CERES CARS will use existing interfaces at the LaRC ASDC
• CERES CARS software development is extensively reusing existing
Terra/Aqua processing codes
• Primary required NPP related enhancements are:
– Use CERES raw data in new Raw Data Record (RDR) format:
 Develop a preprocessor to distribute/sort RDR contents into Aqua-like files that
can be used as inputs to existing CERES processing stream
– Use VIIRS data products:
 Use VIIRS sub-sampled calibrated radiance and aerosol products in place of
instantaneous MODIS data.
CERES FM-5 Delta Design Review
9
NOAA IDPS Assumptions
• Acquire CCSDS Packets
• HDF Wrap CCSDS Packets
• Generate & Stage RDRs for all APIDs for SDS
& CLASS
• Maintain CDFCB
• Provide samples of each CERES RDR
• Provide 28 hours of CERES RDRs for testing
CERES codes at LaRC
CERES FM-5 Delta Design Review
10
SDS/Land PEATE Assumptions
• Subscribe to SD3E for all CERES RDRs (Science,
Diagnostic, and Housekeeping/Telemetry)
• Acquire CERES RDRs and forward to ASDC via existing
Land PEATE interface
• Support CERES reordering of missing data
• Produce aggregated VIIRS SDRs (MODIS HDF-EOS
format) in 5 minute granules (VIAE, VMAE, VDAE). Subsample and ship 5 minute granules (VIMD) to ASDC
• Produce VIIRS aerosol in 5 minute granules (VAOT) and
ship to ASDC
• Post-launch: When VIIRS calibration and/or algorithm
changes, reprocess VIIRS from covers-open forward and
ship to ASDC to support CDRs
CERES FM-5 Delta Design Review
11
Software Development Process
Subsystem Product
Integration and
Verification
Requirements
Management
New Requirement
Received, Logged,
and
Analyzed
Requirement
Accepted
and
Assigned
Subsystem
Updates and Unit
Testing
CM Validation and
Product Integration
Updates
Integrated Into
Baseline
CM Testing
ASDC Validation
Operational
Testing at ASDC
Subsystem
Science Testing
Release to
SIT
ValRx
Testing
Pre-CM Testing
Software Development Process is in the following
document currently under CERES Management review:
Delivery to
CM
CERES FM-5 Delta Design Review
Clouds and the Earth’s Radiant Energy System (CERES)
National Polar-orbiting Operational Environmental Satellite
System (NPOESS) Preparatory Project (NPP)
Data Management System (DMS)
Software Management Plan Version 0
12
Science Data Processing
Approach for CERES FM-5
Processing is divided into 2 Streams:
•
Instrument and ERBE-like
–
–
–
–
–
Autonomous stream, dependent only on CERES data
In production within 48 hours of power-on
Aids in verifying Instrument health
Critical for establishing CERES calibration/validation
Produces ERBE-like data sets
 Data sets exist for CERES instruments on TRMM, Terra, and Aqua
 Similar data sets exist for ERBE instruments on ERBS, NOAA-9 and NOAA-10
•
Fused data sets beginning with Cloud/Convolution/Inversion
processing and the SSF
– Use inputs from imager and other data sources
 Higher resolution imager data is Point Spread Function (PSF) weighted
– Produces higher quality climate data
 Data sets exist for CERES instruments on TRMM, Terra, and Aqua
– Will not be in production immediately after covers open
 Requirement is to ingest VIIRS radiance, geolocation, and aerosol data when it
becomes available from Land PEATE
CERES FM-5 Delta Design Review
13
Simplified CERES Processing Flow
Autonomous Subsystems affected
by changing NPP inputs
CERES
RDRs
SSI
Instrument
Autonomous CERES Subsystems
BDS
SS2
Instantaneous
ERBE-like
ES-8
SS9
TISA
Gridding
SFC
SS3
ERBE-like
TISA
ES-4
ES-9
IES
VIIRS
Radiances
Geolocation
Aerosols
SS4
Compute Clouds,
Convolve with
CERES, compare
TOA/surface
fluxes
SSF
SS10
TISA
Averaging
SRBAVG
Subsystems affected by NPP input
AVG
SS5
Compute fluxes
at levels
CRS
SS6
TISA
Gridding
FSW
SS7 & 8
TISA
Averaging
ZAVG
SYN
CERES FM-5 Delta Design Review
14
Current CERES Climate Data Record
Production Architecture
INSTR:
Instrument
Production Data
Set
Geolocate
and Calibrate
Earth
Radiances
1
ERBE-like
Inversion to
Instantaneous
TOA Fluxes
2
BDS:
BiDirectional
Scans
ERBE-like
Averaging to
Monthly TOA
Fluxes
3
EID6:
ERBE-like
Regional Data
IES: Instrument Earth Scans
MODIS CID:
VIIRS CID:
Cloud Imager
Data
Determine
Cloud Properties,
TOA
and Surface Fluxes
4
SSF: Single
Scanner Footprint
TOA/Surface Fluxes
and Clouds
Compute
Surface &
Atmospheric Radiative
Fluxes
5
SURFMAP:
Surface Map
CRS:
Clouds and Radiative
Swath
Grid
Radiative
Fluxes and
Clouds
6
FSW: Monthly
Gridded Radiative
Fluxes and Clouds
Time
Interpolate, Compute
Fluxes
7
CRH:
Clear Reflectance
History
ES-9:
ERBE-like
Monthly Regional
Averages
ES-8: ERBE-like
Instantaneous
TOA Estimates
Compute
Monthly & Regional
TOA and Surface
Averages
10
SFC: Monthly
Gridded
TOA/Surface
Fluxes and Clouds
Grid TOA and
Surface Fluxes
9
ES-4:
ERBE-like Monthly
Geographical
Averages
SRBAVG:
Monthly
TOA/Surface Averages
MWH:
Microwave
Humidity
Regrid
Humidity and
Temperature Fields
12
MOA:
Meteorological, Ozone,
and Aerosol Data
APD:
Aerosol Data
GAP:
Gridded Analysis
Product
OPD:
Ozone
Profile Data
Grid GEO
Narrowband
Radiances
11
GGEO:
Gridded GEO
Narrowband Radiances
SYNI:
Intermediate
Synoptic Radiative
Fluxes and Clouds
Compute
Regional, Zonal and
Global Averages
8
CERES FM-5 Delta Design Review
GEO:
Geostationary
Narrowband Radiances
SYN
Synoptic Radiative
Fluxes & Clouds
AVG:
Monthly Regional Radiative
Fluxes & Clouds
ZAVG:
Monthly Zonal & Global
Radiative Fluxes & Clouds
15
CERES Input Data
Type of Data
Parameter
Freq
Source
Comments
Description
CERES RDRs
Instrument level 0 data, attitude, ~131/day
and ephemeris
Land PEATE
Expecting 3 types of RDRs: Science,
Telemetry, and Diagnostic
VIIRS L1B Calibrated
Radiance Data
Imager radiances & Geolocation 288/day
Land PEATE
Aerosols
~144/day
Have identified radiance subsets and
will provide associated code
Aerosol data
Aerosol (Coln) Optical
thickness, type/size
1/day
MODAPS
For Terra/Aqua using MODIS
MOD08 and MATCH. Plan to do same
Ozone data
Ozone Profile
1/day SMOBA
NCEP SMOBA
Meteorological data
3-D Met Data
1/day OMI
4/day
OMI
GMAO
2-D atmospheric data
24/day
2-D constants
2-D constants
1
2/day
Geostationary data
MCIDAS data from 5
geostationary satellites per
month
120/day
Global Hydrology Resource
Center (GHRC)
University of Wisconsin
Space Science and
Engineering Center (SSEC)
SURFMAP(Snow/Ice)
Snow/Ice Map
4/day
NCEP/NESDIS
SURFMAP(Snow/Ice)
Snow/Ice Map
1/day
NSIDC
for NPP
Precipitable Water
Only every 3rd hour is used for
production
Existing Data Source
CERES FM-5 Delta Design Review
16
CERES Publicly Available
Output Products
Data Product
Data Product Name
File
Freq
BDS
Bidirectional Scan
1/day
ES-8
ERBE-Like Inst TOA Filtered Radiances
1/day
ES-9
ERBE-Like Monthly Regional Averages
1/mo
ES-4
ERBE-like Monthly Geographical Averages
1/mo
SSF
Single Scanner Footprint TOA/Surface Fluxes & Clouds
1/hr
SFC
Monthly Gridded Radiative Fluxes & Clouds
36/mo
SRBAVG
Monthly TOA/Surface Averages
5/mo
CRS
Clouds and Radiative Swath
1/hr
FSW
Monthly Gridded Radiative Fluxes & Clouds
60/mo
SYN
Synoptic Radiative Fluxes and Clouds
1/day
AVG
Monthly Regional Radiative Fluxes and Clouds
1/mo
ZAVG
Monthly Zonal and Global Radiative Fluxes & Clouds
1/mo
CERES FM-5 Delta Design Review
17
Data Production Scenarios
•
CERES strives to receive all possible FM-5 level 0 data and VIIRS data
•
Instrument
–
–
–
Ingest manager at ASDC to monitor missing RDR files and request those files from Land PEATE
Land PEATE has agreed to provide CERES 99% of RDR files produced by IDPS within 1 month
Data is actually processed three times
 Baseline1-QC run ~ 6 hours after end of the day. Used for quick instrument assessment. Files not needed after
Edition1-CV processes.
 Edition1-CV run ~5-20 days after the end of the month. Assume all available data has been ingested.
Calibration/Validation data set used to compute gains and spectral response functions.
 Edition2 run 6-12 months later using Edition1-CV as input. Applies best available instrument gains and spectral
response functions.
•
Cloud/Convolution/Inversion processing of the SSF
–
–
–
Ingest manager at ASDC monitors missing MODIS files and requests missing files from MODAPS.
VIIRS files to be handled in similar fashion.
Once in routine production, Land PEATE has agreed to provide CERES 99% of available VIIRS files
within 2 months of data date.
Requires CERES and VIIRS radiance/geolocation data to avoid data gaps in SSF files
 If either one missing, a data gap results
 VIIRS aerosol data also needed. If unavailable, aerosol parameters set to CERES default fill values and no data gap
results.
–
Current Terra and Aqua SSFs typically run 6-12 months after data date. NPP processing will be very
similar to Terra and Aqua.
CERES FM-5 Delta Design Review
18
CERES CARS Requirements Management
• CERES Level 3 and Level 4 requirements are being
baselined in a CM-controlled document
• CERES Level 3 requirements have been reviewed by
the SDS and will be provided to the SDS for
inclusion in the SDS Requirements Specification
• Requirements change requires approval by Project
Management and the Configuration Control Board
• Requirements are mapped into subsystem(s) for
implementation
• Requirements are mapped into tests for verification
and validation
CERES FM-5 Delta Design Review
19
CERES Data Sources
CERES CARS
Land PEATE
CERES RDRs
CERES RDRs
Instrument
processing
VIIRS Geo/Rad
Clouds
processing
ERBE-like
processing
VIIRS Geo/Rad
SD3E
Sub-sampling
VIIRS Aerosols
VIIRS Aerosols
Archive,
Distribution
User
Community
VIIRS Subsample code
1- The Land PEATE receives CERES RDRs and VIIRS xDRs from the SD3E.
2- CERES RDRs are passed directly to the CERES CARS, where they are processed by the Instrument subsystem to
produce inputs to the ERBE-like and Cloud subsystems.
3- The Land PEATE produces MODIS-like VIIRS radiance/geolocation and aerosol files.
4- The Land PEATE sub-samples the MODIS-like VIIRS radiance/geolocation data using software provided by CERES.
5- The Land PEATE sends the sub-sampled VIIRS files and the MODIS-like VIIRS aerosol files to the CERES CARS.
6- The CERES CARS runs the downstream subsystems, beginning with Clouds, as data becomes available.
7- Climate-quality data products are archived and made available to the user community along with Data Quality
Summaries and user support.
CERES FM-5 Delta Design Review
20
CERES SDS Requirements
NPP CERES FM-5 Science Data Segment Requirements:
• The ASDC Ingest subsystem shall have the capability of ingesting, verify
by Checksum, and archiving CERES RDRs, sub-sampled VIIRS
radiance/geolocation files, and VIIRS aerosol data received from the
Land PEATE.
• The CERES CARS Preprocessor of the Instrument Subsystem shall
have the capability of producing level-0, attitude, and ephemeris files as
expected by the processing software.
• The CERES DMT shall have the capability of generating and delivering
to the ASDC the following Science Subsystem code:
- Instrument (at launch)
- ERBE-like (at launch)
- Clouds and subsequent subsystem software (12+ months after
launch)
• The CERES DMT shall provide software to the Land PEATE to subsample VIIRS radiance/geolocation files, and work with the Land
PEATE to ensure that the sub-sampling code executes properly.
CERES FM-5 Delta Design Review
21
CERES SDS Requirements
NPP CERES FM-5 Science Data Segment Requirements (cont.):
• The CERES DMT shall have the capability of modifying and redelivering
production code for all 12 CERES Subsystems to produce CERES NPP data
products.
• The ASDC shall have the capability of generating, archiving, and
disseminating climate-quality data products.
• The CERES DMT and Science Team shall have the capability of validating
and writing quality summaries for all CERES data products.
• The ASDC shall have the capability of providing quality summaries and user
support for data customers.
CERES FM-5 Delta Design Review
22
Requirements Verification
1. Baseline LaRC FM-5 L4 requirements by placing CERES CARS
Requirements Specification Document on NX (CERES CARS, 09/2008)
2. Baseline SDS FM-5 L3 requirements by adding CERES to SDS
Requirements Specification Document (SDS, 09/2008)
3. Map FM-5 L3 requirements to SDS Testing Scorecard and schedule (Vic
Buczkowski, 10/2008)
4. Produce Requirements Traceability Matrix to map LaRC L4
requirements to CERES CARS Subsystems and test cases (CERES
CARS, 10/2008)
5. Conduct FM-5 SDS L3 testing as documented in SDS Testing Scorecard
(SDS/LaRC Test Team, (02/2009)
6. Conduct FM-5 LaRC L4 testing as documented in LaRC Requirements
Traceability Matrix (LaRC Test Team, (mid-2009)
CERES FM-5 Delta Design Review
23
Instrument Subsystem Design Approach
Instrument Design Approach
and
CERES DMT RDR Tests
Denise Cooper
Contributing Team Members:
Ashley Alford
Dale Walikainen
Mark Timcoe
CERES FM-5 Delta Design Review
24
Instrument Design Approach
• Extensive reuse of existing software for instrument subsystem
– Proven system used for current CERES processing
– Majority of software written in Ada
• Newly developed code to be in C++
– RDR Preprocessor
 Creates 24-hr Level-0 and Toolkit compliant Ephemeris and Attitude data files
• Ada code will be updated to allow processing of FM-5 data
• PGE to process incoming NPP Level-0 data will be the first
priority, other PGEs will be converted following successful testing
of this PGE.
CERES FM-5 Delta Design Review
25
Instrument Design Approach
• PGE providing data to ERBE-like already in C, only changes will
be to allow processing of FM-5 data.
• Initial version of all code will use the SDP Toolkit due to schedule
constraints. Updates to remove dependency on the SDP Toolkit
expected after launch.
• CERESlib updates to create a C-version of existing F90 Metadata
creation routines.
– Facilitate integration of existing Metadata with Ada, C and C++ software.
CERES FM-5 Delta Design Review
26
NPP Instrument Subsystem
2 files needed to process
(time span noon yesterday noon today)
CERES
RDRs
To ERBE-Like
Ephemeris
Data
~133 Science &
Diagnostic RDRs
with Diary data
Preprocessor
L0
Data
Up to 3 files per day
(Science, Calibration &
Diagnostic)
BDSs
Geolocate
&
Calibrate
Pre-ES8
1 file per day
(Binary containing
Geolocated,
Filtered radiances)
Pre-Es8
Generator
IESs
Attitude
Attitude
Attitude
Data
Attitude
Data
Attitude
Data
Data
Data
14 files needed to process
(time span hr 22 day before hr 00 next day)
Up to 6 files per day
BDS, BDSS, BDSD,
BDSM, BDSP & BDSG
(HDF4 SDS & Vdatas containing
Geolocation values
Raw data counts
Filtered radiances
Instrument H&S info)
Up to 24 files per day
(HDF4 Vdata containing
Geolocation values
Filtered radiances)
To Convolution
Coeff
Data
CERES FM-5 Delta Design Review
27
ERBE-like Spectral Corrections
No code changes needed to support NPP
Spectral
Response
Function
Unfiltering
Coefficients
Unfiltering
Modtran
Data
CERES FM-5 Delta Design Review
28
ERBE-like Inversion
No code changes needed to support NPP
From Instrument Subsystem
ES4
EID6
Pres8
1 file per day
(Binary containing
Geolocated,
Filtered radiances)
1 file per month
(HDF4 containing
Regional, Zonal & Global
Gridded fluxes)
1 file per day
(Binary database
Info)
Monthly
Averaging
Daily
Inversion
ES9
Unfiltering
Coefficients
ES8
1 file per month
(HDF4 containing
Global Average fluxes)
1 file per day
(HDF-EOS4 containing
Geolocated
Filtered & Unfiltered radiances &
fluxes)
CERES FM-5 Delta Design Review
29
CERES DMT RDR Tests
•
Objectives
– Ensure the RDR preprocessor is able to read the CERES Science and
Diagnostic RDR HDF5 files and create the properly formatted Level-0,
Ephemeris and Attitude data files that are compatible with the updated
Ada and the converted C++ code.
•
Description
– Assume ingest of RDRs by ASDC; Alternative is to access RDRs stored
in specified location if not available from the ASDC
– Test with selected RDRs on the development platform to provide output
that will be used by the Level-0 data processor to show that output Level0, Ephemeris and Attitude data is as expected by this PGE.
– Test on the development platform with minimum of 28-hrs of RDR data
through the Level-0 PGE, on to the Pre-ES8 generator to create the data
used by ERBE-like.
CERES FM-5 Delta Design Review
30
CERES DMT RDR Tests
• Description (Cont’d)
– Pre-delivery testing on production platform to ensure all systems work on
the production platform and generate scientifically equivalent data.
 Ensures all software works as expected on the production platform before
delivery
 Provides expected output used by CM to verify their test results
– CM testing using Instrument Subsystem Test Plan to verify that all
necessary updates have been delivered and provide output equivalent to
the provided expected output.
– ASDC SIT operational testing: e.g. Dec. 31, Jan. 1, Feb. 28/29, etc.
• Participants
– Ashley Alford, Denise Cooper, Mark Timcoe, Land PEATE, LaRC ASDC
SIT
CERES FM-5 Delta Design Review
31
CERES DMT RDR Tests
Science
Level-0 File
RDRs
Science &
Diagnostic
~133 per data day
(Level-0 packets &
Diary data)
Calibration
Level-0 File
RDR
Pre-processor
Diagnostic
Level-0 File
Total 10391 packets
Per data day
Attitude
Data Files
12 Data files
(2 hr data per file)
Ephemeris
Data Files
Data files
(Noon previous day Noon next day)
CERES FM-5 Delta Design Review
32
Instrument Schedule
• RDR Format Documentation Provided (Jun. 2008)
• Unpacking of RDR Wrapper Implemented (Jul. 2008)
• Conversion of RDR Contents to CERES Level 0 Format
Implemented (Sep. 2008)
• Selection and Conversion of Attitude/Ephemeris Data
Implemented (Sep. 2008)
• Informal Testing with Actual RDR (dependent on IDPS providing
early RDR) (Oct. 2008)
• Instrument Subsystem Enhancements Integrated and Tested (Dec.
2008)
• Instrument Subsystem Verification Testing Complete
2009)
(Jan.
• Delivery to ASDC through DMT CM for Formal Validation Tests
(by Mar. 13, 2009)
CERES FM-5 Delta Design Review
33
Clouds Subsystem Design Approach
CERES Clouds Subsystem Design Approach
and
CERES DMT VIIRS Tests
Sunny Sun-Mack
Contributing Team Members:
Yan Chen
Walt Miller
CERES FM-5 Delta Design Review
34
CERES SSF Processing
(Subsystem 4)
Subsystem 4 - Determine Cloud Properties, TOA and
Surface Fluxes, is made up of 3 different sets of code
• Clouds - VIIRS inputs will necessitate changes to
support NPP
–
–
Processes the imager data
Resulting pixel-level output written to temporary file
• Convolution - minor changes needed to support NPP
–
–
Merges the pixel-level data with the CERES footprint data
Resulting footprint-level output written to a temporary file
• Inversion - no change needed to support NPP
–
–
–
Reads the footprint level output file containing CERES radiance data and cloud properties
Computes unfiltered radiances, TOA fluxes, and surface fluxes
Generates the SSF data product which is archived and publicly released
CERES FM-5 Delta Design Review
35
NPP Impact on CERES Clouds and
Convolution Design Approach
• VIIRS will replace MODIS as imager source
–
Ability to generate certain cloud properties changes
–
Some radiances, CO2 slicing, may be missing
• CERES IES (input from Instrument Subsystem) interface will
remain the same
• New Point Spread Function (PSF) ancillary file will be generated
to account for VIIRS imager resolution
• SSF interface change possible as CERES migrates to Edition3
CERES FM-5 Delta Design Review
36
Clouds Subsystem Design Approach
• Newly developed VIIRS sub-sampling code at LaRC
– VIIRS Sub-sampling Preprocessor
 Written in C language
 Input:
NPP_VIAE ( Radiances from Imaging “I” Channels )
NPP_VMAE ( Radiances and geolocations from
Moderate-Resolution “M” Channels)
NPP_VDNE ( Radiances and geolocations from Imaging
Day-Night Band (channel) “DNB” )
 Output:
NPP_VIMD_SS ( Radiances and geolocations from all above
three Imaging, Moderate and DNB, but
sub-sampled and sub-channeled with
certain channels being averaged )
CERES FM-5 Delta Design Review
37
Clouds Subsystem Design Approach
• VIIRS sub-sampling code will run at the Land PEATE
– The package to be delivered to Land PEATE by
CERES CM at LaRC
 VIIRS sub-sampling code
 Process Control Generator to produce process control file (PCF)
 Input data and Expected sub-sampled output
 Test Plan (PDF)
– VIIRS sub-setting code will be run at Land PEATE
 The output from VIIRS sub-sampling code, NPP_VIMD_SS, will be
produced at Goddard Land PEATE
 Land PEATE will then deliver NPP_VIMD_SS to LaRC ASDC
• LaRC ASDC will ingest and archive NPP_VIMD_SS
• NPP_VIMD_SS will be the Cloud Imagery Data (CID) as input to
CERES Clouds Subsystem.
CERES FM-5 Delta Design Review
38
VIIRS Sub-setting Data Formats
1. Sub-setting NPP_VIAE
Resolution: 375 meter
Radiances from VIIRS Imaging Channels
Sub-setting Channels
NPP_VIAE
NPP_VIMD_SS
I1 = 0.64 m
I1 = 0.64 m
I2 = 0.865 m
I3 = 1.61 m
I3 = 1.61 m
I4 = 3.74 m
I4 = 3.74 m
I1 = 11.45 m
I1 = 11.45 m
CERES FM-5 Delta Design Review
39
VIIRS Sub-setting Data Formats
1. Sub-setting NPP_VIAE ( Cont’d)
Averaging 4 Imaging Pixels to 1 Moderate Resolution Pixel
I Pixels (NPP_VIAE)
M Pixels (NPP_VIMD_SS)
CERES FM-5 Delta Design Review
40
VIIRS Sub-setting Data Formats
1. Sub-setting NPP_VIAE ( Cont’d)
Sub-setting Samples
NPP_VIAE
Scan-lines x Pixels = 4608 x 6400
Every other pair
of scan-lines
Every other
pair of pixels
NPP_VIMD_SS
Scan-lines x Pixels = 2304 x 3200
CERES FM-5 Delta Design Review
41
VIIRS Sub-setting Data Formats
1. Sub-setting NPP_VIAE ( Cont’d): SDSs Kept
SDS Name (NPP_VIAE)
Description
Dimensions
SDS Name (NPP_VIMD_SS)
Dimensions
Radiance_Img_I1
Radiances for I1 in moderate resolution
(6144,6400)
Radiance_Img_I1_Avg
(1536,1600)
Radiance_Img_I3
Radiances for I3 in moderate resolution
(6144,6400)
Radiance_Img_I3_Avg
(1536,1600)
Radiance_Img_I4
Radiances for I4 in moderate resolution
(6144,6400)
Radiance_Img_I4_Avg
(1536,1600)
Radiance_Img_I5
Radiances for I5 in moderate resolution
(6144,6400)
Radiance_Img_I5_Avg
(1536,1600)
Reflectance_Img_I1
Reflectance for I1 in imagery resolution
(6144,6400)
Reflectance_Img_I1_Sub
Reflectance_Img_I1_Avg
(3072,3200)
(1536,1600)
Reflectance_Img_I3
Reflectance for I3 in imagery resolution
(6144,6400)
Reflectance_Img_I3_Sub
Reflectance_Img_I3_Avg
(3072,3200)
(1536,1600)
BrightTemp_Img_I4
Brightness temperature for I4 at imagery
resolution
(6144,6400)
BrightTemp_Img_I4_Sub
BrightTemp_Img_I4_Avg
(3072,3200)
(1536,1600)
BrightTemp_Img_I5
Brightness temperature for I5 at imagery
resolution
(6144,6400)
BrightTemp_Img_I5_Sub
BrightTemp_Img_I5_Avg
(3072,3200)
(1536,1600)
QF_VIIRS_I1_SDR_1
Quality control flag for I1
(6144,6400)
QF_VIIRS_I1_SDR_1_Sub
(3072,3200)
QF_VIIRS_I1_SDR_3
Scan quality control flag for I1
(192,4)
QF_VIIRS_I1_SDR_3
(192,4)
QF_VIIRS_I3_SDR_1
Quality control flag for I3
(6144,6400)
QF_VIIRS_I3_SDR_1_Sub
(3072,3200)
QF_VIIRS_I3_SDR_3
Scan quality control flag for I3
(192,4)
QF_VIIRS_I3_SDR_3
(192,4)
QF_VIIRS_I4_SDR_1
Quality control flag for I4
(6144,6400)
QF_VIIRS_I4_SDR_1_Sub
(3072,3200)
QF_VIIRS_I4_SDR_3
Scan quality control flag for I4
(192,4)
QF_VIIRS_I4_SDR_3
(192,4)
QF_VIIRS_I5_SDR_1
Quality control flag for I5
6144,6400)
QF_VIIRS_I5_SDR_1_Sub
(3072,3200)
QF_VIIRS_I5_SDR_3
Scan quality control flag for I5
(192,4)
QF_VIIRS_I5_SDR_3
(192,4)
CERES FM-5 Delta Design Review
42
VIIRS Sub-setting Data Formats
1.Sub-setting NPP_VIAE ( Cont’d): The SDSs not being subset
SDS Name
(NPP_VIAE)
Description
Dimensions
Radiance_Img_I2
Radiances for I2 in moderate
resolution
(6144,6400)
Reflectance_Img_I2
Reflectance for I2 in imagery
resolution
(6144,6400)
QF_VIIRS_I2_SDR_1
Quality control flag for I2
(6144,6400)
QF_VIIRS_I2_SDR_3
Scan Quality control flag for I2
(192,4)
CERES FM-5 Delta Design Review
SDS Name
(NPP_VIMD_SS)
Dimensions
43
VIIRS Sub-setting Data Formats
2. Sub-setting NPP_VMAE
Resolution: 750 meter
Radiances and Geolocation from VIIRS Moderate Channels
Sub-setting Channels
NPP_VMAE
NPP_VIMD_SS
M1 = 0.412 m
M3 = 0.488 m
M3 = 0.488 m
M2 = 0.445 m
M4 = 0.555 m
M4 = 0.555 m
M5 = 0.672 m
M7 = 0.865 m
M7 = 0.865 m
M6 = 0.746 m
M8 = 1.24 m
M8 = 1.24 m
M10 = 1.61 m
M9 = 1.378 m
M9 = 1.378 m
M12 = 3.7 m
M11 = 2.25 m
M11 = 2.25 m
M13 = 4.05 m
M14 = 8.55 m
M14 = 8.55 m
M15 = 10.763 m
M15 = 10 763 m
M16 = 12.013 m
M16 = 12.013 m
CERES FM-5 Delta Design Review
44
VIIRS Sub-setting Data Formats
1. Sub-setting NPP_VMAE ( Cont’d)
Sub-setting Samples
NPP_VMAE
Scan-lines x Pixels = 3072 x 3200
Every other
scan-line
Every other
pixel
NPP_VIMD_SS
Scan-lines x Pixels = 1536 x 1600
CERES FM-5 Delta Design Review
45
VIIRS Sub-setting Data Formats
2. Sub-setting NPP_VMAE (Cont’d)
NPP_VMAE and NPP_VIMD_SS
Latitude
Longitude
Solar Zenith Angle
All Geolocation Info in NPP_VMAE is included in
the Subset NPP_VIMD_SS
Solar Azimuth Angle
Sensor Zenith Angle
Sensor Azimuth Angle
Satellite Range
Terrain Height
NPP_VMAE:
Global Attributes
Scan Start Time
Scan Mid Time
CERES FM-5 Delta Design Review
NPP_VIMD_SS:
SDS (64 bits float)
46
VIIRS Sub-setting Data Formats
•
2. Sub-setting NPP_VMAE (Cont’d): SDSs Kept
SDS Name
Description
Dimensions
SDS Name
Dimensions
Radiance_Mod_M11
Radiances for M11 in moderate resolution
(3072,3200)
Radiance_Mod_M11_Sub
(1536,1600)
Radiance_Mod_M14
Radiances for M14 in moderate resolution
(3072,3200)
Radiance_Mod_M14_Sub
(1536,1600)
Radiance_Mod_M15
Radiances for M15 in moderate resolution
(3072,3200)
Radiance_Mod_M15_Sub
(1536,1600)
Radiance_Mod_M16
Radiances for M16 in moderate resolution
(3072,3200)
Radiance_Mod_M16_Sub
(1536,1600)
Radiance_Mod_M3
Radiances for M3 in moderate resolution
(3072,3200)
Radiance_Mod_M3_Sub
(1536,1600)
Radiance_Mod_M4
Radiances for M4 in moderate resolution
(3072,3200)
Radiance_Mod_M4_Sub
(1536,1600)
Radiance_Mod_M7
Radiances for M7 in moderate resolution
(3072,3200)
Radiance_Mod_M7_Sub
(1536,1600)
Reflectance_Mod_M11
Reflectance for M11 in moderate resolution
(3072,3200)
Reflectance_Mod_M11_Sub
(1536,1600)
Reflectance_Mod_M3
Reflectance for M3 in moderate resolution
(3072,3200)
Reflectance_Mod_M3_Sub
(1536,1600)
Reflectance_Mod_M4
Reflectance for M4 in moderate resolution
(3072,3200)
Reflectance_Mod_M4_Sub
(1536,1600)
Reflectance_Mod_M7
Reflectance for M7 in moderate resolution
(3072,3200)
Reflectance_Mod_M7_Sub
(1536,1600)
Reflectance_Mod_M8
Reflectance for M8 in moderate resolution
(3072,3200)
Reflectance_Mod_M8_Sub
(1536,1600)
Reflectance_Mod_M9
Reflectance for M9 in moderate resolution
(3072,3200)
Reflectance_Mod_M9_Sub
(1536,1600)
BrightTemp_Mod_M14
Brightness temp. for M14 at mod resolution
(3072,3200)
BrightTemp_Mod_M14_Sub
(1536,1600)
BrightTemp_Mod_M15
Brightness temp for M15 at mod resolution
(3072,3200)
BrightTemp_Mod_M15_Sub
(1536,1600)
BrightTemp_Mod_M16
Brightness temp for M16 at mod resolution
(3072,3200)
BrightTemp_Mod_M16_Sub
(1536,1600)
CERES FM-5 Delta Design Review
47
VIIRS Sub-setting Data Formats
2. Sub-setting NPP_VMAE (Cont’d): SDSs Kept
SDS Name
Description
Dimensions
SDS Name
Dimensions
QF_VIIRS_GEO_MOD_2
Quality control flag for geo location
(3072,3200)
QF_VIIRS_GEO_MOD_2_Sub
(1536,1600)
QF_VIIRS_M11_SDR_1
Quality control flag for M11
(3072,3200)
QF_VIIRS_M11_SDR_1_Sub
(1536,1600)
QF_VIIRS_M11_SDR_3
Scan quality control flag for M11
(192,4)
QF_VIIRS_M11_SDR_3
(192,4)
QF_VIIRS_M14_SDR_1
Quality control flag for M14
(3072,3200)
QF_VIIRS_M14_SDR_1_Sub
(1536,1600)
QF_VIIRS_M14_SDR_3
Scan quality control flag for M14
(192,4)
QF_VIIRS_M14_SDR_3
(192,4)
QF_VIIRS_M15_SDR_1
Quality control flag for M15
(3072,3200)
QF_VIIRS_M15_SDR_1_Sub
(1536,1600)
QF_VIIRS_M15_SDR_3
Scan quality control flag for M15
(192,4)
QF_VIIRS_M15_SDR_3
(192,4)
QF_VIIRS_M16_SDR_1
Quality control flag for M16
(3072,3200)
QF_VIIRS_M16_SDR_1_Sub
(1536,1600)
QF_VIIRS_M16_SDR_3
Scan quality control flag for M16
(192,4)
QF_VIIRS_M16_SDR_3
(192,4)
QF_VIIRS_M3_SDR_1
Quality control flag for M3
(3072,3200)
QF_VIIRS_M3_SDR_1_Sub
(1536,1600)
QF_VIIRS_M3_SDR_3
Scan quality control flag for M3
(192,4)
QF_VIIRS_M3_SDR_3
(192,4)
QF_VIIRS_M4_SDR_1
Quality control flag for M4
(3072,3200)
QF_VIIRS_M4_SDR_1_Sub
(1536,1600)
QF_VIIRS_M4_SDR_3
Scan quality control flag for M4
(192,4)
QF_VIIRS_M4_SDR_3
(192,4)
QF_VIIRS_M7_SDR_1
Quality control flag for M7
(3072,3200)
QF_VIIRS_M7_SDR_1_Sub
(1536,1600)
QF_VIIRS_M7_SDR_3
Scan quality control flag for M7
(192,4)
QF_VIIRS_M7_SDR_3
(192,4)
QF_VIIRS_M8_SDR_1
Quality control flag for M8
(3072,3200)
QF_VIIRS_M8_SDR_1_Sub
(1536,1600)
QF_VIIRS_M8_SDR_3
Scan quality control flag for M8
(192,4)
QF_VIIRS_M8_SDR_3
(192,4)
QF_VIIRS_M9_SDR_1
Quality control flag for M9
(3072,3200)
QF_VIIRS_M9_SDR_1_Sub
(1536,1600)
QF_VIIRS_M9_SDR_3
Scan quality control flag for M9
(192,4)
QF_VIIRS_M9_SDR_3
(192,4)
CERES FM-5 Delta Design Review
48
VIIRS Sub-setting Data Formats
2. Sub-setting NPP_VMAE ( Cont’d): The SDSs not being subset
SDS Name
Description
Dimensions
Radiance_Mod_M1
Radiances for M1 in moderate resolution
(3072,3200)
Radiance_Mod_M10
Radiances for M10 in moderate resolution
(3072,3200)
Radiance_Mod_M12
Radiances for M12 in moderate resolution
(3072,3200)
Radiance_Mod_M13
Radiances for M13 in moderate resolution
(3072,3200)
Radiance_Mod_M2
Radiances for M2 in moderate resolution
(3072,3200)
Radiance_Mod_M5
Radiances for M5 in moderate resolution
(3072,3200)
Radiance_Mod_M6
Radiances for M6 in moderate resolution
(3072,3200)
Radiance_Mod_M8
Radiances for M11 in moderate resolution
(3072,3200)
Radiance_Mod_M9
Radiances for M11 in moderate resolution
(3072,3200)
Reflectance_Mod_M1
Reflectance for M1 in moderate resolution
(3072,3200)
Reflectance_Mod_M10
Reflectance for M10 in moderate resolution
(3072,3200)
Reflectance_Mod_M2
Reflectance for M2 in moderate resolution
(3072,3200)
Reflectance_Mod_M5
Reflectance for M5 in moderate resolution
(3072,3200)
Reflectance_Mod_M6
Reflectance for M6 in moderate resolution
(3072,3200)
BrightTemp_Mod_M12
Brightness temperature for M12 at moderate resolution
(3072,3200)
BrightTemp_Mod_M13
Brightness temperature for M12 at moderate resolution
(3072,3200)
CERES FM-5 Delta Design Review
SDS Name
Dimensions
49
VIIRS Sub-setting Data Formats
2. Sub-setting NPP_VMAE ( Cont’d): The SDSs not being subset
SDS Name
Description
Dimensions
QF_VIIRS_M10_SDR_1
Quality control flag for M10
(3072,3200)
QF_VIIRS_M10_SDR_3
Scan quality control flag for M10
(192,4)
QF_VIIRS_M12_SDR_1
Quality control flag for M12
(3072,3200)
QF_VIIRS_M12_SDR_3
Scan quality control flag for M12
(192,4)
QF_VIIRS_M13_SDR_1
Quality control flag for M13
(3072,3200)
QF_VIIRS_M13_SDR_3
Scan quality control flag for M13
(192,4)
QF_VIIRS_M1_SDR_1
Quality control flag for M1
(3072,3200)
QF_VIIRS_M1_SDR_3
Scan quality control flag for M1
(192,4)
QF_VIIRS_M2_SDR_1
Quality control flag for M2
(3072,3200)
QF_VIIRS_M2_SDR_3
Scan quality control flag for M2
(192,4)
QF_VIIRS_M5_SDR_1
Quality control flag for M5
(3072,3200)
QF_VIIRS_M5_SDR_3
Scan quality control flag for M5
(192,4)
QF_VIIRS_M6_SDR_1
Quality control flag for M6
(3072,3200)
QF_VIIRS_M6_SDR_3
Scan quality control flag for M6
(192,4)
CERES FM-5 Delta Design Review
SDS Name
Dimensions
50
VIIRS Sub-setting Data Formats
3. Sub-setting NPP_VDNE
Resolution: 750 meter
Radiances and Geolocation from VIIRS Day-Night Channel
Only One channel-->No
Sub-setting Channel
Keep All Geolocation SDSs
Latitude
DNB = 0.7 m
Longitude
Solar Zenith Angle
Solar Azimuth Angle
Sensor Zenith Angle
Sub-setting Sample
Sensor Azimuth Angle
Subset every other scan-line and
every other pixel
CERES FM-5 Delta Design Review
Satellite Range
Terrain Height
51
VIIRS Sub-setting Data Formats
4. File Size Summary
INPUT:
NPP_VIAE ---> 460 MB / granule
NPP_VMAE ---> 560 MB / granule
NPP_VDNE ----> 330 MB / granule
OUTPUT
NPP_VIMD ---> 400 MB / granule
~ 70% Data Reduction
CERES FM-5 Delta Design Review
52
NPP CERES Cloud Subsystem
NPP VIIRS Subset: NPP_VIMD_SS
To CERES Convolution
NPP VIIRS CID
EIPD
(Cookie Dough)
MODIS Aqua CID
EQCG
MODIS Terra CID
EQCB
TRMM VIRS CID
ECVS
ECV
Cloud
Imager Data
(CID)
Output
Clouds
Processor
GMAO
MOA
Emissivity Maps x 4
IGBP
Snow/ice Model
Water %
BiDir Model
Elevation
Surface Maps
Clear
Reflectance
History
Algorithm
Ancillaries
Directional Model
Angular
Models
CERES FM-5 Delta Design Review
53
Convolution Subsystem Description
Convolve imager radiances, cloud properties, and aerosol into CERES footprint
Instrument
Cloud Pixel
IES
EIPD
X 24 hours
SSFI
Cookie dough
Convolution
SSFAI
FQCI
PSF
FQC
Inversion
CERES FM-5 Delta Design Review
54
CERES DMT VIIRS Tests
•
Objectives
Ensure the VIIRS Sub-sampler processor is able to:
(1) Read
NPP_VIAE ( Imager resolution radiances),
NPP_VMAE ( Moderate resolution radiances and geolocations)
NPP_VDNE (Day / Night band radiances and geolocations)
(2) Subset
sub-channel
sub-sample
averaging of imagery resolution data
(3) Produce VIIRS sub-sampled output
NPP_VIMD_SS
CERES FM-5 Delta Design Review
55
CERES DMT VIIRS Tests
• Description
– If less than a couple of hours of NPP VIIRS data is available, then Land
PEATE pushes VIIRS NPP_VIAE, NPP_VMAE, and NPP_VDNE to a
CERES/VIIRS designated computer
– For larger volumes of data, the Land PEATE sends all test NPP VIIRS
data types to the ASDC for ingest, and then subsequently staged for access
by the VIIRS Sub-sampler on a CERES/VIIRS computer
– Execute VIIRS Sub-sampler on local CERES/VIIRS computer to create
and verify sub-sampled output using input VIIRS data from Land PEATE
– Load VIIRS Sub-sampler and selected VIIRS test data in the file space
and computer designated for CERES processing at the Land PEATE, and
then conduct testing at the Land PEATE and verify sub-sampled output
– CERES DMT delivers VIIRS Sub-sampler (tar files) and test plan to the
CERES CM for delivery to the Land PEATE
– Land PEATE conducts VIIRS Sub-sampler testing creating sub-sample
– Land PEATE verifies sub-sample and also send sub-sampled output to
either the ASDC or to a designated CERES/VIIRS computer for additional
verification
CERES FM-5 Delta Design Review
56
CERES DMT VIIRS Tests
• Required Input
• Expected Output
NPP_VIAE
NPP_VMAE
NPP VIIRS
Sub-setting
Processor
NPP_VIMD_SS
NPP_VDNE
• Participants
– Sunny Sun-Mack, Yan Chen, Tammy Ayers, Walt Miller, CERES Science
Team members: Patrick Minnis, Norman Loeb, Tom Charlock, Dave
Kratz and Dave Doelling, Land PEATE, and LaRC ASDC
CERES FM-5 Delta Design Review
57
CERES CARS Delivery Process
to the Land PEATE
• Following unit, integration and verification testing by the CERES
DMT, the VIIRS Sub-sampler must be delivered to the Land
PEATE for final testing
• CERES DMT creates tar files containing source code, process
control generator, selected NPP VIIRS input data and expected
sub-sampled output
• CERES DMT updates Test Plan for testing at the Land PEATE
• CERES DMT delivers the above tar files along with the Test Plan
to CERES CM
• CERES CM delivers the tar files and Test Plan to the Land
PEATE’s designated machine. A “Delivery Notification” email is
sent to the Land PEATE and all concerned
CERES FM-5 Delta Design Review
58
CERES CARS Delivery Process
to the Land PEATE (cont.)
• Land PEATE conducts VIIRS Sub-sampler testing
• When the Land PEATE testing is successfully completed, the Land
PEATE:
– Sends an email to all concerned regarding the status
– Promotes the delivery to their production environment
– Makes the delivery (as tar files) available in a designated area for CERES
CM to retrieve
– Sends a notification email to CERES CM that this action has taken place.
• Upon receiving this notification, CERES CM:
– Retrieves the tarred delivery
– Places it in the CERES CM repository
• Subsequent changes to CM controlled software are documented
and tracked using a CERES Software Configuration Change
Request (SCCR) and redelivered through CERES CM
CERES FM-5 Delta Design Review
59
NPP VIIRS Sub-sampler Schedule
• Data Format of VIIRS Subset for CERES Defined (June 1, 2008)
Completed on Time.
• Data Product Catalog Pages Documenting VIIRS Subset for CERES
Defined (August 1, 2008) Completed on Time
• Sub-sampler Code Complete and Running Locally at LaRC on Linux
Machine (October 1, 2008)
• Goddard Land PEATE Accepts the Sub-sampler Code Delivery and
Produces Sub-sampled Output from Input Data (December 1, 2008)
• Verify a Few Days of Sub-sampled Data Produced at GSFC (January
15, 2009)
• ASDC Ingests Sub-sampled VIIRS Data; Verified by CERES DMT
(February 26, 2009)
CERES FM-5 Delta Design Review
60
CERES DMT Testing Approach
•
The CERES DM task was successfully appraised at CMMI Capability Level 3
/Equivalent Maturity Level 2
•
SSAI is appraised at CMMI Capability Level 3/Equivalent Maturity Level 2
•
CERES DMT uses a structured CMMI-based development process that
includes unit and integration testing
•
Functional (e.g., verification) tests are executed to verify requirements for all
enhancements
•
Regression tests are executed to verify existing functionality
•
Delivery packages for CERES and VIIRS Sub-sampler to include:
– Test Plan
– Expected output
– All inputs
– Code and scripts
•
Every CERES code delivery (to ASDC or Land PEATE) is under
configuration management (CM) control
•
Documented in CERES NPP DMS Software Management Plan Version 0
CERES FM-5 Delta Design Review
61
CERES CARS Testing Strategy
• Conversion of RDRs to Level 0 data will be tested by the DMT using
defined formats and available test RDRs prior to delivery of code to
the ASDC
– Testing conducted in development environment
• Transmission of RDRs by the Land PEATE to the ASDC will be tested
when:
– CERES RDRs are available to the Land PEATE from the SD3E
– Interface between ASDC and the Land PEATE has been established
• Final testing of conversion of RDRs to Level 0 data will involve:
– Testing in the CERES operations test environment
– Land PEATE sending all three RDR data types to the ASDC
– ASDC ingesting data and acknowledging receipt to the Land PEATE
– Instrument subsystem reading RDRs staged from ASDC archive and correctly
converting to Level 0 format
CERES FM-5 Delta Design Review
62
CERES CARS Testing Strategy
• VIIRS Sub-sampler will be tested by the DMT using test data provided
by the Land PEATE prior to code delivery to the Land PEATE
– Testing conducted in development environment
– Testing also conducted on designated Land PEATE computer
• Transmission of sub-samples by the Land PEATE will be tested when:
– VIIRS Sub-sampler has been tested and delivered to the Land PEATE
– VIIRS data is available to the Land PEATE from the SD3E or other source
– Interface between ASDC and the Land PEATE has been established
• Final testing of Sub-sampler will involve:
– Execution of Sub-sampler by the Land PEATE to produce sub-sampled data
product
– Transmission of the sub-sampled data to the ASDC for ingest and archive
– CERES DMT stages data from ASDC archive and verifies sub-sample
CERES FM-5 Delta Design Review
63
DMT NPP Schedule
10/1/08: VIIRS sub-sampling software complete and tested at LaRC
12/1/08: VIIRS sub-sampling software running at SDS Land PEATE, expected sub-sampled
output has been verified by CERES DMT
1/15/09: Large set of VIIRS sub-sampled output produced at SDS Land PEATE and verified
at LaRC by CERES DMT
3/13/09: Instrument subsystem code needed to support NCT3 complete and tested by DMT
5/15/09: Instrument subsystem code delivered to ASDC, tested, and promoted to operations
08/13/09: Instrument subsystem code needed to support NCT4 complete and tested by DMT
 Interim delivery to correct NCT3 problems and prepare for NCT4
10/15/09: Instrument subsystem code delivered to ASDC, tested, and promoted to operations
11/19/09: Instrument subsystem code needed to support launch complete and tested by DMT
1/15/10: Instrument subsystem code delivered to ASDC, tested, and promoted to operations
CERES FM-5 Delta Design Review
64
CERES CARS Build Schedule
CERES Data Management System
Projected Build promotion dates to ASDC Operations:
Build 1: NCT3 Functionality
-
May 2009
Build 2: NCT4 Functionality
-
October 2009
Build 3: Launch-Ready System
-
January 2010
CERES FM-5 Delta Design Review
65
ASDC Implementation Approach
ASDC Implementation Approach
Michael Little
CERES FM-5 Delta Design Review
66
ASDC Supports CERES PI in Ingest, Archive,
Distribution, Production
•
Ingesting and Archiving inputs for all Data Products
– Use by Production in Producing Data Products
– Use by Science Team in analysis and algorithm refinement
– Long-term stewardship of data including migration across media changes
 Moving from Tape archive with Disk Cache to Disk archive with Tape Backup
•
Distribute CERES Data Products
– External Customers order data products via ASDC Order Tool, ESDIS ECHO




ASDC Order Tool provides interim support
ESDIS ECHO provides access to all EOS data products
Subscription Services
Plan to supply CLASS with subscription service as an offsite backup
– Internal users via file sharing
 Moving from Request/deliver model to online Fiberchannel/NFS
•
Produce Data Products for CERES PI
– Accept and test Production codes from DMT
– Accept Production Requests from DMT
– Run codes consistently to produce expected data products
 Now by procedure; expect automation within one year
CERES FM-5 Delta Design Review
67
ASDC Evolution Approach
•
Provide CERES Production Services at ASDC
– Meet constraints of CERES PGEs
 Avoid increasing workload on DM/Science workforce
 Support FORTRAN, Ada, and IDL languages
 Meet increasing memory and disk access requirements
– Use lessons learned to improve CERES production
 Improve I/O bandwidth limitations to allow multiple instances of some subsystems
 Reduce unplanned outages by increasing storage reliability
 Reduce dependence on active tape archive to improve labor intensive staging processes
– Increase production capacity for FM-5 and reprocessing Terra/Aqua
– Replace current hardware at end-of-life
•
Provide Access to CERES Data Products for Internal Users
– Production
– LaRC Science Community
– CERES Data Management Team
•
Improve Order Throughput for External Customers
CERES FM-5 Delta Design Review
68
ASDC & SCF Integrated Architecture
Fiberchannel Connectivity
ASDC SDP Production
56 P6
256GB
ASDC JBOSS
Server
& TS-1
& Ingest
56 P6
256GB
Db
Server
56 P6
256GB
64 x86 SMP
64 x86 SMP
256 GB
256 GB
Common
SANhead
(NFS)
(AFP)
(Samba)
LB1
SCF Cluster
(Access through
Sun Grid Engine)
SCF Interactive Processors
Db
Server
64 x86 SMP
128 GB
LB1
14 4P6@8GB
16 x86 HS12
64 GB
64 x86 SMP
256 GB
56 P6
256GB
Instrument
Instrument
ERBE-like
TISA
TISA
SARB
Consodine
SOFA
Clouds
Clouds
WWW
WWW
SRB/Power
SRB/Power
56 P6
256GB
Fiberchannel Switch
220TB
DS4800
220TB
DS4800
220TB
DS4800
220TB
DS4800
ASDC Disk Archive
(R/W Production)
(R/O SCF)
220TB
DS4800
220TB
DS4800
64 x86 SMP
128 GB
220TB
DS4800
ECS DUE
Processing
FC
Monitor
Sim
Server
CERES FM-5 Delta Design Review
220TB
DS4800
SCF Disk Array
64 x86 SMP
128 GB
Cloud Modeling
Cluster (Xu)
GPU Experiment (YongHu)
3755
Tesla
3755
Tesla
69
ASDC & SCF TCP/IP Architecture
56 P6 256GB
Zamboanga
SCF SGE Head
56 P6 256GB
56 P6 256GB
56 P6
256GB
Load
1
Magneto
IBM Director
x3650
ASDC SGE Head
64 x86 SMP
256 GB
Cluster Mon
(HW TBD)
64 x86 SMP
256 GB
ASDC JBOSS Server & TS-1
ANGe
Db
Servers
64 x86 SMP
128 GB
ECS DUE Processing
ANGe
Db
Servers
4x16 x86 SMP
128 GB
LaRCNet Switch
SH1
x3650
SH2
x3650
FC
Monitor
Sim
Server
ASDC Disk Archive (R/W Production)(R/O SCF)
220TB
220TB
220TB
220TB
DS4800
DS4800
DS4800
DS4800
220TB
DS4800
3755
Tesla
3755
Tesla
GPU Experiment
NISN WAN
192.168.16.x
220TB
DS4800
Load
2
SCF Interactive Processors
14 4P6@8GB
Center Firewall
Common SANhead
(NFS, AFP, Samba)
220TB
DS4800
WWW Proxy
Servers
64 x86 SMP
256 GB
Instrument
Instrument
ERBE-like
TISA
TISA
SARB
SARB
SOFA
Clouds
Clouds
WWW
WWW
SRB/Power
SRB/Power
ASDC SDP Production (SGE)
SCF Cluster (SGE)
56 P6 256GB
220TB
DS4800
SCF Disk Array (R/W)
CERES FM-5 Delta Design Review
64 x86 SMP
128 GB
Cloud
Modeling
Cluster
Legend
LaRCNet 100Mbps Ethernet
Private Net (DS Mgt)
Private Net (GPFS)
Cluster & Mgt Net (SGE)
Green Indicates SubSys Function
70
ASDC Ingest Processes for NPP
Land
PEATE
Subscription
PDR
ANGe
Drop-Box
Files
PDR
Files
PDR
Files
ASDC Ingest Processor
Files
For Orderable
Data Products
Files
Location
Metadata
CERES FM-5 Delta Design Review
I/F
To be
Resolved
NOAA
CLASS
ECHO
Metadata
Adapter
Metadata
& GUID
Local
Production
ASDC
Delivery
Server
ASDC
Archive
ANGe
Core db
IMS
db
71
NPP Impacts to the ASDC
Impact
Increased storage capacity and
performance needed for new CERES
data set
Increased processing capability
needed for new CERES data set
Additional CERES data production
requirements effect on staffing
Operating procedures required for
new data set and experience of staff
Ability of facility to accommodate
additional hardware with respect to
space, HVAC and power
Increased data load impact on
network throughput
User access to current and future
CERES data products
Resolution
Additional disks purchased with fiberchannel connectivity to
production processors.
Higher density tapes expand capacity of tape archives.
Additional processors purchased and added to Linux cluster
increasing processing capacity.
Faster data access from new disk archive vs. tape archive.
Improvements in staging/ingest/archive reliability will offset
staff workload for production.
Staff experienced in processing CERES data products.
Job production using on existing Subsystem DMT Ops
Manuals requiring minor updates for NPP.
Staff experienced in processing CERES data products.
ASDC has space/HVAC for expansion.
Power upgrades 09/08/08
New architecture will improve internal data production
network throughput and access by SCF users
May need additional bandwidth to Land PEATE for VIIRS
New architecture will speed ordering CERES data products
New data products will be available through current ordering
tools when released to public
CERES FM-5 Delta Design Review
72
Ingest/Archive Sizing Estimates
ASDC can accommodate expected CERES data volumes from the Land PEATE
CERES RDRs: Science, Diagnostic, and Telemetry
File sizes are 1370.8 KB, 685.5 KB and 27.5 KB; excluding HDF overhead, about 700 KB/file
Estimate: (1370.8 + 700) KB/file * 131 files/day + 685.5 KB/file * 1 file/day+ 27.5 KB/file * 12 files/day
= 272.2903 MB/day
Sub-sampled VIIRS radiance and geolocation data:
Output File Size: Approximately 400 MB per 5-min file
Estimate:
400 MB/file * 288 files/day = 115,200.0 MB/day
VIIRS Aerosol data:
File Size: ~ 632 MB per 5-min file, as of June 1, 2008
57 MB per 5-min file if changing to 6 x 6 aggregation for both Land and Ocean
32 MB per 5-min file if changing to 8 x 8 aggregation for both Land and Ocean.
Estimate:
~12 files per hour or ~288 files/day * 632 MB/file = 182,016 MB/day
~12 files per hour or ~288 files/day * 57 MB/file = 16,416 MB/day
Note: The Land PEATE expects to switch to 6x6, or possibly 8x8, aggregation prior to launch
Total Volume - Large Aerosol Files = 297,488.29 GB/day 108,583.226 GB/yr or 108.58 TB/yr
Total Volume - Small Aerosol Files
= 131.888.29 GB/day 48,139.226 GB/yr or 48.14 TB/yr
CERES FM-5 Delta Design Review
73
Security Compliance
•
ASDC Approval to Operate Based on August, 2007 Review
– System Security Plan: SC-010-M-LRC-1000
 Approving Official: Steve Jurczyk, Deputy Center Director
 System Owner: John Kusterer, ASDC Manager
 System Security Manager: Michael Little, ASDC Systems Engineer
– Re-certification in progress based on CIO mandated schedule re-alignment
– Re-certification based on ANGe/IBM installation to occur next spring
 Include consolidation of SCF and ASDC Systems into single System
•
IT Security Issues
– Reconciliation of OCIO Mandated ODIN conversion with ASDC ITS needs
 ODIN to take over support of workstations, desktops, etc
 ASDC to retain control/responsibility for servers
– Implementation of unannounced ITS measures by OCIO sometimes interferes with
ingest and delivery over network
 Agency CIO mandated transfer of all Center Firewalls to NISN is not fully understood
CERES FM-5 Delta Design Review
74
CERES Production Code Test & Evaluation
•
Proven Processes Based on 10 Years of Experience in CERES Code Delivery,
Acceptance, Testing, Evaluation and Production
•
CERES DMT Develops Code for Production of Data Products
–
–
–
–
–
–
–
•
PGE(s) for each Sub-system (SS) tested and internally verified
DMT and ASDC develop test cases
Delivery to ASDC includes sample data, Operators Manual, code
Delivery Evaluation and Code Compilation by DMT Configuration Management comparing to
sample data products and using test cases
Delivery to ASDC for SS Integration and Test to evaluate producibility and completeness and
comparison to sample data products
ASDC and DMT work together to identify and resolve any problems through Software Trouble
Tickets
Promotion to Production status and ASDC CM when approved by Operations Readiness
Review (ORR)
Data Product Production Validation Testing
–
–
Once code is in production, sample data products (ValRx) are produced with appropriate key
months
ValRx data products are evaluated by Science Team and approved prior to release into an
orderable status
 Unsuitable results may result in re-delivery
–
Release also requires a Quality Summary by the SS Science Lead
CERES FM-5 Delta Design Review
75
ASDC Validation Testing Approach
Systems delivered to the ASDC undergo Validation testing in a controlled environment
Problem report resolutions
incorporated into
configured software for
redelivery (if needed) to the
ASDC for validation
Problem reports generated
and tracked for all identified
problems
Resolutions
DMT
Subsystems
under CERES
CM delivered
to the ASDC
for Validation
Testing
CERES CM
Problems
Delivered
Subsystems
inspected for
completeness
by ASDC CM,
unique testing
scenarios
identified
Subsystems
validated in
operations
testing
environment for
all new
requirements
and
enhancements
ASDC CM
ASDC
Validation
Testing
Delivered
subsystems
regression
tested for
existing
functionality
ASDC
Regression
Testing
Operations
Readiness
Review (ORR)
following
successful
Validation
Testing and/or
mitigation of
existing issues
ASDC CCB
And Project
Management
Data
Production
and
Operations
ASDC
Operations
CERES FM-5 Testing to Operations Life-Cycle
CERES FM-5 Delta Design Review
76
NPP-Level Testing and Status
• Testing of the RDRs requires the SD3E to access data from the IDPS,
the RDRs are then accessed by the Land PEATE and sent to ASDC
– ASDC will ingest the RDR data and acknowledge receipt
– ASDC will test data production when available after 05/09
• Testing of VIIRS data requires Land PEATE to run sub-sampler and
send sub-sampled radiance/geolocation and aerosol data to the ASDC
– Land PEATE must be able to generate or access valid VIIRS data
– ASDC will ingest data and acknowledge receipt
• Ingest capability will be ready to support Early EEO
• Ingest and data production capability will be ready to support NCT3
• Interim delivery to address NCT3 problems to support NCT4 success
• All NPP-related functionality available at L-90 days for Launch
CERES FM-5 Delta Design Review
77
CERES ASDC RDR Tests
• Objective
– Verify that the Land PEATE can access CERES RDRs from the SD3E and
successfully transfer the data to the ASDC for ingest and archive
• Description
–
–
–
–
–
–
SD3E reads CERES RDR data from IDPS or through alternative source
Land PEATE accesses RDRs from the SD3E and transfers data to the ASDC
ASDC ingests RDRs and acknowledges receipt back to Land PEATE
ASDC archives data for staging and processing by Instrument Subsystem
ASDC will test data production when systems and RDRs available after 05/09
Additional reorder and data management functions will also be tested
• Input
– CERES RDRs generated by IDPS or from Proxy data provided to Land PEATE
• Output
– CERES RDRs ingested and archived at ASDC
– CERES Level 0 data for instrument subsystem processing available after 05/09
• Participants
– ASDC, Land PEATE, SD3E, CERES DMT
– IDPS when CERES RDR capability is available
CERES FM-5 Delta Design Review
78
CERES ASDC VIIRS Tests
• Objective
– Verify execution of the VIIRS Sub-sampler by the Land PEATE and the
transfer of the sub-sampled radiance/geolocation and aerosol data to the ASDC
• Description
–
–
–
–
–
–
–
–
SD3E reads VIIRS data from IDPS or through alternative source
Land PEATE must be able to generate or access valid VIIRS data
Land PEATE executes VIIRS Sub-sampler and sends sub-sample to ASDC
ASDC ingests sub-sampled data and acknowledges receipt to the Land PEATE
ASDC archives data for staging and processing by Clouds subsystem
Sub-sampled and aerosol data examined by science and DMT staff
ASDC will test data production when systems are available after 05/09
Aerosol data ingest and archive tested when available from the Land PEATE
• Input
– VIIRS data generated by IDPS or from Proxy data provided by Land PEATE
• Output
– VIIRS sub-samples generated by the Land PEATE
– VIIRS aerosol data product generated by the Land PEATE
• Participants
– ASDC, Land PEATE, SD3E, CERES DMT, IDPS
CERES FM-5 Delta Design Review
79
Operational Support
•
System Upgrades will Enhance Current Capabilities
– CERES production system already runs at the ASDC
– ANGe storage management upgrade should be operational this week
– Hardware upgrades to be phased in with existing CERES Ingest, Archive,
Distribution and Production capabilities in Dec09, reducing risk to NPP needs
•
Procedures and Documentation are available and tested
– Operations staff with CERES processing experience already in place with defined,
proven procedures
– ASDC staff already work closely with Land PEATE staff through interactions
supporting MODIS data transfers provided from the same data center (MODAPS)
•
System availability will remain at 24x7
– Current production model has minimized need for 24x7 staffing
– System hardware and architecture evolution will increase system reliability and
reduce workforce requirements
•
Improved system performance and additional CERES data products will
lead to increased customer satisfaction
CERES FM-5 Delta Design Review
80
Documentation and Agreements
•
Most documentation and agreements already exist as part of CERES on
TRMM, Terra, and Aqua
– http://science.larc.nasa.gov/ceres/docs.html
– Documents may be updated if needed
– New documents may be written explicitly for NPP, to meet special requirements
•
Overall roles and responsibilities are documented in CERES Science and
Data Products Working Agreement
– Working agreement between CERES IT and NPP Project Office
– May need to be revisited when NPP transfers to NOAA leadership
– Working Agreement in hands of NPP SEWG
•
Operations Agreement (OA) being developed between ASDC and Land
PEATE
– Based on existing MODAPS OA for MODIS data sent to ASDC
– On-going relationship between MODAPS and ASDC for MODIS
•
No OA planned for ADS/CLASS interface
– Expect to use standard customer subscription agreement
CERES FM-5 Delta Design Review
81
ASDC Supports NPP Schedule
08/31/08
09/15/08
Sep-Nov08
11/13/08
Nov08
Nov/Dec08
11/26-12/23/08
12/01/08
Jan/Feb09
05/08/09
05/15/09
06/09-15/09
07/01/09-02/02/10
10/15/09
12/02-09/09
01/15/10
02/02/10
03/02/10
06/02/10
Initial Operational Capability (IOC): CERES in ANGe
Complete Installation/Configuration of IBM hardware
Test/Characterize IBM Installation
Initial Delivery of CERES RDR to ASDC Ingest
Complete LaTIS conversion into ANGe
Design/Develop/Test NPP Inst, VIIRS into ANGe
Complete CERES RDR and VIIRS ICD
IOC IBM ASDC+SCF Archive and Processing System
Debug Aerosols/VIIRS Sub-sample into ANGe with Land PEATE
SDS Test2: Instrument RDRs into ANGe from Land PEATE
Build 1 ORR: NCT3 Functionality
NCT3
Design/Develop/Test Other NPP into ANGe
Build 2 ORR: NCT4 Functionality/NCT3 Problem Corrections
SDS Test3/NCT4
Build 3 ORR: Launch-Ready System
Instrument/VIIRS IOC
MRR
NPP Launch
CERES FM-5 Delta Design Review
82
Overall Schedule
• CERES Schedule is baselined
• CERES Schedule incorporated into NPP Schedule
• Schedule available as separate handout
CERES FM-5 Delta Design Review
83
Resource Assessment
•
Required resources have already been assessed and hired
•
Current experienced CERES staff leveraged for CERES FM-5
•
CERES FM-5 staff augmented with key hires to address technical
challenges
– Major SSAI job fair resulted in hundreds of resumes
•
Backfill positions created in current CERES support filled with new
talent
•
Resource estimate based on approximate lines of code to be generated
•
Using industry productivity standards, CERES CARS has the resources
in place needed to do the job
•
Additional resources needed for NPP testing, processes, and working
groups not included in industry standards
•
Additional issues and requirements expected
•
Ongoing CERES projects supported by SSAI can provide short-term
support if critical needs arise
CERES FM-5 Delta Design Review
84
Risk Analysis
Title
Raw Data
Record
(RDR)
Formats Not
Fully
Defined
Description
Given that CERES Raw Data
Record (RDR) formats are not
fully defined or understood in
time, there is a possibility the
code needed to support testing
may not be complete.
L
C
Impact Owner Action
2
1
Technical
Schedule
Erika
Geier
M
Comments
05/29/08
The CERES team
is developing a
giver/receiver list
and this item is on
the list.
07/23/08
IPO/Raytheon and
Janet Smith all
Know about this
risk.
Unavailability
of CERES
Raw Data
Record
(RDR) Test
Data
Given that CERES Raw Data
Record (RDR) test data from
IDPS is not available prior to
testing, there is a possibility
the code needed to support
testing may not be complete.
3
Legend: L – Likelihood
1
Technical
Schedule
Erika
Geier
W
07/23/08
We have requested
test data; although,
it doesn't sound
like we'll get
meaningful test
data.
C – Consequence
CERES FM-5 Delta Design Review
M – Mitigate
W - Watch
85
CERES Issues/Concerns
• Need single sample of the Science, Diagnostic,
Housekeeping/Telemetry RDRs to verify that we understand
format no later than 10/15/08
– CERES data content may be proxy, sample, or fill data
– Prefer attitude/ephemeris not be fill data
• Desire 3 days (~ 28 consecutive hours) of CERES RDRs to run
through preprocessor and Instrument Subsystem to aid in code
development no later than 11/14/08
CERES FM-5 Delta Design Review
86