Transcript Status of EOS MLS Science Data Processing

Science Processing Status
of the
Earth Observing System
Microwave Limb Sounder
(EOS MLS)
Elmain Martinez
September 27, 2010
Aura Science Meeting - Data System Working Group
Boulder, Colorado
Jet Propulsion Laboratory
California Institute of Technology
Copyright 2010 California Institute of Technology. Government sponsorship acknowledged.
Topics
Aura
•
•
•
•
•
Recap MLS Interfaces & Dataflow
Data Processing Status
Software Status
Processing Environment Improvements
Summary
27 September 2010
EOS MLS
MLS
2
System Interfaces and Dataflow
UK MLS Science Computing Facility (SCF)
US MLS Science Computing Facility (SCF)
SDP Toolkit
• Scientific Algorithm Development
• Scientific Analysis
• Science Processing SW Development
• Quality Control & Assessment
Meteorological Data
Delivered Algorithm Package
Data Quality Assessment
Data Management Layer
All SIPS Inputs
All Standard Products
Diagnostics
Logs
PGEs
Calibration Files
Production Policies
Science Investigator-led Processing System (SIPS)
Product
Generation
Executables
(PGEs)
MLS L1B/2 PDRs
DN, PAN, PDRD
Scheduling/Planning Layer
Fetch
Input
Data
EOSDIS
Core
System
(ECS)
MLS L0 – PDS
Aura SC Engr.
Aura SC Definitive Orbit
Aura SC Attitude
Operational Meteorological Data
Leap Second & UTC Pole Data
Store
Output
Data
Goddard
Space Flight
Center (GSFC)
Earth Science
(GES)
Data and
Information
Services
Center (DISC)
Data Management Layer
27 September 2010
EOS MLS
3
DATA PROCESSING STATUS
27 September 2010
EOS MLS
4
MLS Near-Real-Time Status
• SIPS NRT configuration is continuing to deliver Ozone and
Temperature L2 products using V2.23
– Ingests pre-processed NRT MLS PDS data from the GES DISC
• 15 minute granules
– L1 and L2 PGS software was modified in 2008 to improve product
delivery timeliness
– L2 products are delivered to SCF and GES-DAAC; available ~3 hours
after observation
• Customers include GMAO, NOAA/NCEP, NOAA/NESDIS/STAR,
Environment Canada, ECMWF, and Met Office
27 September 2010
EOS MLS
5
Forward Processing Status
• SIPS is processing all incoming data with V2.23
– L1/2 products delivered to SCF and GES-DAAC
• V2.23 in operation since March 2008
– This is the 4th minor versions of V2
•
•
•
•
v02.20 – initially released Nov 2006
v02.21 – de-convolve DACS radiances in Level 1
v02.22 – increased performance without change of quality
v02.23 – dealt with data variances produced by Aura anomaly
– (data since Dec 4, 2008)
• We will transition from V2.23 to V3.3 once the Data Quality
Document is released
27 September 2010
EOS MLS
6
Reprocessing Status
• V3.10 reprocessing halted in Nov 2009 due to data quality
concerns
– Products delivered to GES DISC, but never publicly released; now removed
• SIPS is currently reprocessing all science data with V3.30
– Reprocessing began in April 2010
• More than halfway through the 72 months (2273 days) to be reprocessed
– V3.30 products being delivered to GES DISC; to be publicly released in Oct.
2500
Days Processed
2000
1500
Reprocess Forecast
1000
Reprocessed
Forward Proc. V3.3
500
0
27 September 2010
EOS MLS
7
SOFTWARE STATUS
27 September 2010
EOS MLS
8
Major Improvements in V3.3
• Software changes:
– Processing with GEOS5 meteorology for dates after Sept 1, 2009
• MERRA data for dates prior
–
–
–
–
–
–
Using HDF5 1.8.3 and SDP Toolkit 5.2.16
Products are augmented to be HDF5-NetCDF compliant
Software built with 64-bit compiler
Additional error handling added to scheduler
Products to be released to public once the Quality Document is finalized
See the poster “Version 3.30 Products from EOS MLS” for further details
• Science improvements
– CH3CL (Methyl Chloride) is a new standard product
• See the Validation and Climatology poster on this species
– Revised spectroscopy to reflect best-fit to unknown continuum
– Special handling of Core+R5 phase if THz is turned off
27 September 2010
EOS MLS
9
Version 4 Plans
• Product Generation Software
– Target release date is Spring 2012
– Improvements currently planned:
•
•
•
•
Read various parameters using HDF5, e.g. temperature
Retrieve species in clouds
Flag radiances to not be used when clouds are detected
Generating a new product, CH3OH
– Intended to account for radiances seen in other species
– Integrate latest available version of HDF5 and SDP Toolkit
• NRT improvements:
–
–
–
–
XML-formatted metadata
Browse product for choosing granules to download
Capability for Data Casting (RSS)
Better forward model such as 2nd derivative
27 September 2010
EOS MLS
10
PROCESSING ENVIRONMENT
IMPROVEMENTS
27 September 2010
EOS MLS
11
SIPS Processing Environment Improvements
• Replaced old IBM 32 bit production cluster with a Dell 64 bit
cluster in June, 2010
– Specs
•
•
•
•
•
48 nodes
8 CPUs per node
146 GB storage per node
2.93 GHz
24 GB memory (vs 16GB on older clusters)
• Benchmark tests for a single day completed 3 hours faster
compared to an older cluster; ~27% improvement in speed
– About 6.34 additional days per week with the 27% performance
improvement
– With the additional cluster and its higher relative performance, weekly
output has increased by apx. 22 additional days per week
27 September 2010
EOS MLS
12
SIPS HW/Network Architecture
JPL router
sips-switch (LAN)
JPL net
Operations
Dev/Test
NRT
SIPS Server
Dev/Test (2)
L2 (V3.30)
sftp Server
sftp Server
L2 (V3.30)
File Server/
L1B (V3.30)
L2 (V3.30)
DB Server
Changes since 9-09:
L1B (V2.23)
L1B (V2.23)
L2 (V3.30)
DB Server
L2 (V2.23)
NRT Server (2)
L1B (3, V30)
• Operations:
- Replaced aging L2 Ops cluster
- Replaced aging L1 server
• Development
- Replaced aging NFS/L1 server
Replaced
27 September 2010
EOS MLS
13
SIPS Future Plans
• No upgrades planned for next FY
• FY12 and beyond we will be looking at:
– Additional storage for V4 production
– Replace aging servers
– Replacing aging cluster
27 September 2010
EOS MLS
14
SCF Processing Environment Improvements
• Improvements in the last year:
– Added 169 TB raw NetApp storage
– Retired standalone NAS devices due to age.
• Moved data to the 60 TB NetApp
– Retired aging test cluster
– MLS webpage upgraded
• Future plans:
– No upgrades planned for next FY
• but there never seems to be enough storage…
– FY12 and beyond we will be looking at:
• Purchase additional storage for V4 processing
• Replace 32 bit cluster with 64 bit
• Replace aging NetApp filer
27 September 2010
EOS MLS
15
SCF HW/Network Architecture
JPL Network
SCF Internal Network
User Processing (4)
Touchscree
n
Test Cluster
Network Storage
224 TB
DB Server (+ backup)
NIS, Email (+backup)
Lab & Flight Ops (5)
Dev Cluster
Data Retrieval IF
Collaboration Server
Workstations (34)
Analysis Cluster
Web Server
Expanded
27 September 2010
EOS MLS
16
Science Community Support
• Mobile Phone data tracking of MLS daily plots
– Accessible from MLS webpage
• Hurricane portal (http://tropicalcyclone.jpl.nasa.gov)
– Tropical Cyclone Information System (TCIS) brings together satellite and in situ
data sets from various sources (e.g. MLS, OMI, QuikSCAT, and others) so users
can find information for a particular tropical cyclone.
• Site up since April 2010
• 2005 data loaded, other years to follow
• Site has an analysis tool that allows users to plot histograms, maps, and profiles for
many different data sets and products.
• Uses MLS data products for temp, ozone, CO2, water vapor
• MLS webpage improvements, e.g. NRT Viewer
– See the poster “The Microwave Limb Sounder Website: Improved Access to Data and
Information” for more information
27 September 2010
EOS MLS
17
SUMMARY
27 September 2010
EOS MLS
18
Recap of Future Plans
• V4 is about 19 months away
– Algorithm cycle is usually 18 months or more, but it could be sooner if
Level 2 algorithms can mature
• SIPS hardware is stable and should be for a couple years
• SCF to make another leap to keep up…to purchase a 64 bit
development machine and additional storage in the future
27 September 2010
EOS MLS
19