Transcript CDAAC status and strategy

CDAAC Status and Strategy
Bill Schreiner
and CDAAC Team
COSMIC Retreat
Nov 4, 2010
Outline
•
•
•
•
•
CDAAC Objectives
CDAAC Overview
CDAAC Status
Summary of CDAAC mini-review
Processing COSMIC Follow-on missions and missions of
opportunity
CDAAC Objectives
• Provide high quality GNSS RO data to the community in
NRT, post-processing, re-processing (re-analysis)
• Improve data processing algorithms to maximize impact of
RO data on research and operations
• Support data users, TACC
• Conduct scientific investigations when appropriate and
efficient
CDAAC Personnel
•
•
•
•
•
Program Director: Bill Kuo
Chief Scientist: Chris Rocken
CDAAC Manager: Bill Schreiner
S/W Engineers
– Doug Hunt, Lead
– Maggie Sleziak-Sallee, Web
– Teresa Vanhove, GPS
– Jason Lin, SE
– Karl Hudnut, System Admin/Engineer
Scientists
– Bill Schreiner, GPS Lead
– Sergey Sokolovskiy, RO Inversions
– Xinan Yue, Ionosphere
– Janet Zeng, RO Inversions
– Tae-Kwon Wee, 1DVAR
Main CDAAC Functions
• RO Payload Operation
– Configuration control (firmware and tracking configuration)
– Scientific/technical guidance for commanding, operation
– Near real-time monitoring, trouble-shooting, and Q/C data analysis
• Data Processing and Analysis
–
–
–
–
–
–
–
–
Level0 unformatting and QC
GPS ground processing (ZTD, site estimation, clock estimation)
LEO POD, and atmospheric excess phase
Absolute TEC generation
Inversions (neutral atmosphere and ionosphere)
Retrievals (1DVAR)
NWP and correlative data handling
Product QC and analysis
CDAAC Functions (cont)
• System Operation and Monitoring
-
H/W, O/S and NFS filesystem
System fail-overs
CDAAC operations
• Input data stream monitoring
– RTS downlink
– GPS Bit-grabber operation and monitoring
– GPS ground data (IGS, NRCan, EUMETSAT, COSMIC
sites)
– IGS and IGU orbits, clocks and EOP
– Bernese configuration
– NCEP GFS, ECMWF, radiosonde, ionosonde
• Data Management, Dissemination, and Archival
• Support Data Users
CDAAC Hardware
Real-Time Setup
– Dual String system
– Redundant Input/Output IOcomputers
– Redundant Real-Time
Operational cluster computers
(with RAIDS)
– Real-Time staging cluster to
stage (test) software updates
– Backup power and cooling
Post-Processing Setup
– R&D cluster
– Re-Analysis cluster for climate
re-analysis
– FTP Server with RAID
Active
I/O
Computer
Backup
I/O
Computer
CDAAC Processing Flow
CDAAC File types in RED
Atmospheric processing
atmPhs
LEO data
Level 0--level 1
Excess Phase
Full Spectrum
Abel Inversion
atmPrf
(S4)
1-D Var
Moisture
Correction
wetPrf
Fiducial
data
POD (Orbits
and clocks)
bfrPrf
Real time Task Scheduling Software
Profiles
Ionospheric processing
podTec (TEC)
scnLv1 (S4)
tipLv1 (Radiance)
Excess Phase
Processing
ionPhs
(L1,L2
amp/phase)
Abel Inversion
ionPrf
(EDPs)
Combination
with other data
Best Effort Monitoring
• Monitor the system regularly throughout the
day M-F 8am-8pm
• CDAAC Ops team monitors the system 3
times per day on weekends and holidays
• Available by email and cell phone
• CDAAC reliability estimated > 99.5%
CDAAC Data Processing Status
• Near Real-Time
-
COSMIC
SAC-C and C/NOFS
• Post-Processing (2007.3200-2009.2650)
-
GPSMET, GPSMET-AS, CHAMP, SAC-C, GRACE, COSMIC
Will attempt to understand 3 “noisy” CHAMP months (2007.3200)
• Re-Processing (2009.2650)
-
Published new CHAMP re-processing
COSMIC re-processing finished, published soon
Then start re-processing for GPSMET, GRACE, TerraSAR-X, SACC, Metop/GRAS, C/NOFS
COSMIC Data Product QC
•
•
•
•
Output file saved from each process
Hard failures logged in cosmic_badprf table
Most products have BAD flags in files and database
Some product files include error characteristics
• LEO Orbit Determination (Post-fit residual test, Orbit overlap test)
• Neutral atmospheric excess phase files (SNR and phase noise tests)
• Neutral atmospheric profiles (Bending angle noise and bias tests, checks for L2
quality, presence of scintillation, gross checks on N/T/P)
• Absolute TEC data arcs (cycle slip detection/correction, outliers)
• Electron Density Profiles (under development)
• Scintillation Indices (under development)
• TIP Night-side Radiances (BAD flag for incomplete integration time, motor not at
proper destination, ill-defined sensitivity)
CDAAC 3.0 Status
• To be released by late November 2010
• RPM-based install system
• Improved data access, website, DB interface, monitoring,
configuration management, archiving, documentation
• All other packages updated to latest versions: perl,
postgres, apache, Bernese, ..
• Real-time clock subsystem
• Improved excess phase for SD processing
• Use of higher resolution real-time and post-processing first
guess models (GFS high resolution and ERA-40 interim)
CDAAC Review
• Held Oct 27, 2010
• What CDAAC development efforts are required for
COSMIC and other missions? How much effort?
• How do we know our CDAAC GPS RO data processing is
optimal?
– Review of algorithms and software internally and via publication
– Comparison of results (POD, BA, N, T, WV) with other data
processing software and investigate and understand differences (i.e.
more detailed GRAS study)
– Comparison with correlative data, models?
CDAAC System changes













Better system for managing code for production/research
Simplify addition of new missions
names in CDAAC code
Restructure ROAM
Refactor genXphase
Add more test suites
Clean up unused code
Continue to add missions
Improved QC, bad flags, error estimates
Multi-GNSS capability, new observables!
Low latency processing
Better systems management
Need to develop CDAAC Level0-Level1 processing modules
Improved monitoring scripts
GNSS Processing
• Add GNSS capability, new observables
• Improve NRT ZTD
- Compare algorithm with Suominet
• Improve NRT GNSS clock estimation
- 5-sec GNSS clocks
• Improve NRT and PP LEO POD (to 0.1 mm/sec)
– Process data from both POD antennas
– Compute/Apply PCV maps
– Implement SLR processing for validation
• Improve excess phase processing
- Review algorithms, fix bugs,
- 5-sec GNSS clocks
- refactor
• Perform additional validation studies
Neutral Atmospheric Inversions
• Restructure ROAM
• Add GNSS capability, new observables
• Improve wave optics processing
– looking for better filtering approaches
– looking for alternative methods (like recently introduced WDF)
• Improve bending angle optimization
- testing methods with reduced weight of climatology
- validation by independent data sources in the stratosphere
• Improve 1DVAR retrieval code and documentation
• Perform additional validation studies (e.g. integrate ROPP
package, RO-Trends+)
- Requires investigation and understanding the sources of the
differences
atmPhs
1) Phase Connection
conPhs
2) Bending Angle
Generation (WO,GO)
benPrf
3) Bending Angle
Correction, Connection
bcnPrf
4) Bending Angle
Optimization, Inversion
atmPrf
1) Mission-dependent; 2-4) Mission-independent.
1) Input positions, velocities, raw phase and amplitude,
clim. model.
Processing removal of NDM, connection of the phase,
down-sampling to one rate.
Output positions, velocities, connected phase,
amplitude, HSL, lat, lon, height of TP.
2) Input output from 1
Processing retrieval of WO (Phase Matching and FSI
or CT2) and GO bending angles for L1, L2, (L5)
Output GO and WO bending angles for L1,L2,(L5)
3) Input output from 2
Processing ionospheric correction (incl. additional
smoothing of L4), connection of GO and WO bending
angles.
Output ionosphere free connected bending angles
4) Input output from 3, clim. model, atm. model.
Processing optimization of bending angles,
inversion of N,T,P.
Output N,T,P.
Comparison of COSMIC retrieved temperatures to ECMWF for
Different optimizations of BA
2009.015-045
Current CDAAC approach:
- clim. fitted to obs. BA
- mixed with weighting function
- WF different for different occs.
- on average fitted clim. >50%
above h~45 km
Tested approach:
- obs. BA smoothed with window
increasing from 1.5 km at h=30 km
to 15 km at h=60 km
- clim. >50% above h=60 km
for all occs.
Ionospheric Processing
• Absolute TEC uncertainty under investigation
- DCB estimation, Code/phase leveling uncertainty
• Electron Density Profiles
- Improving EDPs by using information on horizontal gradients with DA
• Improve scintillation products
• Add GNSS capability, new observables
• Perform additional validation studies
- With Paul Straus of Aerospace, JPL
1-2 TECu uncertainty can be ignored for ground based GPS or low
elevation LEO GPS observations. But it might have problem if we want to
study the plasmasphere using COSMIC sTEC [eg,2006.230-245]
TEC Precision from Collocated Tracks
COSMIC (2006.230-245)
Data assimilation retrieval

Comparison of the
latitude and
altitude Ne and its
retrieval error from
simulation

LT=13

Manmade plasma
cave disappears in
DA retrieval.
Processing NRT data from MOOs at
CDAAC
• development, testing, monitoring of RTS downlink
• decoding and formatting binary level 0 science payload
data. Effort depends on payload architecture
• system modification, configuration and management
• data processing modifications
• testing and validation
• NRT operations and monitoring,
• post-processing, archival, maintenance
• program management
• Recent experience with C/NOFS and SAC-C valuable
C-2 Operational Data Processing
Requirements
•
•
•
•
•
•
•
•
•
•
•
Reliable and low latency input data streams
Primary and Backup DPC (Data Processing Center)
Development system (GNSS capable)
Staging system to test processing changes
Configuration management of systems
Communication access between systems (DPCs,
SOCC, RTSs). Increased IT Security?
24/7 monitoring of near real-time operations
Maintenance and on call technical support
Operator Training
Secondary Science payload processing
Post-Processing
Configuration Management for C-2
Cfengine controls
Cfengine
Configuration
Control
Machine
SVN
configuration
repository
• Log rotation
• Shared file systems
• Processing Daemons
• Software integrity
• Fail-over between
I/Os and processing
clusters
• Disk space/cleanup
• Syncing
Primary
DPC
Staging
Single string
Backup
DPC
Development
Single string
PostProcessing
C-2 DPC Options
with CDAAC
Option A
Option B
Primary
NOAA (single string hw ~$100K)
NOAA (single string hw ~$100K)
Hot Backup
UCAR (single string hw ~$100K)
UCAR staffing (TBD)
TDPC (single string hw ~$100K)
TDPC staffing (TBD)
Staging
NOAA (single string hw ~$100K)
NOAA (single string hw ~$100K)
Monitoring
24/7 at NOAA (~5 FTE)
Best effort at UCAR (TBD)
24/7 at NOAA (~5 FTE)
24/7 at TDPC (~5 FTE)
R&D
UCAR (single string hw ~$100K)
UCAR staffing (TBD)
UCAR (single string hw ~$100K)
UCAR staffing (TBD)
On Call
Tech
Support
UCAR staffing (TBD)
UCAR staffing (TBD)
PostProcessing
UCAR (single string hw ~$150K)
UCAR (single string hw ~$150K)
CDAAC R&D Schedule for C-2
Recent Activities
• June 30th 2010 meeting with NOAA/OSDPD (Kathy Kelly)
- Review of GPS RO processing at CDAAC
- HW/SW data processing architecture at NSOF
- Held additional telecon
- Working to test CDAAC within NSOF architecture
- Providing info for NOAA CLASS archive
• Oct 14th 2010 meeting with AF/Aerospace
- Review of CDAAC and SW architecture
- Strong interest in using CDAAC to process space weather
payloads
- Additional meetings in planning
• F-7/C-2 JPMO Meeting Dec 7-8, 2010
- Discuss data processing issues
Recent RO Research and Science Apps
• Investigated structural uncertainty of RO in LT. Published in JGR
• Comparing different methods of estimation of the ABL depth from RO.
Investigating global variability. Publishing results soon.
• Detection of the sporadic E clouds from RO. In press at GRL
• Investigation of the response of the UTLS in terms of gravity waves
and tidal variations from COSMIC data to the sudden stratospheric
warming event in 2009
• COSMIC EDPs used to estimate ionosphere High Transition Heights
(HTH). In press at JASTP
• Investigated global ionospheric response observed by COSMIC
Satellites during the January 2009 Stratospheric Sudden Warming
Event. In press at JGR-SP
• Submitted data assimilation paper to improve EDPs. In revision at JGR
Recent RO Research and Science Apps
• Global Evaluation of Radiosonde Water Vapor with RO data, Published
in J of Remote Sensing
• Evaluation of AMSU and AIRS with RO
• Impacts of GPS radio occultation data assimilation on regional weather
predictions. Published in GPS Solutions
• Working on Round-2 of RO-Trends inter-agency comparison
• UCAR funded by EUMETSAT to evaluate Metop/GRAS RO data. Final
Report due in early Nov