Transcript GDAC Status - GlobColour

Distribution of Medspiration products
from JPL Global Data Assembly
Center (GDAC)
GHRSST Medspiration and GlobCOLOUR Users
Meeting
4-6 Dec 2006
Villefranche sur mer, France
Ed Armstrong, Jorge Vasquez, Andrew Bingham, Sue Heinz,
Alex Cervantes, Qui Chau, Tim McKnight, Wendy Zhang
[email protected]
Introduction - GHRSST data management
GHRSST international and regional task (and cost)
sharing model: Built on 3 core components
1) Regional centers of SST expertise produce near realtime
data (L2P/L4) for their sensors
2) Distribution and coordination from regional centers requires
“Clearinghouse” for data and metadata management
• Fulfilled by the JPL Global Data Assembly Center (GDAC)
• Built on the 13 year expertise of the NASA Physical Oceanography
DAAC in satellite data management
3) Archiving and reanalysis occurs the Longterm Stewardship
and Reanalysis Center (NOAA NODC)
The GDAC serves as the link between data users and
producers, and between producers and longterm data
stewardship (between 1 and 3)
GDAC System Architecture
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
Overview of GDAC tasks
Ingest realtime project data files
L2P, L4, L2P_GRIDDED data and metadata files from RDACs
Stage on public FTP site for 45 days
Deliver to long-term archive (LTSRF; NOAA NODC) after
30 days
Provide a variety data access methods (FTP, OPenDAP,
POET)
Ingest and quality assure metadata into the Master
Metadata Repository. Produce FGDC metadata products
for LTSRF
Provide web-base interface for querying database and
data discovery
Ancillary data production for unfilled L2P files
Provide USO and applications development
MODIS L2P (collaboration with the NASA OBPG)
Support and value added services
FGDC metadata
Creating and staging FGDC metadata for LTSRF interface
OPenDAP
L2/L4 accessible via JPL server http://dods.jpl.nasa.gov
IDL and C L2P software readers
Read entire L2P structure, can be used to test integrity of L2P
RDAC assistance with L2P/L4/HRDDS file format and metadata
anomalies
L2P subsetting design in progress
Value add L2P with ancillary filling
Wind , sea ice, solar insolation, aerosol ( from ECMWF, FNMOC
etc.)
Filling for solar insolation
ECMWF Solar
insolation
Same day
AVHRR L2P
filled with solar
insolation data
Services continued
Web portal to GHRSST activities and data:
http://ghrsst.jpl.nasa.gov
Webex site for intraproject collaboration on applications and
user information
Application User Services (AUS)
Supporting applications development (S. Heinz)
Actively addressing user inquiries
Data products and supported sensors
27 unique L2 and L4 products
GHRSST
GHRSST
GHRSST
GHRSST
GHRSST
GHRSST
GHRSST
GHRSST
GHRSST
GHRSST
GHRSST
GHRSST
GHRSST
GHRSST
GHRSST
GHRSST
GHRSST
GHRSST
GHRSST
GHRSST
GHRSST
GHRSST
GHRSST
GHRSST
GHRSST
GHRSST
GHRSST
Level
Level
Level
Level
Level
Level
Level
Level
Level
Level
Level
Level
Level
Level
Level
Level
Level
Level
Level
Level
Level
Level
Level
Level
Level
Level
Level
2P European Medspiration AA TSR SST
2P European Medspiration A VHRR16 Global SST
2P European Medspiration A VHRR16 Local SST
2P European Medspiration A VHRR17 Global SST
2P European Medspiration A VHRR17 Local SST
2P European Medspiration A VHRR18 Global SST
2P European Medspiration A VHRR18 Local SST
2P European Medspiration NAR16 SST
2P European Medspiration NAR17 SST
2P European Medspiration SEVIRI SST
2P European Medspiration AMSRE SST
2P European Medspiration TMI SST
4 European Medspiration Ult ra-High Foundation SST
2P European Medspiration NAR18 SST
4 European UKMET OSTIA SST
2P USA Remote Sensing Syst ems AMSRE SST
2P_GRIDDED USA Remote Sensing Systems AMSRE SST
2P USA Remote Sensing Syst ems TMI SST
2P_GRIDDED USA Remote Sensing Systems TMI SST
2P USA NA SA MODIS Terra SST
2P USA NA SA MODIS Aqua SST
2P NAVOCEA NO A VHRR18 Local SST
2P USA OSDPD GOES11 SST
2P USA OSDPD GOES12 SST
2P USA NAVOCEA NO A VHRR17 Local SST
2P USA NAVOCEA NO A VHRR17 Global SST
2P USA NAVOCEA NO A VHRR18 Global SST
AVHRR
AATSR
MODIS
AMSRE
TMI
SEVIRI
GOES
L4 blends
MODIS L2P
SST
Bias
Flags
STD
MODIS L2P granule with OC
SST
K490
Chlorophyll A
GOES L2P
SST
AOD
SSI
Wind
Data Ingest Statistics -2006
2006 data ingest volum es
3.5 TB total
700
Volume in GB
600
500
400
300
200
100
0
1
2
3
4
5
6
7
8
9
10
11
10
11
month (2006)
2006 files ingested
327408 files total
70000
Number of files
60000
50000
40000
30000
20000
10000
0
1
2
3
4
5
6
7
month (2006)
8
9
Data Distribution Statistics -2006
Volume distributed in 2006
Total is 3.1 TB, 560K files
1200
Users
1000
EUR
180
48
NAVO 5 3
REMSS 4 0
GB
800
JPL
600
400
200
0
1
2
3
4
5
6
7
8
9
10
11
month (2006)
User breakdown
2
1
Brazil
78
Canada
99
Commercial
Denmark
France
JPL
US Defense
NASA
3
9
Org
Russia
UK
8
1111
Unknown
53
Vol(GB)
1 5 3 .5
1 0 4 4 .1
1 8 0 6 .8
9 1 .9
#Files
45795
187941
211018
117036
Data uses and impact studies
Data Use and Impact Studies from the Multisensor Improved
Sea Surface Temperature Project (MISST)
NAVOCEANO: impact of assimilation of high resolution L2P
including AATSR and SEVIRI on NCODA (NRL Coupled Ocean Data
Assimilation system) and K10 merged SST product
NOGAPS TC track
Although there was no significant difference in the track
forecast errors overall, there were areas where the use
of the MISST SST analyses resulted in significantly
improved NOGAPS TC track forecasts
Data Use Cases and impact studies
Additional Data Use and Impact Studies
NOAA-RSS: Hurricane intensity impact used blended microwave L4
UK Met: L2P for OSTIA L4 merged products
RSS: L4 blend using AMSRE, TMI, MODIS
NOAA: improved Reynolds OI using TMI and AATSR L2P
FNMOC: evaluate AATSR and SEVIRI in operational SST blend
Canadian Meteorological Forecast Office: AVHRR L2P
MISST research
MISST research
Aerosols
• AATSR used to study impact of aerosols on infrared measurements as
compared to AVHRR and MODIS
– Crucial for assigning flag or weight on IR measurements in SST blends
Diurnal warming models for deriving foundation temperature
• SEVIRI measurements have excellent calibration and resolution
necessary to resolve the daily diurnal cycle of SST
Skin layer temperature
• Used to derive CO2 flux maps (B. Ward)
• AATSR has most accurate skin layer measurement available
Data Use Cases continued
US regional users for climate, weather and ecosystem
studies at federal or state agencies, non profit organizations,
universities coordinated by the GDAC.
• Shorterm prediction and research center (SPORT)
SPORT supports infusion of NASA
observations, data assimilation and
modeling research into NWS forecast
operations and decision making at the
regional and local level.
Using the OSTIA product for modeling
and forecasting activities for the US
Gulf and North Atlantic regions.
SPORT is generating daily images
available at:
http://weather.msfc.nasa.gov/
sport/sstAnimation/
Data Use Cases
Pacific Disaster Center
The PDC has pledged their support to prepare and disseminate gridded GHRSST data
via the APNHVA.
Atlas is published both as Asia Pacific basin-wide and Hawaiian Island region.
Support disaster community applications for preparedness, mitigation, response and
recovery activities.
User include Hawaii State Civil Defense, FEMA, NOAA, UN agencies, Thailand’s National
Disaster Warning Center and Vietnam’s Disaster Management Committee.
http://www.pdc.org
Data Use Cases continued
US regional users for climate, weather and ecosystem
studies at federal or state agencies, non profit organizations,
universities.
• Shorterm prediction and research center (SPORT)
• Pacific Disaster Center
– Asia Pacific Natural Hazards Information Network
•
•
•
•
NASA Aquarius Salinity Mission
Gulf of Maine Ocean Observing System (GOMOOS)
Southern California Coastal Ocean Observing System (SCCOOS)
Ecosystem Management
– California State University Humbolt Bay
– NOAA Protected Species Management
• Southeast Phytoplankton Monitoring Network (SEPMN)
• ECCO assimilation
GHRSST metadata model
Follows the NASA Directory Interchange Format (DIF)
Granule records ( File Record [FR] )
Dataset record ( Dataset Description [DSD] )
XML based
L2P/L4 products only accepted with valid metadata
Metadata ingested into a “clearinghouse”: the Master Metadata
Repository (MMR)
Support for FGDC metadata standard implemented for
distribution and archiving at the LTSRF
Future support
ISO19115
Master Metadata Repository (MMR)
MMR
mySQL database (ver 4x)
• Fully relational, transactions implemented
Separate multi-table relational databases for DSD and FR
records linked with one common field
DSD metadata records jointly maintained by Met Office and
GDAC
Over 150K FR records ingested without error
QA checkers/XML parsers for FR and DSD records
implemented
Web search interface
Supports output to FGDC format through XML stylesheet
conversions
MMR Search Interface
Interface to the LTSRF
FGDC records created for 30 day (and older data).
30 day old data and metadata extracted from an
operational FTP site by NOAA
GHRSST data products expired from GDAC
(PO.DAAC) FTP site after 45-60 days (“rolling store”)
Size of the rolling store can easily be adjusted to
accommodate changing disk space resources
To date 100% file integrity between GDAC and LTSRF
DIF to FGDC conversion
Extract FR from MMR
Format to XML
Extract and
format DSD to XML
Apply XSL
FGDC formatted record
GDAC future activities
Integrated into the PO.DAAC in 2008
ISO19115 support
Routine Ancillary L2P filling
Delivery of L4 and L2P products via the POET
visualizer and subsetter
Google Earth browse capability
Additional L2P, L2P_GRIDDED, and L4 products
NetCDF 4 support
Ingestion of metadata into ECHO
GDAC team
Jorge Vazquez - Project manager
Ed Armstrong - GDAC Cognizant Engineer
Andy Bingham - Data management and Oceanids developer
Sue Heinz - Applications and User Services lead ([email protected])
Alex Cervantes - Software engineer for Oceanids development and L2P
ancillary fill
Qui Chau - Software engineer for L2P ancillary fill, MODIS L2P, L2
subsetting development
Wendy Zhang - Software engineer on MMR
Tim McKnight - Operations engineer ([email protected])
Pat Liggett - PO.DAAC team and management
Backup - Data Access
ftp://podaac.jpl.nasa.gov/GHRSST
Organized by:
datatype / sensor / RDAC / year / DOY
OPenDAP:
dods.jpl.nasa.gov
User support:
Email [email protected]
OCEANIDS OVERVIEW
REPORTS &
ACCOUNTABILTY
DATA PROV IDERS
INGEST / QC
DATA
STORE
CATALOGUE
DISTRIBUTE
USERS