Transcript TIGGE Data Archive at NCAR 8th GIFS-TIGGE Working Group World Meteorological Organization Geneva 22-24 February, 2010 Doug Schuster Steven Worley Dave Stepaniak Others.

TIGGE Data Archive at NCAR
8th GIFS-TIGGE Working Group
World Meteorological Organization
Geneva
22-24 February, 2010
Doug Schuster
Steven Worley
Dave Stepaniak
Others
Archive Summary
 Online Data
 Period, most recent two weeks
 ~ 5 TB , public products
 ~ 1 TB, data preparation, subsetting
 ~ 65 GB Tropical Cyclone Track Data (CXML)
 Offline Data
 Full period of record
 ~ 300 TB, NCAR MSS system
Provider Summary
 Archive Status
 Service Improvements
 Special Projects
 Metrics
 Future Service Additions
Data Delivery (02/2010)
• Data exchanges using UNIDATA LDM, HTTP and FTP
• 440 GB/day (ECMWF = 332 GB/day)
• 1.7+ million 2-D gridded fields/day
Archive Inhomogeneities
Parameter vs Provider Matrix
Variable
LvL
Geopotential Z
PL
Specific H
PL
T
PL
U-velocity
PL
V-velocity
PL
Potential Vor
PT
Potential T
PV
U-velocity
PV
V-Velocity
PV
U 10m
SL
V 10m
SL
CAPE
SL
Conv. Inhib.
SL
Land-sea
SL
Mean SLP
SL
Orog.
SL
Skin T
SL
Snow D. H20
SL
Snow F. H20
SL
ECWF
UKMO
JMA
NCEP
CMA
CMC
BOM
MetF
KMA
PL = Pressure Level, PT = 320K θ Level, PV = ± 2 Potential Vorticity Level, SL = Single/Surface Level
CPTC
Parameter vs Provider Matrix
Variable
LvL
Soil Moist.
SL
Soil T
SL
Sunshine D.
SL
Surf. DPT
SL
Surf. ATmax
SL
Surf. ATmin
SL
Surf. AT
SL
Surf. P
SL
LW Rad. Out
SL
LH flux
SL
Net Rad
SL
Net Therm. Rad
SL
Sensible Rad.
SL
Cloud Cov
SL
Column Water
SL
Precipitation
SL
Wilt. Point
SL
Field Cap.
SL
ECWF
UKMO
JMA
NCEP
CMA
CMC
BOM
MetF
KMA
PL = Pressure Level, PT = 320K θ Level, PV = ± 2 Potential Vorticity Level, SL = Single/Surface Level
CPTC
2009 Service Improvements
 Improved Web Portal Performance
 User data selection interface runs smoother.
 Automated capability to stage offline data for download.
 Added “Client for URLs” (cURL) based parameter
subsetting capability.
 Directly extract selected 2D grids from files across multiple
data providers.
 Downloads managed from user’s workstation.
 Script templates are provided to drive “subscription style”
parameter download capability.
 Strengthened data ingest processing.
 System monitoring (disk capacity, LDM status, etc)
 Failover mechanisms for LDM downtime.
 Data integrity checking.
2009/2010 Supported Special Projects
 NCAR Research Applications Laboratory (RAL)
Google African Meningitis Forecast Initiative
 Dr. Thomas Hopson
 Integrate weather and health data to manage African
meningitis outbreaks (in Ghana as starting point).
 Looking for 2-week lead time to position vaccines.
 Processing ECMWF data from present backwards
through the archive for model calibration.
 Total Precipitation, 2m Temperature, 2m Dew Point.
 Global Subarea 40 deg N to 40 Deg S.
 0.5 x 0.5 degree resolution.
 Assist RAL and Unidata in providing “Real-time” access to
TIGGE based Multi-Model Ensemble Forecast data.
 http://www.ral.ucar.edu/projects/ghana/
 http://www.google.org/predict.html
NCAR TIGGE Full Model Data Archive
Usage
TB
NCAR TIGGE Data Access Metrics
16
14
12
10
8
6
4
2
0
2006
cURL
impact
2007
2008
TB Accessed
TB Downloaded
2009
Year
YEAR
GB Accessed
GB
Downloaded
Files
Downloaded
Active Users /
Unique Users
2006
5
5
10
0/5
2007
900
900
32,000
10 / 50
2008
1,300
1,100
40,000
10 / 20
2009
13,700
1,900
68,000
10 / 30
2010
80
20
500
3/5
Total
15,985
3,925
140,510
33 / 80
NCAR TIGGE CXML Archive Usage
YEAR
GB
Downloaded
Files
Downloaded
# Active Users /
# Unique Users
2008
1
860
7 / 30
2009
26
12,300
17 / 35
2010
0.5
500
3/6
Total
27.5
13,660
27 / 81
Planned Service Improvements
 Additional Online Storage
45 TB data online (3 months of most recent data).
15 TB for data processing request output.
Interface driven data processing and immediate download
capability extended from 2 weeks to 3 months.
 Improved “Offline” Storage.
Data being transitioned from current NCAR MSS system to
new HPSS system.
Improved access speeds to offline data.
Add data processing capability to offline data?
 Expanded/Updated Computing.
Improved LDM ingest reliability.
Faster “turn-around” time on user sub-set requests.
Potential for subscription services?
Potential Service Improvements
 Model Validation Data Portal (Pending
Funding)?
 Wide variety of observational and analyzed data sources.
 In Situ Observation data (NCEP operationally collected data).
 Ocean based observations (ICOADS).
 Global tropical cyclone position and intensity data.
 Atmospheric Reanalysis (JRA, ERA, NNRR, CFSR).
 Global PCP datasets.
 Suggested TWG sets to support FDP’s.
 Will need discussions to set priorities for development.
 Data sub-set and output format selection capability.
 Integrated With TIGGE model data portal
 Data analysis tools and codes.
 Model Evaluation Tools offered by the Developmental Testbed
Center at NCAR
http://www.dtcenter.org/mrt/users
http://tigge.ucar.edu/