Transcript JOSS Data Management (Williams)

http://www.joss.ucar.edu/ihop/dm/
Composites: What and Why
A composite dataset is a collection (over some time
period and region) of similar data (e.g. surface
met) from a variety of sources, put into a common
format, and passed through a uniform quality
control.
Why does UCAR/JOSS develop composites?
- Provides data in a uniform format with QC.
- Allows determination of network/site problems.
- Useful for model applications.
- Prevents duplication of effort.
months/year
weeks/months
days/months
Time
GENERAL DATASET PROCESSING LIFE CYCLE
ACTION
• Phone, email, paperwork, $, meetings,
procurement arrangements.
(Days to months)
1
Contact SRC Agency to Acquire Data and Metadata
• (Minutes to months)
2
Wait for SRC Agency Data/Metadata Preparation
• Auto retrieval, FTP, online requests,
etc. (minutes to days)
3
Acquire Data and Metadata OR Create “Best” Metadata
• Verify AOI, TOI, Format; spot check
values, visual inspection or limited
processing (Minutes to days)
4 Examine Data & Metadata for Consistency and Completeness
• Format change, new parameters
manual “cleanup”, etc.
(Days to months)
• Purchase hardware, configuration
management, etc.
(Minutes to days)
• Put into common format, run data checkers
“check gross limits” (Days to months)
• HQC for SFC, Gross limits for precip,
Composite level and value checkers, limited
statistics (% G, B, D by network, stn,
parameter for SFC). (days to months)
5
Develop New Software or Modify Existing Software
6
Acquire Computer System Space and Time
7
Process Data and Metadata then Check
Incomplete?
8 Form Composite with Like Datasets
9
Incomplete?
Add stn info to final
list
Perform QC and Final Data Checks. Generate Statistics
Prepare ASCii format data & docs for CD-Roms
Create EBUFR format data & DOCS for CODIAC
Prepare, manufacture, & distribute CD-Roms
Load Data and Metadata onto CODIAC
Incomplete?
Processing
Problems?
z:\cully\Esop96\esop3.ppt
Hourly Surface Meteorological Data
Composite (743 stations)
CoAgMet (*) (23)
AWOS (*) (110)
RAWS (+) (10)
SCAN (o) (6)
GWMD (*) (18)
Ameriflux (+) (3)
HPCN (o) (72)
Texas ET (o) (19)
KVII-TV (+) (4)
MADIS (o) (246)
NMSU (*) (7)
5-min sites (o) (225)
Composite Data Sets To Be Developed
by UCAR/JOSS for IHOP-2002
• Surface Meteorological - 1 min, 5 min,
and Hourly
• Precipitation - 15 min, Hourly, and
Daily
• Soundings - High Resolution and
5 hPa
1-min Surface Meteorological Data
Composite (65 stations)
ASOS (41)
ARM SMOS (14)
ABLE AWS (5)
NCAR Supp. (5)
5-min Surface Meteorological Data
Composite (224 stations)
Oklahoma Mesonet (115)
NCAR ISS/ISFF (10)
West Texas Mesonet (34)
1-min sites (65)
15-minute Precipitation Composite
NCDC Coop (429)
5-min surface (224)
Hourly Precipitation Composite
NCDC Coop (509)
NCEP hourly (1155)
Hourly Met (743)
Daily Precipitation Composite
NCDC Coop (1581)
Hourly Met (743)
NCEP Daily (1611)
IHOP 2002 DATA BASE STATUS
(as of 30 September 2003)
• 215 of 254 Data sets (85%) have been submitted or
linked. 204 are available on-line (11 in progress)
• Field Catalog (w/reports and preliminary products)
is on-line
• Data submission instructions and guidelines are
available on-line
• Upper Air Composites (hi-res/5 hpa) are available
on-line
• Surface Composites to be available Fall 2003
(TRICK or TREAT!)
• For details see: http://www.joss.ucar.edu/ihop/dm/