Transcript Ecmwf observations data handling Milan Dragosavac Slide 1 WMO BUFR training course Langen 17-20 April 2007 Slide 1

Ecmwf observations
data handling
Milan Dragosavac
Slide 1
WMO BUFR training course
Langen
17-20 April 2007
Slide 1
Analysis
IFS
Forecast
ODB
Data
transport
Data
transport
Data
acquisition
MDB
RDB
FDB
PDB
MARS
General applications
Slide 2
Print
Plot
Display
WMO BUFR training course
Langen
17-20 April 2007
Slide 2
Data acquisition
 Acquire data
 Process data
 Archive data ( not MARS)
 Outgoing ( send data
outside Ecmwf)
Slide 3
WMO BUFR training course
Langen
17-20 April 2007
Slide 3
Acquisition process
 GTS data
 ftp
 Other data
Slide 4
WMO BUFR training course
Langen
17-20 April 2007
Slide 4
Acquisition outgoing
 Qscat satellite data in BUFR
form
 SSMI brightness
temperatures
 TOVS 1B calibrated and
converted into BUFR format
Slide 5
WMO BUFR training course
Langen
17-20 April 2007
Slide 5
MEB feed
 Restores any old data
 Join segmented bulletins
 Write bulletins in MEB
 statistics
Slide 6
WMO BUFR training course
Langen
17-20 April 2007
Slide 6
Message data base
 “STREAM”_YYYYMMDD
 AIRC_20070216
 AIRS_20070216
 AMAP_20070216
 GRIB_20070216
 75 streams in total
 One stream per day
Slide 7
WMO BUFR training course
Langen
17-20 April 2007
Slide 7
Message data base attributes
 TIME
character( 4,1)
 OBSTYPE
character( 4,1)
 TTAAii
character( 6,1)
 BBB
character( 3,1)
 CCCC
character( 4,1)
 _SOURCE
character(80,1)
 _CHECKSUM
longinteger
 _TIMESTAMP
character(20,1)
Slide 8
WMO BUFR training course
Langen
17-20 April 2007
Slide 8
Message data base attributes continued
 _data_size
longinteger
 _bulk_size
longinteger
 _owner
character( 3,1)
 _last_access
time(0)
 _access_number
longinteger
 _compress
character( 1,1)
 _data
bulk(20,0,512,1)
Slide 9
WMO BUFR training course
Langen
17-20 April 2007
Slide 9
Message data base indices
 UNIQUE I_timestamp_SYNO_20070215 on (_TIMESTAMP)
 UNIQUE I_comp_SYNO_20070215 on (TIME, TTAAii, BBB,
_CHECKSUM)
 NORMAL (2,62) I_OBSTYPE_SYNO_20070215 on
(OBSTYPE)
Slide 10
WMO BUFR training course
Langen
17-20 April 2007
Slide 10
Message data base
 100 Gbytes allocated + 150 Gbytes for METOP data
 Over 22 Gbytes / day
 Saved on ECFS according to OBSTYPE
 All data can be reprocessed if Data base is lost
 HP cluster 4 CPU 64 bit processors
 4 operational machines + 1 for development
 Raid disks 1.2 Tbytes
Slide 11
WMO BUFR training course
Langen
17-20 April 2007
Slide 11
MEB data arrival
 Monitoring
arrival times of
different data
types
 Analysis cycle
 Average number
of days
Slide 12
WMO BUFR training course
Langen
17-20 April 2007
Slide 12
Message data base data availability
 Data availability
according to synoptic
hour
Slide 13
WMO BUFR training course
Langen
17-20 April 2007
Slide 13
MEB x-interface
 Allows MEB
search
 Find
corresponding
BUFR
observation in
REB
 Expand BUFR
and GRIB data
 Write data into
file
Slide 14
WMO BUFR training course
Langen
17-20 April 2007
Slide 14
MEB x-interface
 REB BUFR observations
corresponding to MEB
bulletin
Slide 15
WMO BUFR training course
Langen
17-20 April 2007
Slide 15
Preprocessing suite
 MEB – message data base
feed task
 deqc – decoding, quality
control and BUFR creation,
writing into REB
 ex00 – extraction for
analysis
 bc00 – extraction for
boundary condition suite
 wmo – ftp WMO station list
Slide 16
WMO BUFR training course
Langen
17-20 April 2007
Slide 16
Decoding
 Decoding
 Quality control
 BUFR creation
 Global grib fields creation
 Writing into REB
Slide 17
WMO BUFR training course
Langen
17-20 April 2007
Slide 17
REB extraction for analysis
 14 files created
 Conventional observations
 Satellite data
 Sea surface temperatures
 Ice analysis
Slide 18
WMO BUFR training course
Langen
17-20 April 2007
Slide 18
REB – report data base
 One table per observation type per day
 AIRC_20070215
 ATOV_20070215
 B002_20070215
 CREX_20070215
 TEMP_20070215
 SSBT_20070215
 QSCA_20070215
 statistic
Slide 19
WMO BUFR training course
Langen
17-20 April 2007
Slide 19
REB attributes
 TIME
character(6,1)
 IDENT
character(9,1)
 _BUFR_TYPE
integer
 _BUFR_SUBTYPE
integer
 _LATITUDE1
float(6)
 _LONGITUDE1
float(6)
 _COR1
integer
 _QC
integer
Slide 20
WMO BUFR training course
Langen
17-20 April 2007
Slide 20
REB attributes continued
 _TIMESTAMP
character(20,1)
 NR_SUBSETS
integer
 _data_size
longinteger
 _bulk_size
longinteger
 _owner
character(3,1)
 _last_access
time(0)
 _access_number
longinteger
 _data
bulk(20,0,512,1)
Slide 21
WMO BUFR training course
Langen
17-20 April 2007
Slide 21
REB indices
 UNIQUE I_ts_SYNO_20070215 on (_TIMESTAMP)
 UNIQUE I_comp_SYNO_20070215 on (TIME, IDENT,
_BUFR_SUBTYPE, _LATITUDE1, _LONGITUDE1
 NORMAL (2,15) I_extract_SYNO_20070215 on (TIME,
_BUFR_SUBTYPE, IDENT)
Slide 22
WMO BUFR training course
Langen
17-20 April 2007
Slide 22
REB reports data base
 Just bellow 22 Gbytes / day
 100 Gbytes allocated + 150 Gbytes for METOP data
 All data can be re-decoded at any time
 Data can be accessed through MARS request and
Metview
 Archived in MARS
Slide 23
WMO BUFR training course
Langen
17-20 April 2007
Slide 23
REB x-interface
 Search REB
 Find original bulletin in
MEB
 Expand BUFR and GRIB
data
 Write data to file
 Access WMO station list
data base
Slide 24
WMO BUFR training course
Langen
17-20 April 2007
Slide 24
BUFR x-interface
 Expands BUFR message
 Any subset can be viewed
 Filtering features
Slide 25
WMO BUFR training course
Langen
17-20 April 2007
Slide 25
WMO Data Base
 WMO_a_master
 WMO_a_master_manual
 WMO_a_master_tape
Slide 26
WMO BUFR training course
Langen
17-20 April 2007
Slide 26
WMO_a_master attributes
 Region_country
integer not null
 Block
smallint not null
 Station_index
integer not null
 Station_subindex
smallint not null
 Principal _station
character(1,1)
 Station_name
text(20,20,30,1)
 Latitude
real
 Longitude
real
Slide 27
WMO BUFR training course
Langen
17-20 April 2007
Slide 27
WMO_a_master attributes continued
 Pressure_elevation
integer
 Pressure_elevation_flag
character(1,1)
 Ground_elevation
integer
 Ground_elevation_flag
character(1,1)
 Pressure_level
smallint
 Surf_prog_00
character(2,1)
 Surf_prog_03
character(2,1)
 Surf_prog_06
character(2,1)
 Surf_prog_09
character(2,1)
 Surf_prog_12
WMO BUFR training course
character(2,1)
Slide 28
Langen
17-20 April 2007
Slide 28
WMO_a_master attributes continued
 Surf_prog_15
character(2,1)
 Surf_prog_18
character(2,1)
 Surf_prog_21
character(2,1)
 Ua_type_00
smallint
 Ua_type_06
smallint
 Ua_type_12
smallint
 Ua_type_18
smallint
 Modify_flag
character(1,1) not null
Slide 29
WMO BUFR training course
Langen
17-20 April 2007
Slide 29
WMO_a_master indices
 UNIQUE I_wmo_a_master ON (block,station_index,
station_subindex)
Slide 30
WMO BUFR training course
Langen
17-20 April 2007
Slide 30
WMO station list
Slide 31
WMO BUFR training course
Langen
17-20 April 2007
Slide 31
ODB Observational data base
 Interface to analysis
 Data base contains unpacked bufr data and original bufr
data
 Loading done in parallel with load balance achieved on
supercomputer due to bufr features
 Analysis updates ODB
 Analysis bufr feedback files created
 Archiving of feedback files in MARS
Slide 32
WMO BUFR training course
Langen
17-20 April 2007
Slide 32
Loading ODB with observations
 Tasks running in
parallel
 2-16 processors
used per task
 Load balancing
achieved
Slide 33
WMO BUFR training course
Langen
17-20 April 2007
Slide 33
ODB to BUFR
 odb2bufr family
Slide 34
WMO BUFR training course
Langen
17-20 April 2007
Slide 34
Odbviewer
Slide 35
WMO BUFR training course
Langen
17-20 April 2007
Slide 35
Odbviewer
Slide 36
WMO BUFR training course
Langen
17-20 April 2007
Slide 36
Odbviewer
Slide 37
WMO BUFR training course
Langen
17-20 April 2007
Slide 37
Odbviewer
Slide 38
WMO BUFR training course
Langen
17-20 April 2007
Slide 38
Odbviewer
Slide 39
WMO BUFR training course
Langen
17-20 April 2007
Slide 39
Odbviewer
Slide 40
WMO BUFR training course
Langen
17-20 April 2007
Slide 40
Odbviewer
Slide 41
WMO BUFR training course
Langen
17-20 April 2007
Slide 41
Future
 Sharp increase in volume of observations expected
 IASI data with 8641 channels per observation
 Data bases have crucial role in observations handling
 Binary data representation allows easy handling of
observations as bulk objects
Slide 42
WMO BUFR training course
Langen
17-20 April 2007
Slide 42