Transcript BUFR software Milan Dragosavac Slide 1 WMO BUFR training course Langen 17-20 April 2007 Slide 1

BUFR software
Milan Dragosavac
Slide 1
WMO BUFR training course
Langen
17-20 April 2007
Slide 1
ECMWF Data Formats
 Standard WMO data formats ASCI and binary
 FM 94-XII Ext. BUFR
 FM 92-XII Ext. GRIB
 BUFR for observations and verification scores
 GRIB for fields
 Archiving formats BUFR and GRIB only
Slide 2
WMO BUFR training course
Langen
17-20 April 2007
Slide 2
Requirements for BUFR software
 Suitable for all observations data handling
 Building blocks for other applications
 Encoding
 Decoding
 Low maintenance costs
 Portable
 User friendly
Slide 3
WMO BUFR training course
Langen
17-20 April 2007
Slide 3
Programming languages used
 Fortran 77
- All BUFR subroutines and functions
C
- Reading/writing BUFR data
- gbyte, sbyte routines to do bit manipulations
Slide 4
WMO BUFR training course
Langen
17-20 April 2007
Slide 4
Ecmwf BUFR software
 How to get the software?
The software can be downloaded from Ecmwf web site
http://www.ecmwf.int/products/data/software/
 Go to Free download and get bufr_000320.tar.gz file
 gunzip bufr_000320.tar.gz
 cd bufr_000320
 Read README file for explanations
 Execute make command which will create libbufr.a
 cd examples
Slide 5
 Running make will create number of executables
WMO BUFR training course
Langen
17-20 April 2007
Slide 5
Ecmwf BUFR software
 Content of bufr_000320 directory
-rwxr--r--
1 maa ma
462 Jul 19 2006 Makefile
-rwxr--r--
1 maa ma
467 Jul 19 2006 Makefile.in
-rwxr--r--
1 maa ma 11201 Jul 19 2006 README
drwxr-xr-x 2 maa ma 4096 Jan 22 17:13 bufr_main_c_program
drwxr-xr-x 2 maa ma 4096 Feb 14 08:35 bufrdc
drwxr-xr-x 2 maa ma 8192 Feb 14 08:35 bufrtables
-rwxr--r--
1 maa ma 5327 Jul 19 2006 build_library
drwxr-xr-x 2 maa ma 4096 Feb 7 07:52 config
Slide 6
WMO BUFR training course
Langen
17-20 April 2007
Slide 6
Ecmwf BUFR software
drwxr-xr-x 2 maa ma 4096 Jan 5 11:42 data
drwxr-xr-x 2 maa ma 4096 Jul 19 2006 doc
drwxr-xr-x 2 maa ma 4096 Feb 14 08:28 examples
-rwxr--r--
1 maa ma
1330 Jul 19 2006 install
-rwxr--r--
1 maa ma
1196 Jul 19 2006 licence
drwxr-xr-x 2 maa ma 4096 Jan 22 16:35 options
drwxr-xr-x 2 maa ma 4096 Feb 14 08:35 pbio
Slide 7
WMO BUFR training course
Langen
17-20 April 2007
Slide 7
Ecmwf BUFR software
 config directory contains various configuration files
for different platforms and compilers
 options directory contains various options for different
platforms
 You can make additional configurations and options if
needed
Slide 8
WMO BUFR training course
Langen
17-20 April 2007
Slide 8
Ecmwf BUFR software
 pbio
pbio directory contain all c and fortran routines needed
for BUFR and CREX IO
all routines are part of bufr and crex libraries
 bufrdc
bufrdc directory contain all bufr software source code
routines which are written in fortran 77
Slide 9
WMO BUFR training course
Langen
17-20 April 2007
Slide 9
Ecmwf BUFR software
 examples
decode_bufr.F
tdexp.F
bufr2crex.F
create_bufr.F
 doc
doc directory contain bufr software User’s Guide and
Reference Manual in pdf format
Slide 10
WMO BUFR training course
Langen
17-20 April 2007
Slide 10
Bufr format
Binary universal form for the representation of
meteorological data
 Indicator section
 Identification section
 Optional section
 Data description section
 Data section
 End section
Slide 11
WMO BUFR training course
Langen
17-20 April 2007
Slide 11
Bufr indicator section
Octet number Content
1-4
BUFR four letters in CCITT IA5
5-7
Total length of Bufr message in bytes
8
Bufr edition number ( currently 4 )
Slide 12
WMO BUFR training course
Langen
17-20 April 2007
Slide 12
Identification section
Octet number Content
1-3
4
Length of section 1
Bufr master table (zero if WMO FM 94-XII used )
5-6
Originating / generating centre
7-8
Originating / generating sub-centre
Slide 13
WMO BUFR training course
Langen
17-20 April 2007
Slide 13
Identification section continued
Octet number Content
9
Update sequence number (zero for original BUFR
message; incremented by one for updates)
10
Bit 1 = 0 no optional section
Bit 1 = 1 optional section follows
Bit 2-8 set to zero ( reserved )
11
Data category ( Bufr table A )
12
International data sub-category (common Tab 13)
13
Local sub-category
Slide 14
WMO BUFR training course
Langen
17-20 April 2007
Slide 14
Identification section continued
Octet number
Content
14
Version number of master table used
15
Version number of local tables used ( set to 0 or
255 if no local tables used
16 - 17
Year ( 4 digits )
18
Month
19
Day
20
Hour
21
Minute
22
Second
23-
Slide 15
Reserved for local use by ADP centres
WMO BUFR training course
Langen
17-20 April 2007
Slide 15
Optional section
Octet number Content
1-3
Length of section in bytes
4
Set to zero ( reserved )
5-
Reserved for local used by ADP centres
Slide 16
WMO BUFR training course
Langen
17-20 April 2007
Slide 16
Data description section
Octet number Content
1-3
4
5-6
7
Length of section in bytes
Set to zero ( reserved )
Number of data subsets
Bit 1 = 1 observed data
Bit 1 = 0 other data
Bit 2 = 1 compressed data
Bit 2 = 0 non compressed data
Bits 3-8 set to zero ( reserved )
8
17
Collection of descriptors Slide
defining
the data
WMO BUFR training course
Langen
17-20 April 2007
Slide 17
Data section
Octet number Content
1-3
Length of section in bytes
4
Set to zero ( reserved )
5-
Binary data as defined by sequence descriptors
Slide 18
WMO BUFR training course
Langen
17-20 April 2007
Slide 18
End section
Octet number Content
1-4
“7777” ( coded according to CCITT IA5 )
Slide 19
WMO BUFR training course
Langen
17-20 April 2007
Slide 19
Ecmwf Bufr software
 Bufr is table driven code form
- no need to transfer element names, units …
- no need to change bufr software for new observations
 Bufr table B – classification elements
- reference number
- element name
- element unit
- scale
- reference value
- data width in bits
WMO BUFR training course
Langen
Slide 20
17-20 April 2007
Slide 20
Ecmwf Bufr software
 Bufr table D – list of common sequences to describe the
data
- table B elements
- operators
- other table D descriptors
 Bufr table C – text and meaning of code/flag tables
Slide 21
WMO BUFR training course
Langen
17-20 April 2007
Slide 21
Ecmwf Bufr software
 Bufr table naming convention
Bssswwwwwxxxxxyyyzzz.TXT
Cssswwwwwxxxxxyyyzzz.TXT
Dssswwwwwxxxxxyyyzzz.TXT
sss
- Master table number ( zero for WMO
meteorological tables
wwwww - Originating sub-centre
xxxxx
- Originating centre
yyy
- Version number of master tables used
Slide 22
zzz
- Version number of local tables used
WMO BUFR training course
Langen
17-20 April 2007
Slide 22
Ecmwf Bufr software
 PATH for Bufr tables
export BUFR_TABLES=/xxx/
setenv BUFR_TABLES /xxx/
The path must end with back slash “/”
 Defaults
Integer missing value indicator
NVIND=2147483647
Real missing value indicator
Slide 23
RVIND= 1.7E38
WMO BUFR training course
Langen
17-20 April 2007
Slide 23
Decoding Bufr data
There are four steps in bufr data processing
 Open bufr input file
 Read in one bufr message at a time
 Expand bufr message
 Use the data
Repeat last three steps until the end of file is reached
Slide 24
WMO BUFR training course
Langen
17-20 April 2007
Slide 24
PBIO routines
FORTRAN routines to handle products in an unblocked binary file
subroutine PBOPEN ( kunit, filename, mode, kerr )
Input arguments:
filename - character variable
mode
- character variable ‘r’ read, ‘w’ write, ‘a’ append, ‘r+’ read and
write
Output arguments:
kunit - integer unit number
kerr - integer status return code
Slide 25
WMO BUFR training course
Langen
17-20 April 2007
Slide 25
PBIO routines
kerr - 0, successful
-1, file does not exist
-2, invalid file name
-3, invalid open mode specified
Slide 26
WMO BUFR training course
Langen
17-20 April 2007
Slide 26
PBIO routines
To read Bufr product
subroutine PBBUFR (kunit, karray, kinlen, koutlen, kret)
Input parameters are integers:
kunit
- unit number obtained from pbopen
karray - array big enough to hold the product
kinlen - size in bytes of the array
Output parameters are integers:
koutlen - actual size in bytes of the product read into karray
kret
- status return code
WMO BUFR training course
Langen
17-20 April 2007
Slide 27
Slide 27
PBIO routines
kret from pbbufr subroutine:
0
Product has been successfully read
-1
End-of-file is hit before a product is read.
(Note that EOF does not cause a program fail.
This value must be explicitly caught by the
caller to avoid looping at EOF)
-2
There has been an error in the file-handling
(e.g. the file contains a truncated product)
-3
The size of karray is not sufficient for the
Slide 28
product
WMO BUFR training course
Langen
17-20 April 2007
Slide 28
PBIO routines
subroutine PBCLOSE (kunit, kret)
Input argument is an integer:
kunit - unit number from PBOPEN
Output argument is an integer:
kret – status return code
0 - OK
-1 - error in handling the file
Slide 29
WMO BUFR training course
Langen
17-20 April 2007
Slide 29
PBIO routines
File handling FORTRAN subroutines
subroutine PBSEEK (kunit, koffset, kstart, kret)
Input arguments are integers:
kunit - unit number from PBOPEN
kstart - 0, if koffset is an absolute count from the
beginning of the file
1, if koffset is a relative offset from the current
byte position in the file
2, if koffset is an absolute offset from the end of
Slide 30
file
WMO BUFR training course
Langen
17-20 April 2007
Slide 30
PBIO routines
Output argument is an integer:
kret - status return code
>= 0 Byte offset from the start of file after positioning
-1 End-of-file is hit before the product is read
-2 Error in handling the file
Slide 31
WMO BUFR training course
Langen
17-20 April 2007
Slide 31
PBIO routines
subroutine PBREAD (kunit, karray, kount, kret)
Input arguments are integers:
kunit - unit number from PBOPEN
kount - number of bytes to read from the file
Output arguments:
karray - an integer array to accept the bytes from the
read
kret – status return code
>=0 number of bytes read
Slide 32
-1
EOF reached before the product is read
-2
Error in file handling
WMO BUFR training course
Langen
17-20 April 2007
Slide 32
PBIO routines
subroutine PBWRITE( kunit, karray, kount, kerr)
Input arguments are integers:
kunit – unit number from PBOPEN
karray- an integer array holding bytes for write
kount – number of bytes to write to the file
Output arguments:
kret – status return code
>= 0 number of bytes written to theSlide
file
33
-1
Error in writing to the file
WMO BUFR training course
Langen
17-20 April 2007
Slide 33
Bufr software FORTRAN routines
 Bufr expansion
subroutine BUFREX( kbufl, kbuff, ksup, ksec1, ksec2,
ksec3, ksec4, kelem, cnames,
cunits, kvals, values, cvals, kerr)
Input arguments:
kbufl
- length of bufr message in words
kbuff
- an integer array containing bufr message
kelem - expected number of expanded elements
kvals - the size of values array in Slide
words
34
WMO BUFR training course
Langen
17-20 April 2007
Slide 34
Bufr software FORTRAN routines
Output arguments:
ksup
- an array containing supplementary information
ksec0
- an array containing section 0 information
ksec1
- an array containing section 1 information
ksec2
- an array containing section 2 information
ksec3
- an array containing section 3 information
ksec4
- an array containing section 4 information
cnames - character array containing element names
cunits
- character array containing element units
values - real*8 array containing element
values
Slide 35
kerr
- status return code
WMO BUFR training course
Langen
17-20 April 2007
Slide 35
Bufr software FORTRAN routines
ksec0: an integer array ( size of 3 )
Array
index
Content
1
Length of section 0 in bytes
2
Total length of bufr message in bytes
3
Bufr edition number ( currently 4 )
Slide 36
WMO BUFR training course
Langen
17-20 April 2007
Slide 36
Bufr software FORTRAN routines
ksec1: an integer array ( size of 40 )
Array
index
Content
1
Length of section 1 in bytes
2
Bufr edition number ( currently 4 )
3
Origination centre
4
Update sequence number
5
Flag ( presence of section 2 in the message )
6
Bufr message type ( Bufr table A )
Slide 37
WMO BUFR training course
Langen
17-20 April 2007
Slide 37
Bufr software FORTRAN routines
ksec1 continued
Array
index
Content
7
Bufr message sub-type ( local use)
8
Version number of local tables used
9
Year
10
Month
11
Day
12
Hour
13
Minute
WMO BUFR training course
Slide 38
Langen
17-20 April 2007
Slide 38
Bufr software FORTRAN routines
ksec1 continued
Array
index
Content
14
Bufr master tables used
15
Version number of master table used
16
Originating sub-centre
17
International sub-category
18
Second
19-
Local ADP centre information (byte by byte)
Slide 39
WMO BUFR training course
Langen
17-20 April 2007
Slide 39
Bufr software FORTRAN routines
ksec2
Array
index
Content
1
Length of section 2 in bytes
2-
RDB key in packed form
Slide 40
WMO BUFR training course
Langen
17-20 April 2007
Slide 40
Bufr software FORTRAN routines
ksec3
Array
index
Content
1
Length of section 3 in bytes
2
Reserved
3
Number of subsets
4
Flag ( data type, compression)
Slide 41
WMO BUFR training course
Langen
17-20 April 2007
Slide 41
Bufr software FORTRAN routines
ksec4
Array
index
Content
1
Length of section 4 in bytes
2-
Reserved
Slide 42
WMO BUFR training course
Langen
17-20 April 2007
Slide 42
Bufr software FORTRAN routines
ksup – an integer array (size 9) containing supplementary
information
Array index
Content
1
Dimension ksec1 array
2
Dimension ksec2 array
3
Dimension ksec3 array
4
Dimension ksec4 array
5
Total number of expanded elements
6
Number of subsets
7
Total number of elements in cvals array
8
43
Total bufr message length in Slide
bytes
9
Dimension of ksec0 array
WMO BUFR training course
Langen
17-20 April 2007
Slide 43
Bufr software FORTRAN routines
 Values array is of type real*8
 If bufr table B element is CCITTIA5, corresponding
element contain a real number, when truncated to an
integer represents
index*1000+length
where
index – subscript of the element in CVALS array where
character strings are stored
length – number of characters in the string
Slide 44
WMO BUFR training course
Langen
17-20 April 2007
Slide 44
Bufr software FORTRAN routines
In the case of multi-subset data, one dimensional array
VALUES contains all subsets. Index to values array of
i-th element of observation is:
index=i+(nsub-1)*kelem
Slide 45
WMO BUFR training course
Langen
17-20 April 2007
Slide 45
BUFR SOFTWARE fortran ROUTINES
 Bufr encoding
subroutine BUFREN (ksec0, ksec1, ksec2, ksec3, ksec4,
ktdlen, ktdlst, kdlen, kdata, kelem,
kvals, values, cvals, kbufl, kerr)
Input arguments:
ksec0
- an integer array containing section 0 information
ksec1
- an integer array containing section 1 information
ksec2
- an integer array of 4096 words containing section
2 information
Slide 46
ksec3
- an integer array containing section 3 header
information
WMO BUFR training course
Langen
17-20 April 2007
Slide 46
Bufr software FORTRAN routines
ksec4
- an integer array containing section4 header
information
ktdlen - an integer, number of data descriptors to be
packed in section 3 of bufr message
ktdlst
- an integer array containing ktdlen descriptors
kdlen
- an integer, dimension of kdata array
kdata
- an integer array containing delayed replication
values and possibly new reference value if
required
kelem - an integer containing expected number of
expanded elements
kvals
Slide 47
- an integer containing expected number of data
values
WMO BUFR training course
Langen
17-20 April 2007
Slide 47
Bufr software FORTRAN routines
values - real*8 array of kvals words containing element
values
cvals
- character*80 array of kvals words containing
CCITTIA5 elements
Output arguments:
kbufl - an integer containing the length of bufr message
in words
kbuff - an integer array containing packed bufr message
kerr
- integer, status return code
Slide 48
WMO BUFR training course
Langen
17-20 April 2007
Slide 48
Bufr software FORTRAN routines
 Printing routines
Subroutine BUPRS0 (ksec0)
Subroutine BUPRS1 (ksec1)
Section 2 is for local use – no general printing routines
Subroutine buukey (ksec1,ksec2,key,ksup,kerr)
Subroutine BUPRS2 (ksup, key)
Slide 49
WMO BUFR training course
Langen
17-20 April 2007
Slide 49
Bufr software FORTRAN routines
 To print section3
Subroutine BUSEL (ktdlen, ktdlst, ktdexl, ktdexp, kerr)
Or
Subroutine BUSEl2(ksubset, kelem, ktdlen, ktdlst, ktdexl,
ktdexp, cnames, cunits, kerr)
Subroutine BUPRS3 (ksec3, ktdlen, ktdlst, ktdexl, ktdexp,
kelem, cnames)
Slide 50
WMO BUFR training course
Langen
17-20 April 2007
Slide 50
Bufr software FORTRAN routines
 To print data
Subroutine BUPRT (k, ksub1,ksub2,kelem,cnames,cunits,
cvals, kvals, values, ksup, ksec1, kerr)
 Useful subroutines
Subroutine BUS012 (kbufl, kbuff, ksup, ksec0, ksec1,
ksec2, kerr)
Subroutine BUS0123 (kbufl, kbuff, ksup, ksec0, ksec1,
Slide 51
ksec2, ksec3, kerr)
WMO BUFR training course
Langen
17-20 April 2007
Slide 51
Bufr software FORTRAN routines
 Bufr template design
Basic idea is to know one to one correspondence
between data descriptors expanded and the data values
To achieve that use:
subroutine BUXDES ( k, ksec1, ktdlen, ktdlst, kdlen,
kdata, kelem, ktdexl, ktdexp,
cnames, cunits, kerr)
Input arguments:
k
- an integer, 0 no print; 1 – print Slide 52
WMO BUFR training course
Langen
17-20 April 2007
Slide 52
Bufr software FORTRAN routines
ksec1 - an integer array containing section 1 information.
Set the following words
ksec1(2) – Bufr edition number
ksec1(3) – Originating centre
ksec1(8) – Version number of local tables used
ksec1(15) – version number of master table used
ktdlen – number of data descriptors
ktdlst- list of data descriptors to be packed in section 3
kdlen – dimension of kdata array
kdata - an integer array containing delayed
replications
Slide 53
kelem – expected number of expanded elements
WMO BUFR training course
Langen
17-20 April 2007
Slide 53
|Bufr software FORTRAN routines
Output arguments:
ktdexl – number of expanded elements
ktdexp – integer array containing list of expanded
elements
cnames – character *64 array containing element names
cunits – character*24 array containing element units
kerr – status return code
Slide 54
WMO BUFR training course
Langen
17-20 April 2007
Slide 54
Bufr software FORTRAN routines
 Program to use for template design
…/bufr_000320/examples/tdexp.f
or
Bufrviewer application
Slide 55
WMO BUFR training course
Langen
17-20 April 2007
Slide 55
Bufr software FORTRAN routines
 Resolving bit maps
- Bit maps are very useful bufr features which associate
quality control, first order statistics, departures,
retained/replaced and substituted values with original
values
- One application might have many bitmaps referring
backward to the data
Slide 56
WMO BUFR training course
Langen
17-20 April 2007
Slide 56
subroutine BUBOX (ksub, ksup, kelem, kwtr, cnames,
cunits, kvals, values, kbox, kapp, klen,
kboxr, vals, cboxn, cboxu, kerr)
ksub – subset number
ksup – array containing supplementary information
kelem – expected number of elements, it must be the same
as used in call to bufrex
kwtr – an integer array containing list of bufr table B
reference numbers ( ktdexp, output from busel
routine)
Slide 57
WMO BUFR training course
Langen
17-20 April 2007
Slide 57
Bufr software FORTRAN routines
cnames – array containing elements names
cunits - array containing element units
kvals – expected number of data values
values – array containing expanded values
Output arguments:
kbox – number of elements in the first column of box
kapp - number of applications
klen - max index for number of rows. The next column
starts at KLEN +1 element or
Slide 58
index=i +(KAPP-1)*klen
WMO BUFR training course
Langen
17-20 April 2007
Slide 58
Bufr software FORTRAN routines
kboxr – array containing bufr table B reference numbers
vals – a real*8 array containing boxed values
cboxn – character*64 array, boxed element names
cboxu – character*24 array, boxed element units
kerr – status return code
Slide 59
WMO BUFR training course
Langen
17-20 April 2007
Slide 59
Bufr software FORTRAN routines
 subroutine buprtbox(kbox, kapp, klen, kboxr, vals,
cboxn, cboxu)
Input arguments:
kbox
- an integer containing number of elements in
the first column of the box
kapp
- an integer containing number of applications
klen
- an integer containing max index for number of
rows. The next column starts at klen+1
element or index =i+(kapp-1)*klen
Slide 60
WMO BUFR training course
Langen
17-20 April 2007
Slide 60
Bufr software FORTRAN routines
kboxr
- an integer array containing bufr table B
reference numbers
vals
- a real*8 array containing boxed values
cboxn
- a character*64 array containing boxed names
cboxu
- a character*24 array containing boxed units
Slide 61
WMO BUFR training course
Langen
17-20 April 2007
Slide 61
Bufr software FORTRAN routines
 Example output
1 OPERATOR
************** 222000.0
2 GENERATING CENTRE( CODE TABLE 00
**************
98.0
3 GENERATING APPLICATION (CODE TAB
**************
1.0
4 STATISTICS (008024/008023)
**************
********
5 INCREMENTAL UPDATE NUMBER
**************
********
6 MINIMISATION SIMULATION NUMBER
**************
********
10.0
70.0
320.0
70.0
7 WMO BLOCK NUMBER
8 WMO STATION NUMBER
WMO BUFR training course
Langen
Slide 62
17-20 April 2007
Slide 62
Some Bufr operators
 Quality control information in Bufr
- class 33 element
- associated field
- Operator 222000 + bit map
- 033220 variation analysis report event , valid for whole
report( observation)
Slide 63
WMO BUFR training course
Langen
17-20 April 2007
Slide 63
Some Bufr operators
 Associated field
204001
031021
011001
204000
After expansion
031021 associate field significance
999999 associated field
Slide 64
011001 wind direction
WMO BUFR training course
Langen
17-20 April 2007
Slide 64
Some Bufr operators
 Quality control information using 222000 operator
222000
101005
031031
001031
001032
101005
033007
Slide 65
WMO BUFR training course
Langen
17-20 April 2007
Slide 65
Some Bufr operators
 First order statistics
224000
237000 Use previously define bit map
001031
001032
008023 First order statistics
101005
224255 First order statistics marker
Slide 66
WMO BUFR training course
Langen
17-20 April 2007
Slide 66
Some Bufr operators
 Difference statistical values
225000
237000 Use previously define bit map
001031
001032
008024 Difference statistics
101005
225255 Difference statistics marker
Slide 67
WMO BUFR training course
Langen
17-20 April 2007
Slide 67
Bufr software and pre-processing
 What happens when bufr data reach DPC
 How to organize observation
 Pre-process observations
- unpacking
- reorganization of data
- possible quality control
- grouping of observations which belong to
particular analysis cycle
- creating RDB key for further more efficient processing
- packing the data into bufr
Slide 68
- writing data into RDB
WMO BUFR training course
Langen
17-20 April 2007
Slide 68
Bufr software and pre-processing
 Alternatively
- unpack bufr message
- write unpacked values into DB
- make decisions about archiving
Slide 69
WMO BUFR training course
Langen
17-20 April 2007
Slide 69