Automation of National Observing System

RA VI Seminar on Capacity Building and New technologies in Meteorology: Challenges and Opportunities for the Balkan Countries Sofia, Bulgaria, 11-13 October 2001

Dr. Miroslav Ondráš, Dr. Igor Zahumenský

STRUCTURE (1)

• • • • •

PART I

Why automation?

Limitation and differences Consequences of automation Strategy of automation Associated activities

2

STRUCTURE (2)

• •

PART II

Basic requirements for aws Basic functions of aws 3

PART I

• • • • • Why automation?

Limitation and differences Consequences of automation Strategy of automation Associated activities 4

WHY AUTOMATION ? (1)

• • Continuous increase of demands for regular, timely and on-line data with the increased time resolution is the main driving force for automation and restructuring of National Observing System. Increased time resolution (10 min) becomes a basic requirement to cope with the severe weather forecasting and warnings.

5

WHY AUTOMATION ? (2)

• • AWS may be configured to provide information for synoptic purposes as well as for aviation or climatological or other purposes.

AWS can work in different modes depending on user requirements, automatically switching between modes depending on weather.

6

WHY AUTOMATION ? (3) Advantages (1)

• • • • • • The observation consistency (site-to-site and day to-night); Objective and uniform measurements; High frequency of data provision; Higher accuracy and quality of data; Better timeliness and data availability; More frequent special observations; 7

WHY AUTOMATION ? (4) Advantages (2)

• • • • • Higher density of observations available in real time; Continuous measurement of the atmosphere (each minute up-to-date observations); Data from AWS can be integrated more effectively with the data from other systems; AWS’ data can be more effectively archived; Lesser cost per data piece. 8

LIMITATIONS & DIFFERENCES (1)

• • • • AWS does not provide a horizon-to-horizon evaluation of the weather, only of weather that has passed through the sampling volume of the sensor array (measurements made at a fixed location); Some elements are difficult to automate; Compatibility between AWS and observer outputs (visibility,clouds,present weather,etc) ; AWS requires initial capital investment.

9

LIMITATIONS & DIFFERENCES (2)

• • AWS and observer differ in their methods of sampling and processing the various weather elements: A human observer estimates weather phenomena at a fixed location by integrating in space.

An automatic system estimates weather phenomena at a fixed location by integrating in time.

10

LIMITATIONS & DIFFERENCES (3)

• AWS applies procedures and algorithms to the collected data in order to extrapolate the weather over a wider area.

• AWS provides objective and consistent information while human observations show significant subjectivity and uncertainty.

11

LIMITATIONS & DIFFERENCES (4)

• • • • •

AWS

Fixed location (time-averaged); Representation for • 3-5km of sensor site; Continuous observation; • • • Consistent observation; Report everything detected by sensors.

•

HUMAN

Fixed time (spatial-averaged); Representation horizon to-horizon; Time constraints; Affected by lights, building, human perception; Intelligent filtering.

12

CONSEQUENCES OF AUTOMATION

• • • • •

Automation:

Introduces more technological complexity to the observation process; Influences all phases of data flow transmission-processing-archiving); (measurement Introduces data inhomogeneity (comparing to old data series) ; Influences maintenance system ( replaces observers by technicians for maintenance) ; Requires refreshment courses at all levels.

13

STRATEGY OF AUTOMATION (1)

• • Automation should be seen as a tool to build integrated NOS that fits well in the regional and world-wide composite observing system.

Automation should be done very carefully with respect to present and future limitations and contradictory tendencies (e.g., requirement for conservative approach from Climatology) .

14

STRATEGY OF AUTOMATION (2)

• • Start with the combination of automatic and semi-automatic stations.

Design your AWS to be effective and multipurpose (serving for weather, hydrology and environment monitoring ).

15

STRATEGY OF AUTOMATION (3)

• Implement only those AWSs, which are sufficiently well documented so as to provide adequate knowledge and understanding of their capabilities, characteristics and any procedures and algorithms used.

16

STRATEGY OF AUTOMATION (4)

• • Before full implementation of automation thorough analysis of the functionality and comparison of AWS’ data with manual measurements is required (1 to 2 years of comparison measurements is necessary).

Before full implementation address the differences in instrumentation type, measurement methods, data processing, data control, calibration and maintenance of both types of monitoring networks.

17

ASSOCIATED ACTIVITIES

(1)

Monitoring

• • • Detailed performance monitoring of the functionality of the whole system is a precondition of the successful automated weather monitoring network; It should allow for prompt remedial actions (pulling the data from AWS, filling the gaps, correction of errors); It should go deep enough into the AWS so that long-term drift of sensors can be detected.

18

ASSOCIATED ACTIVITIES

(2)

Calibration

To guarantee data quality and validity there is a need to enhance all levels: • • • Initial calibration Field calibration Laboratory calibration It involves comparison against a known standard to determine how closely instrument output matches the standard over the expected range of operation.

19

ASSOCIATED ACTIVITIES

(3)

MAINTENANCE

• • • Preventive (cleaning); Corrective (AWS component failures); Adaptive (changed requirements or obsolescence of components); • Part of a broader performance monitoring: To ensure rapid response time for periodic transmission of self-checking diagnostic information by the AWS is needed.

20

ASSOCIATED ACTIVITIES

(4)

Documentation

• • • In addition to standard documentation, such as: Documentation of initial siting of the system, sensors (maps, photographs); Ongoing documentation of equipment and siting (metadata) and all changes; Metadata showing changes in the station’s immediate surroundings or sensors;

Documentation of the procedures and algorithms used and all changes to them.

21

ASSOCIATED ACTIVITIES

(5)

Training

• System performance, system reliability and consequently data quality and availability depend on the skills of the staff.

• There is a need for training at all levels of staff not only employed in the observing network divisions.

22

PART II

• • Basic requirements for aws Basic functions of aws 23

BASIC REQUIREMENTS FOR AWS (1)

•

Should be considered from the point of view:

User data requirements (forecasting & warning system, aviation, climatology, etc.).

• • • • Present and future: Observing system design; Data processing system capability; Data management system capability; Telecommunication system capability.

24

BASIC REQUIREMENTS FOR AWS (2) Siting and Exposure (1)

• •

Standard observing site:

On a level piece of ground, covered with short grass or surface representative of the locality. Meteorological sensors should be sited at a distance which is beyond the influence of obstructions such as buildings and trees (distance depends upon the variable as well as the type of obstruction).

25

26

BASIC REQUIREMENTS FOR AWS (3) Siting and Exposure (2)

• •

Standard observing site:

Sensors are should be positioned at the same height (and place) to those of classic instruments, Keep the long term “stability” of exposure of the site (changing of vegetation, buildings, etc.) 27

BASIC REQUIREMENTS FOR AWS (4) Siting and Exposure (3)

• • •

Temperature & Humidity sensor:

Inside a suitable instrument shelter or shield at height of 1.25 to 2.0 m preferred (ventilated or non-ventilated).

Different types and shapes and colors of shields give different results of measurement.

For data comparison and compatibility could be installed in classic Stevenson screen.

28

BASIC REQUIREMENTS FOR AWS (5) Siting and Exposure (4)

• •

Rainfall measurement:

Very open sites which are satisfactory for most instruments are unsuitable for raingauges (need for some degree of shelter or artificial shield).

At 1 m above ground gives different result from measurement made at 3 m or 30 cm height above the ground or inside a pit; 29

BASIC REQUIREMENTS FOR AWS (6) Siting and Exposure (5)

• • •

Wind measurement:

Standard exposure is at a height of 10 m above flat ground in the open terrain (distance from obstructions be a minimum of 10 times the height of obstruction); Wind speed measured at lower height is significantly less than speed measured at 10 m above ground.

Need for another observation point.

30

31

BASIC REQUIREMENTS FOR AWS (7) Sensors (1)

• • • • • •

Care should be taken so that sensors correspond to the user requirements:

Measuring range; Data representation; Data compatibility; Accuracy; Reliability; Long term stability.

32

BASIC REQUIREMENTS FOR AWS (8) Sensors (2)

• •

Measuring range:

Depends mainly on climatological conditions where AWS will be installed: – E.g. different requirements concerning temperature measurement and range should be considered in case of Tropic and High latitude or Polar regions. Depends also on user requirements: – E.g. measurement range of ceilometer CT25K is 0 25.000ft whereas of CT12K is 0-12,500ft only.

33

BASIC REQUIREMENTS FOR AWS (9) Sensors (3)

•

Data representation:

A meteorological observation is intended to be representative of an area in accordance with its application. – For synoptic purposes, it should be representative of a wide area around the station. – For aviation, it should be representative for the areas of runways, landing and take-off.

– For agrometeorological purposes - crop field...

34

BASIC REQUIREMENTS FOR AWS (10) Sensors (4)

•

Data compatibility:

In order to achieve data compatibility when using different types of sensors, shielding and different exposure of sensor for measuring the same variable, corrections to the actual measurements are necessary.

– E.g. in case of measuring precipitation or wind speed in different heights above the ground.

35

BASIC REQUIREMENTS FOR AWS (11) Sensors (5)

• •

Accuracy:

The closeness of the agreement between the result of a measurement and a true value of the measurand. There are different operational accuracy requirements depending on applications as well as different achievable accuracy for individual variables. – E.g. Height of cloud: required accuracy is 10 % for height > 100 m, achievable accuracy (using CT25K) is 50 ft for the whole range of measurement.

36

BASIC REQUIREMENTS FOR AWS (12) Sensors (6)

•

Reliability:

It refers to the reproducibility of a measurement. One can quantify reliability simply by taking several measurements on the same subject. – Poor reliability degrades the precision of a single measurement and reduces your ability to track changes in measurements. – Frequent replacement of unreliable instruments increases significantly the total cost of measurement corresponding variable and decreases quality of measurement.

37

BASIC REQUIREMENTS FOR AWS (13) Sensors (7)

• •

Long term stability:

The ability to keep its known accuracy of measurement over a long period and can be expressed by drift (the stability of the sensor's calibration with time). Good stability means lower calibration costs, saves time and trouble.

38

BASIC REQUIREMENTS FOR AWS (14) Sensor characteristics (1)

• • • • • • Fundamental characteristics of sensors (to ensure accuracy and precision of measurement) are: Resolution; Repeatability; Linearity; Response time; Drift; Hysteresis.

39

BASIC REQUIREMENTS FOR AWS (15) Sensor characteristics (2)

• •

Resolution:

It is the smallest change the device can detect. It is a quantitative expression of the ability of an indicating device to distinguish meaningfully between closely adjacent values of the quantity indicated.

40

BASIC REQUIREMENTS FOR AWS (16) Sensor characteristics (3)

•

Repeatability:

It is the ability of the sensor to measure a variable more than once and produce the same result in identical circumstances.

•

Linearity:

Defines the deviation of the sensor from ideal straight line behavior.

41

BASIC REQUIREMENTS FOR AWS (17) Sensor characteristics (4)

• •

Response time:

Normally defined as the time the sensor takes to measure 63% of the change. The time interval between the instant when a stimulus is subjected to a specified abrupt change and the instant when the response reaches and remains within specified limits around its final steady value.

42

BASIC REQUIREMENTS FOR AWS (18) Sensor characteristics (5)

•

Drift:

It is the stability of the sensor’s calibration with time.

•

Hysteresis:

It is the ability of the sensor to produce the same measurement whether the phenomenon is increasing or decreasing.

43

44

BASIC REQUIREMENTS FOR AWS (19) Sensor characteristics (6)

• • Some of above mentioned characteristics are more important in particular situations than others.

For example: – for monitoring climatic temperature changes a sensor is required which has very little drift, – for measuring wind gusts the repeatability of the device and the response time become more important.

45

BASIC FUNCTIONS OF AWS (1)

• • • • • Data acquisition and (pre)processing; Data check and quality control; Data formats and messages; Data transmission; Data storage.

46

BASIC FUNCTIONS OF AWS (2) Data acquisition and processing (1)

• • • • • • Among others: Sampling of sensor output; Conversion of sensor output; Linearization; Smoothing and Averaging; Corrections; Derived data computation.

47

BASIC FUNCTIONS OF AWS (3) Data acquisition and processing (2)

• •

Sampling

(scanning) of sensor output: A sample – a single measurement, typically one of a series of spot readings of a sensor system (an observation is derived from a number of samples).

Different sampling frequency is used: – – for temperature (5-6 times a minute), for wind gust (every 3 seconds), etc..

48

BASIC FUNCTIONS OF AWS (4) Data acquisition and processing (3)

•

Conversion

of sensor output: It is the transformation of the electrical output values of sensors into meteorological units.

49

BASIC FUNCTIONS OF AWS (5) Data acquisition and processing (4)

•

Linearization:

If the transducer output is not exactly proportional to the quantity being measured, then the signal must be linearized, making use of the instrument’s calibration.

50

BASIC FUNCTIONS OF AWS (6) Data acquisition and processing (5)

• •

Smoothing and averaging: Smoothing

is used to remove noise (random errors and fluctuations not relevant for the application).

Averaging

is used to remove the natural small scale variability of the atmosphere.

They are necessary for obtaining representative observation and compatibility of data from different sensors.

51

BASIC FUNCTIONS OF AWS (7) Data acquisition and processing (6)

•

Corrections:

It allows data to be adjusted to compensate for errors that occurred during the service interval as a result of environmental or instrumentation effects. 52

BASIC FUNCTIONS OF AWS (8) Data acquisition and processing (7)

-

Derived data computation:

Statistic quantities calculations (extremes, totals); Derived data from meteorological parameters (visibility from extinction coefficient, dew point from humidity).

53

BASIC FUNCTIONS OF AWS (9) Data formats and messages (1)

• • • •

Formats

Flexible; for transmission of data should be: Independent of AWS manufactures; Standard; Unambiguous.

54

BASIC FUNCTIONS OF AWS (10) Data formats and messages (2)

• • • • • •

Types of formats:

raw data; 1 min data (engineering format, BUFR); 10 min data (engineering format, BUFR); 1 hour data (engineering format, BUFR); WMO messages (SYNOP, METAR, CLIMAT); pre-processed data in tables & sheets.

55

Formats of messages used in the SHMI operational practice for the transmission of 10 minute (SXSQ42) and hourly (SXSQ41) data from AWOS to the Data processing center

SXSQ42 CCCC YYGGgg (UT) 11iii YYGGgg (local time) 11s n TTT 12s n T x T x T x 13s n T n T n T n 14s n T m T m T m 16s n T g T g T g 17s n T gx T gx T gx 18s n T gn T gn T gn 19s n T gm T gm T gm 21RhRhRh 22Rh x Rh x Rh x 23Rh n Rh n Rh n 24Rh m Rh m Rh m 26PPPPP 27P x P x P x P x P x 28P n P n P n P n P n 29P m P m P m P m P m 30dvdv 31fvfvfv 32dsds 33fsfsfs 34d x d x 35f x f x f x 36GGgg 41s n T 5 T 5 T 5 42s n T 5x T 5x T 5x 43s n T 5n T 5n T 5n 44T 5m T 5m T 5m 46s n T 10 T 10 T 10 47s n T 10x T 10x T 10x 48s n T 10n T 10n T 10n 49T 10m T 10m T 10m 51s n T 20 T 20 T 20 52s n T 20x T 20x T 20x 53s n T 20n T 20n T 20n 54s n T 20m T 20m T 20m 56s n T 50 T 50 T 50 61s n T 100 T 100 T 100 70SSS 71RR 01 RR 02 RR 03 RR 04 RR 05 RR 06 RR 07 RR 08 RR 09 RR 10 (72RR 11 RR 12 RR 13 RR 14 RR 15 RR 16 RR 17 RR 18 RR 19 RR 20 . . .

76RR 51 RR 52 RR 53 RR 54 RR 55 RR 56 RR 57 RR 58 RR 59 RR 60 (72-76 only in SXSQ41) 77T R T R T R 78G l G l G l G l 79G lx G lx G lx G lx 80G ln G ln G ln G ln 81EEE 85GG Tx GG Tn 86GG TGx GG TGn 87GGRh x GGRh n 88GGP x GGP n 89GGT5 x GGT5 n 90GGT 10x GGT 10n 91GGT 20x GGT 20n SXSQ41 has identical structure with some parameters having different meanings.

Explanation of Symbolic letters

General explanation: For the value of T, Rh, P: - T, Rh, P – value in 10 th minute of the period - x – max. value, n – min. value, m – average value in the period (10 minute, 1 hour) Symbolic letters: s n – sign of the temperature (as in SYNOP) TTT- air temperature T g T g T g – surface temp.

R h R h dvdv, R period) h - relative humidity PPPPP – station pressure dsds – wind direction (vector and scalar average of the fvfvfv, fsfsfs – wind speed (vector a scalar average of the period) d x d x – wind direction of the max. gust in the period f T x 5 f x T f x 5 - wind speed of the max. gust in the period GGgg – time of the max. gust T 5 ,T 10 T 10 T 10 , ... – soil temperature SSS – sun duration RR 01 ,RR 02 ,RR 03 . . – amount of precipitation in 1 st , 2 nd ,3 rd , ....

minute of the period T RR T RR T RR – duration of the precipitation in the period G l G l G l G l – global radiation G x G x G x G x – max. value in the period G n G GG n Tx G n G GG n Tn – minimum value EEE – evaporation , GG TGx GG TGn , GGRh x GGRh n , GGP x GGP n , GGT5 x GGT5 n , GGT 10x GGT 10n , GGT 20x GGT 20n – time of max. and min. value in the period 17 56

Engineering Format Example: SXSQ42 LZPE 080640 11867 080636 110163 120167 130163 140165 160153 170157 180153 190155 21080 22080 23077 24079 2609835 2709835 2809835 2909835 3009 31027 3209 33028 3412 35052 360632 410153 420154 430153 440153 460154 470154 480154 490154 510156 520156 530155 540156 560155 70000 7100000100000000010000 77100 780014 790020 800011 852736 862735 873527 882727 893231 902728 913032 923628= Example of a complete CREX message CREX++ T000101 A000 B01015 B01001 B01002 B02001 B04001 B04002 B04003 B04004 B04005 B05001 B06001 B07001 B07006 B10004 B07006 B11001 B11002 B11041 B11043 B11016 B11017 B07006 B12001 B13003 B07006 B12001 R02004 B07061 B12030 ++ Prievidza 11 867 2 2001 09 08 06 40 0000048 00000014 00261 00001 98350 00010 090 0003 //// /// /// /// 00002 016 080 00000 015 00005 015 00010 015 00020 015 00050 015++ 7777 Example of a complete BUFR message containing 52 octets

:

0100001001010101010001100101001000000000000000000011010000000010 00000000000000 0000010010000000000000000000111000000000000000000000000010000000 00000000100000 0001010111010000010000011101000011000000000000000000000000000000 00000000111000 0000000000000000000001100000000000000100000001000000010000001000 00110000000100 00000000 01000000000000000000100000000000100100001111010111011100 01000000 001101 11001101110011011100110111

57

BASIC FUNCTIONS OF AWS (11)

Data Transmission

• • Data can be transmitted in different modes:

Pushing mode:

– At regular time intervals controlled by the AWS time scheduler, – Event driven when certain meteorological thresholds are crossed.

Pulling mode:

– In response to external commands.

58

BASIC FUNCTIONS OF AWS (12) Quality control (1)

•

The purpose: T

o ensure that data meet requirements (resolution, homogeneity, representative, timeliness).

• •

Levels of QC:

Basic (performed at AWS) Extended (performed outside AWS) 59

BASIC FUNCTIONS OF AWS (13) (Basic) Quality control (2)

• • • • Real-time automatic data validity checking applied at the AWS: measurement ranges, temporal consistency of measured values, internal validity & consistency of raw data, calculation of standard deviation of some variables.

60

A. MEASUREMENT RANGES: Limit values for surface temperature The value is considered suspect when MIN 2 = T < MIN 1 or MAX 1 < T = MAX 2; the value is considered erroneous when T < MIN 2 or T > MAX 2 Area Winter Summer 45°S – 45°N 45°N – 90°N and 45°S – 90°S MIN 2 –40°C –90°C MIN 1 –30°C –80°C MAX 1 +50°C +35°C MAX 2 +55°C +40°C MIN 2 –30°C –40°C MIN 1 –20°C –30°C MAX 1 +50°C +40°C MAX 2 +60°C +50°C B. TEMPORAL CONSISTENCY: Suggested tolerances for the temperatures and the tendency as a function of time period between consecutive reports Parameter dt = 1 hour Dt = 2 hours dt = 3 hours dt = 6 hours dt = 12 hours T tol.

Td tol.

Pp tol.

4°C 4°C 3 hPa 7°C 6°C 6 hPa 9°C 8°C 9 hPa 15°C 12°C 18 hPa 25°C 20°C 36 hPa C. CONSISTENCY CHECKS Wind dd/ff The wind information is considered to be erroneous in the following cases: dd = 00 and ff



00; dd



00 and ff = 00; dd = 99 and ff = 00 or ff



05ms -1 ;

61

BASIC FUNCTIONS OF AWS (14)

Data Storage

• • Both processed and manually entered data, including QC status information have to be stored for some time in the AWS.

The extent of database and memory required is to be determined as a function of the maximum possible number of sensors, intermediate data, derived quantities, pre processing, etc.

62

REFERENCES

• • • • Guide to Meteorological Instruments and methods of Observation (WMO – No. 8) Guide on the Global Observing System (WMO – No. 488) Manual on the Global Observing System (WMO – No. 544) Guidance on Automatic Weather Systems and their Implementation (WMO – No. 862) 63