www.cabnr.unr.edu

Download Report

Transcript www.cabnr.unr.edu

Error and Uncertainty in Modeling

George H. Leavesley, Research Hydrologist, USGS, Denver, CO

Sources of Error and Uncertainty

• Model Structure • Parameters • Data • Forecasts of future conditions

Conceptualization of Reality

Sacramento

Effective Parameters and States

input real world heterog.

output identical input

measurement

homog.

(x eff ,



eff )

output model Identical ?

After Grayson and Blöschl, 2000, Cambridge Univ. Press

Precipitation Measurement

Sparse precip gauge distribution Mountain blockage of radar SAHRA – NSF STC

Source: Maddox, et. al. Weather and Forecasting, 2002.

Streamflow Measurement Accuracy (USGS)

• Excellent – 95% of daily discharges are within 5% of true value • Good – 95% of daily discharges are within 10% of true value • Fair – 95% of daily discharges are within 15% of true value • Poor – Do not meet Fair criteria Different accuracies may be attributed to different parts of a given record

Dimensions of Model Evaluation

From Wagener 2003, Hydrological Processes

Dimensions of Model Evaluation

From Wagener 2003, Hydrological Processes

Performance Measures

Mean (observed vs simulated) Standard deviation (observed vs simulated) Root Mean Square Error (RMSE) Mean Absolute Error (MAE)

Performance Measures

Coefficient of Determination R 2 Not a good measure - High correlations can be achieved for mediocre or poor models

Performance Measures

Coefficient of Efficiency E Widely used in hydrology Range – infinity to +1.0 Overly sensitive to extreme values Nash and Sutcliffe, 1970, J. of Hydrology

Seasonal Variability of Nash-Sutcliffe Efficiency

Seasonal Analysis of Daily Nash-Sucliffe Efficiency 01608500

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0 -0.1

0 2 4 6

Month

8 10 12 14 SAC GR4J PRMS

Performance Measures

Index of Agreement d Range 0.0 – 1.0 Overly sensitive to extreme values Willmott, 1981, Physical Geography

Analysis of Residuals

Performance Measure Issues

• Different measures have different sensitivities for different parts of the data.

• Are assumptions correct regarding the nature of the error structures (i.e. zero mean, constant variance, normality, independence, …)?

• Difficulty in defining what constitutes an acceptable level of performance for different types of data.

Dimensions of Model Evaluation

From Wagener 2003, Hydrological Processes

Definitions

•

Uncertainty analysis:

investigation of the effects of lack of knowledge or potential errors on model components and output.

•

Sensitivity analysis:

the computation of the effect of changes in input values or assumptions on model output.

EPA, CREM, 2003

Parameter Sensitivity

The single, “best-fit model” assumption

Error Propagation

Magnitude of Parameter Error 5% 10% 20% 50% %> VAR 0.23963 0.95852 3.83408 23.96303

%> SE 0.11974 0.47812 1.89901 11.33868

soil_moist_max 0.15243 0.60973 2.43891 15.24316

%> VAR 0.16210 0.64840 2.59359 16.20993

%> SE 0.08102 0.32367 1.28849 7.80071

hamon_coef 0.10311 0.41245 1.64981 10.31133

%> VAR 0.07889 0.31556 1.26224 7.88900

%> SE 0.03944 0.15766 0.62914 3.86963

ssrcoef_sq 0.05018 0.20073 0.80293 5.01829

joint 0.32477 1.29908 5.19632 32.47698

Objective Function Selection

routing gw rech soil moisture et et baseflow soil moisture gw rech routing

Relative Sensitivity

S R = (  Q PRED / P I ) * (P I / Q PRED )

Relative Sensitivity Analysis Soil available water holding capacity Evapotranspiration coefficient

Relative Sensitivity Analysis Snow/rain threshold temperature Snowfall adjustment

Parameter Sensitivity

The “parameter equifinality” assumption • Consider a population of models • Define the likelihood that they are consistent with the available data

Regional Sensitivity Analysis

• Apply a random sampling procedure to the parameter space to create parameter sets •Classify the resulting model realizations as “behavioural” (acceptable) or “non-behavioural” •Significant difference between the set of “behavioural” and “non-behavioural” parameters identifies the parameter as sensitive Spear and Hornberger, 1980, WRR

Regional Sensitivity Analysis Sensitive Not sensitive

F (



i | B ) F (



i ) F (



i | B )



= behavioural

F (



j | B ) F (



j ) F (



j | B )



= non-behavioural

Generalized Likelihood Uncertainty Analysis (GLUE)

• Monte Carlo generated simulations are classified as behavioural or non-behavioural, and the latter are rejected. • The likelihood measures of the behavioural set are scaled and used to weight the predictions associated with individual behavioural parameter sets. • The modeling uncertainty is then propagated into the simulation results as confidence limits of any required percentile.

Dotty Plots and Identifiability Analysis behavioural

GLUE computed 95%confidence limits

Figure 3 Rockies Animas 300000.0

250000.0

200000.0

150000.0

100000.0

50000.0

0.0 0.2

0.4 0.6

0.8

300000.0

250000.0

200000.0

150000.0

100000.0

50000.0

0.0

5.0

10.0

15.0

300000.0

250000.0

200000.0

150000.0

100000.0

50000.0

25.0

30.0

35.0

40.0

Sierras Carson Cascades Cle Elum 300000.0

250000.0

200000.0

150000.0

300000.0

250000.0

200000.0

150000.0

100000.0

50000.0

0.0 0.2

0.4 0.6

0.8

rad_trncf 50000.0

0.0

0.2 0.4

0.6 0.8

300000.0

250000.0

200000.0

150000.0

100000.0

50000.0

0.0

5.0

10.0

15.0

soil_moist_max (inches) 50000.0

0.0

5.0

10.0

15.0

300000.0

250000.0

200000.0

150000.0

100000.0

50000.0

25.0

30.0

35.0

40.0

tmax_allsnow (deg F) 50000.0

25.0

30.0

35.0

40.0

Parameter Equifinality Regional Variability Uncalibrated Estimate

Multi-criteria Analysis

Increasing the information content of the data A single objective function: • cannot capture the many performance attributes that an experienced hydrologist might look for • uses only a limited part of the total information content of a hydrograph • when used in calibration it will tend to bias model performance to match a particular aspect of the hydrograph A

multi-criteria approach

overcomes these problems (Wheater et al., 1986, Gupta et al., 1998, Boyle et al., 2001, Wagener et al., 2001).

Identifying Characteristic Behavior

Developing Objective Measures

FD  FQ  FS  1 n D  n D  Q obs  Q com  2 1 n Q  n Q  Q obs  Q com  2 1 n s  n s  Q obs  Q com  2

peaks/timing quick recession baseflow

Pareto Optimality

500 Pareto Solutions

u i p z a t p

SAC-SMA Hydrograph Range

Overall Performance Measures

RMSE min BIAS min

Parameter Sensitivity by Objective Function

Dimensions of Model Evaluation

From Wagener 2003, Hydrological Processes

Evaluation of Model Component Processes Observed SCE Final SCE Annual Runoff Solar Radiation PET Percent Groundwater Nash-Sutcliffe Daily Q Day

Integrating Remotely Sensed Data

Coupling SCA remote sensing products with point measures and modeled SWE to evaluate snow component process

East River 1 0.8

0.6

0.4

MODEL SATELLITE 0.2

0 1/0 1/20 2/9 2/29 3/20 4/9 1996 4/29 5/19 6/8 6/28 7/18

Identifiability Analysis

Identification of the model structure and a corresponding parameter set that are most representative of the catchment under investigation, while considering aspects such as modeling objectives and available data.

Wagener

et al

., 2001, Hydrology Earth System Sciences

Dynamic Identifiability Analysis - DYNIA

Information content by parameter

Dynamic Identifiability Analysis - DYNIA

Identifiability measure and 90% confidence limits

A Wavelet Analysis Strategy

• Daily time series Precipitation • Seasonally varying daily variance (row sums) • Seasonally varying variance frequency decomposition (column sums) 7 6 5 4 3 2 1 1 12 11 10 9 8 2 3 4 5 6 7 8 20 10 0 70 60 50 40 30 100 90 80 • Annual average variance frequency decomposition John Schaake, NWS

Variance Decomposition

Precipitation Precipitation 12 11 10 4 3 2 1 1 9 6 5 8 7 2 3 4 5 6 7 8 100 90 80 70 60 50 40 30 20 10 0 8-day Average Precipitation 3 2 5 4 1 1 7 6 9 8 12 11 10 2 3 4 5 6 7 8 20 18 16 14 12 2 0 6 4 10 8 Streamflow OBSQ 12 11 10 9 8 3 2 1 1 5 4 7 6 2 3 4 5 Component OBSQ2 6 6 5 4 3 2 1 1 12 11 10 9 8 7 2 3 4 5 Component 6 7 8 7 8 2 1 4 3 6 5 8 7 10 9 2 1 4 3 6 5 8 7 10 9 Variance Transfer Functions Variance Transfer Function - Observed 3 2 5 4 1 1 7 6 9 8 12 11 10 2 3 4 5 6 7 8 0.5

0.45

0.4

0.35

0.3

0.25

0.2

0.15

0.1

0.05

0 3 2 5 4 1 1 7 6 9 8 12 11 10 8-day Avg Variance Transfer Function - Observed 2 3 4 5 6 7 8 0.5

0.45

0.4

0.35

0.3

0.25

0.2

0.15

0.1

0.05

Streamflow Obs PRMS OBSQ 12 11 10 9 6 5 8 7 4 3 2 1 1 2 3 4 5 Component ESTQ H 6 7 8 12 11 10 4 3 2 1 1 9 8 7 6 5 2 3 4 5 Component 6 7 8 7 6 10 9 8 5 4 1 0 3 2 10 7 6 9 8 2 1 0 5 4 3 12 11 10 9 6 5 8 7 4 3 2 1 1 2 Estimated Streamflow - Linear 6 7 3 4 ESTQ B 5 12 11 10 4 3 2 1 1 9 8 7 6 5 2 3 4 5 Component 6 7 8 8 7 6 10 9 8 5 4 1 0 3 2 10 7 6 9 8 2 1 0 5 4 3 Linear SAC

Variance Transfer Functions Obs PRMS OBSH 12 11 10 5 4 3 2 1 1 7 6 9 8 2 3 4 5 Component ESTH H 6 12 11 10 5 4 3 2 1 1 7 6 9 8 2 3 4 5 Component 6 7 8 7 8 0.5

0.45

0.4

0.35

0.3

0.25

0.2

0.15

0.1

0.05

0 0.5

0.45

0.4

0.35

0.3

0.25

0.2

0.15

0.1

0.05

0 Variance Transfer Function - Linear 12 11 10 5 4 3 2 1 1 7 6 9 8 2 3 4 ESTH B 5 6 12 11 10 5 4 3 2 1 1 7 6 9 8 2 3 4 5 Component 6 7 8 7 8 0.5

0.45

0.4

0.35

0.3

0.25

0.2

0.15

0.1

0.05

0 0.5

0.45

0.4

0.35

0.3

0.25

0.2

0.15

0.1

0.05

0 Linear SAC

Forecast Uncertainty

Ensemble Streamflow Prediction

Using history as an analog for the future

NOAA USGS BOR

Probability of exceedence Simulate to today Predict future using historic data

12000 10000 8000 6000 4000 2000 0

ESP Animas River @ Durango

Only el nino years Observed 2005 10000 9000 8000 7000 6000 5000 4000 3000 2000 1000 0 1981 1982 1983 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005

2005 ESP Forecast

Forecast Period 4/3 – 9/30 Made 4/2/2005 All historic years

ESP - Animas River @ Durango

1982 1983 1987 1991 1992 1993 1994 1995 1997 1998 2002 2003 2004 2005

Ranked Probability Skill Score (RPSS) for each forecast day and month using measured runoff and simulated runoff (Animas River, CO) produced using: (1) SDS output and (2) ESP technique

RPSS

0.1 0.3 0.5 0.7 0.9

Perfect Forecast: RPSS=1 8 6 4 2 0 J F M A M J J A S O N D Month

SDS

8 6 4 2 0 J F M A M J J A S O N D Month

ESP

Given current uncertainty in long-term atmospheric model forecasts, seasonal to annual forecasts may be better with ESP

Summary

• This presentation has been a selected review of uncertainty and error analysis techniques.

• No single approach provides all the information needed to assess a model. The appropriate mix is a function of model structure, problem objectives, data constraints, and spatial and temporal scales of application.

• Still searching for the unified theory of uncertainty analysis.

Necessary Conditions for a Model to be Considered Properly Calibrated

• Input-output behaviour of the model is consistent with measured behaviour performance • Model predictions are accurate (negligible bias) and precise (prediction uncertainty relatively small) • Model structure and behaviour are consistent with the understanding of reality Gupta, H.V., et al, in review

National and international groups are collaborating to assess existing methods and tools for uncertainty analysis and to explore potential avenues for improvement in this area. http://www.es.lancs.ac.uk/hfdg/uncertainty_workshop/uncert_intro.htm

A Federal Interagency Working Group is developing a Calibration, Optimization, and Sensitivity and Uncertainty Analysis Toolbox International Workshop Proceedings describes this effort: available at http://www.iscmem.org

Future Model Development and Application Coupled Hydrological Modelling Systems

Geochemical Flowpaths GIS Landuse Aquatic, Riparian & Terrestrial

INPUTS

Hydrological Modelling

PREDICTIONS

Model Complexity Scale Uncertainty Analysis Increased Model Complexity

More Parameters More Spatial Interactions More Complex Responses but still data limited ….

MORE MODELLING UNCERTAINTY

Visual Uncertainty Analysis Framework

Modular Modelling System - USGS

Deeper Valley Zone

Bedrock Depth (m)

5 4 3 2 1 0 Conceptualised Bedrock

Model Structures Visualisations of Models & Uncertainty Field Measurements Visualisations of Measurements Improved Representations of Hydrological Processors and Predictions

Freer, et al., Lancaster Univ., UK