Transcript www.cabnr.unr.edu
Error and Uncertainty in Modeling
George H. Leavesley, Research Hydrologist, USGS, Denver, CO
Sources of Error and Uncertainty
• Model Structure • Parameters • Data • Forecasts of future conditions
Conceptualization of Reality
Sacramento
Effective Parameters and States
input real world heterog.
output identical input
measurement
homog.
(x eff ,
eff )
output model Identical ?
After Grayson and Blöschl, 2000, Cambridge Univ. Press
Precipitation Measurement
Sparse precip gauge distribution Mountain blockage of radar SAHRA – NSF STC
Source: Maddox, et. al. Weather and Forecasting, 2002.
Streamflow Measurement Accuracy (USGS)
• Excellent – 95% of daily discharges are within 5% of true value • Good – 95% of daily discharges are within 10% of true value • Fair – 95% of daily discharges are within 15% of true value • Poor – Do not meet Fair criteria Different accuracies may be attributed to different parts of a given record
Dimensions of Model Evaluation
From Wagener 2003, Hydrological Processes
Dimensions of Model Evaluation
From Wagener 2003, Hydrological Processes
Performance Measures
Mean (observed vs simulated) Standard deviation (observed vs simulated) Root Mean Square Error (RMSE) Mean Absolute Error (MAE)
Performance Measures
Coefficient of Determination R 2 Not a good measure - High correlations can be achieved for mediocre or poor models
Performance Measures
Coefficient of Efficiency E Widely used in hydrology Range – infinity to +1.0 Overly sensitive to extreme values Nash and Sutcliffe, 1970, J. of Hydrology
Seasonal Variability of Nash-Sutcliffe Efficiency
Seasonal Analysis of Daily Nash-Sucliffe Efficiency 01608500
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0 -0.1
0 2 4 6
Month
8 10 12 14 SAC GR4J PRMS
Performance Measures
Index of Agreement d Range 0.0 – 1.0 Overly sensitive to extreme values Willmott, 1981, Physical Geography
Analysis of Residuals
Performance Measure Issues
• Different measures have different sensitivities for different parts of the data.
• Are assumptions correct regarding the nature of the error structures (i.e. zero mean, constant variance, normality, independence, …)?
• Difficulty in defining what constitutes an acceptable level of performance for different types of data.
Dimensions of Model Evaluation
From Wagener 2003, Hydrological Processes
Definitions
•
Uncertainty analysis:
investigation of the effects of lack of knowledge or potential errors on model components and output.
•
Sensitivity analysis:
the computation of the effect of changes in input values or assumptions on model output.
EPA, CREM, 2003
Parameter Sensitivity
The single, “best-fit model” assumption
Error Propagation
Magnitude of Parameter Error 5% 10% 20% 50% %> VAR 0.23963 0.95852 3.83408 23.96303
%> SE 0.11974 0.47812 1.89901 11.33868
soil_moist_max 0.15243 0.60973 2.43891 15.24316
%> VAR 0.16210 0.64840 2.59359 16.20993
%> SE 0.08102 0.32367 1.28849 7.80071
hamon_coef 0.10311 0.41245 1.64981 10.31133
%> VAR 0.07889 0.31556 1.26224 7.88900
%> SE 0.03944 0.15766 0.62914 3.86963
ssrcoef_sq 0.05018 0.20073 0.80293 5.01829
joint 0.32477 1.29908 5.19632 32.47698
Objective Function Selection
routing gw rech soil moisture et et baseflow soil moisture gw rech routing
Relative Sensitivity
S R = ( Q PRED / P I ) * (P I / Q PRED )
Relative Sensitivity Analysis Soil available water holding capacity Evapotranspiration coefficient
Relative Sensitivity Analysis Snow/rain threshold temperature Snowfall adjustment
Parameter Sensitivity
The “parameter equifinality” assumption • Consider a population of models • Define the likelihood that they are consistent with the available data
Regional Sensitivity Analysis
• Apply a random sampling procedure to the parameter space to create parameter sets •Classify the resulting model realizations as “behavioural” (acceptable) or “non-behavioural” •Significant difference between the set of “behavioural” and “non-behavioural” parameters identifies the parameter as sensitive Spear and Hornberger, 1980, WRR
0
Regional Sensitivity Analysis Sensitive Not sensitive
1
F (
i | B ) F (
i ) F (
i | B )
i
B
= behavioural
1
F (
j | B ) F (
j ) F (
j | B )
0
j
B
= non-behavioural
Generalized Likelihood Uncertainty Analysis (GLUE)
• Monte Carlo generated simulations are classified as behavioural or non-behavioural, and the latter are rejected. • The likelihood measures of the behavioural set are scaled and used to weight the predictions associated with individual behavioural parameter sets. • The modeling uncertainty is then propagated into the simulation results as confidence limits of any required percentile.
Dotty Plots and Identifiability Analysis behavioural
GLUE computed 95%confidence limits
Figure 3 Rockies Animas 300000.0
250000.0
200000.0
150000.0
100000.0
50000.0
0.0 0.2
0.4 0.6
0.8
300000.0
250000.0
200000.0
150000.0
100000.0
50000.0
0.0
5.0
10.0
15.0
300000.0
250000.0
200000.0
150000.0
100000.0
50000.0
25.0
30.0
35.0
40.0
Sierras Carson Cascades Cle Elum 300000.0
250000.0
200000.0
150000.0
300000.0
250000.0
200000.0
150000.0
100000.0
100000.0
50000.0
0.0 0.2
0.4 0.6
0.8
rad_trncf 50000.0
0.0
0.2 0.4
0.6 0.8
300000.0
300000.0
250000.0
250000.0
200000.0
200000.0
150000.0
150000.0
100000.0
100000.0
50000.0
0.0
5.0
10.0
15.0
soil_moist_max (inches) 50000.0
0.0
5.0
10.0
15.0
300000.0
300000.0
250000.0
250000.0
200000.0
200000.0
150000.0
150000.0
100000.0
100000.0
50000.0
25.0
30.0
35.0
40.0
tmax_allsnow (deg F) 50000.0
25.0
30.0
35.0
40.0
Parameter Equifinality Regional Variability Uncalibrated Estimate
Multi-criteria Analysis
Increasing the information content of the data A single objective function: • cannot capture the many performance attributes that an experienced hydrologist might look for • uses only a limited part of the total information content of a hydrograph • when used in calibration it will tend to bias model performance to match a particular aspect of the hydrograph A
multi-criteria approach
overcomes these problems (Wheater et al., 1986, Gupta et al., 1998, Boyle et al., 2001, Wagener et al., 2001).
Identifying Characteristic Behavior
Developing Objective Measures
FD FQ FS 1 n D n D Q obs Q com 2 1 n Q n Q Q obs Q com 2 1 n s n s Q obs Q com 2
peaks/timing quick recession baseflow
Pareto Optimality
500 Pareto Solutions
u i p z a t p
SAC-SMA Hydrograph Range
Overall Performance Measures
RMSE min BIAS min
Parameter Sensitivity by Objective Function
Dimensions of Model Evaluation
From Wagener 2003, Hydrological Processes
Evaluation of Model Component Processes Observed SCE Final SCE Annual Runoff Solar Radiation PET Percent Groundwater Nash-Sutcliffe Daily Q Day
Integrating Remotely Sensed Data
Coupling SCA remote sensing products with point measures and modeled SWE to evaluate snow component process
East River 1 0.8
0.6
0.4
MODEL SATELLITE 0.2
0 1/0 1/20 2/9 2/29 3/20 4/9 1996 4/29 5/19 6/8 6/28 7/18
Identifiability Analysis
Identification of the model structure and a corresponding parameter set that are most representative of the catchment under investigation, while considering aspects such as modeling objectives and available data.
Wagener
et al
., 2001, Hydrology Earth System Sciences
Dynamic Identifiability Analysis - DYNIA
Information content by parameter
Dynamic Identifiability Analysis - DYNIA
Identifiability measure and 90% confidence limits
A Wavelet Analysis Strategy
• Daily time series Precipitation • Seasonally varying daily variance (row sums) • Seasonally varying variance frequency decomposition (column sums) 7 6 5 4 3 2 1 1 12 11 10 9 8 2 3 4 5 6 7 8 20 10 0 70 60 50 40 30 100 90 80 • Annual average variance frequency decomposition John Schaake, NWS
Variance Decomposition
Precipitation Precipitation 12 11 10 4 3 2 1 1 9 6 5 8 7 2 3 4 5 6 7 8 100 90 80 70 60 50 40 30 20 10 0 8-day Average Precipitation 3 2 5 4 1 1 7 6 9 8 12 11 10 2 3 4 5 6 7 8 20 18 16 14 12 2 0 6 4 10 8 Streamflow OBSQ 12 11 10 9 8 3 2 1 1 5 4 7 6 2 3 4 5 Component OBSQ2 6 6 5 4 3 2 1 1 12 11 10 9 8 7 2 3 4 5 Component 6 7 8 7 8 2 1 4 3 6 5 8 7 10 9 2 1 4 3 6 5 8 7 10 9 Variance Transfer Functions Variance Transfer Function - Observed 3 2 5 4 1 1 7 6 9 8 12 11 10 2 3 4 5 6 7 8 0.5
0.45
0.4
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0 3 2 5 4 1 1 7 6 9 8 12 11 10 8-day Avg Variance Transfer Function - Observed 2 3 4 5 6 7 8 0.5
0.45
0.4
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0
Streamflow Obs PRMS OBSQ 12 11 10 9 6 5 8 7 4 3 2 1 1 2 3 4 5 Component ESTQ H 6 7 8 12 11 10 4 3 2 1 1 9 8 7 6 5 2 3 4 5 Component 6 7 8 7 6 10 9 8 5 4 1 0 3 2 10 7 6 9 8 2 1 0 5 4 3 12 11 10 9 6 5 8 7 4 3 2 1 1 2 Estimated Streamflow - Linear 6 7 3 4 ESTQ B 5 12 11 10 4 3 2 1 1 9 8 7 6 5 2 3 4 5 Component 6 7 8 8 7 6 10 9 8 5 4 1 0 3 2 10 7 6 9 8 2 1 0 5 4 3 Linear SAC
Variance Transfer Functions Obs PRMS OBSH 12 11 10 5 4 3 2 1 1 7 6 9 8 2 3 4 5 Component ESTH H 6 12 11 10 5 4 3 2 1 1 7 6 9 8 2 3 4 5 Component 6 7 8 7 8 0.5
0.45
0.4
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0 0.5
0.45
0.4
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0 Variance Transfer Function - Linear 12 11 10 5 4 3 2 1 1 7 6 9 8 2 3 4 ESTH B 5 6 12 11 10 5 4 3 2 1 1 7 6 9 8 2 3 4 5 Component 6 7 8 7 8 0.5
0.45
0.4
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0 0.5
0.45
0.4
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0 Linear SAC
Forecast Uncertainty
Ensemble Streamflow Prediction
Using history as an analog for the future
NOAA USGS BOR
Probability of exceedence Simulate to today Predict future using historic data
12000 10000 8000 6000 4000 2000 0
ESP Animas River @ Durango
Only el nino years Observed 2005 10000 9000 8000 7000 6000 5000 4000 3000 2000 1000 0 1981 1982 1983 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005
2005 ESP Forecast
Forecast Period 4/3 – 9/30 Made 4/2/2005 All historic years
ESP - Animas River @ Durango
1982 1983 1987 1991 1992 1993 1994 1995 1997 1998 2002 2003 2004 2005
Ranked Probability Skill Score (RPSS) for each forecast day and month using measured runoff and simulated runoff (Animas River, CO) produced using: (1) SDS output and (2) ESP technique
RPSS
0.1 0.3 0.5 0.7 0.9
Perfect Forecast: RPSS=1 8 6 4 2 0 J F M A M J J A S O N D Month
SDS
8 6 4 2 0 J F M A M J J A S O N D Month
ESP
Given current uncertainty in long-term atmospheric model forecasts, seasonal to annual forecasts may be better with ESP
Summary
• This presentation has been a selected review of uncertainty and error analysis techniques.
• No single approach provides all the information needed to assess a model. The appropriate mix is a function of model structure, problem objectives, data constraints, and spatial and temporal scales of application.
• Still searching for the unified theory of uncertainty analysis.
Necessary Conditions for a Model to be Considered Properly Calibrated
• Input-output behaviour of the model is consistent with measured behaviour performance • Model predictions are accurate (negligible bias) and precise (prediction uncertainty relatively small) • Model structure and behaviour are consistent with the understanding of reality Gupta, H.V., et al, in review
National and international groups are collaborating to assess existing methods and tools for uncertainty analysis and to explore potential avenues for improvement in this area. http://www.es.lancs.ac.uk/hfdg/uncertainty_workshop/uncert_intro.htm
A Federal Interagency Working Group is developing a Calibration, Optimization, and Sensitivity and Uncertainty Analysis Toolbox International Workshop Proceedings describes this effort: available at http://www.iscmem.org
Future Model Development and Application Coupled Hydrological Modelling Systems
Geochemical Flowpaths GIS Landuse Aquatic, Riparian & Terrestrial
INPUTS
Hydrological Modelling
PREDICTIONS
Model Complexity Scale Uncertainty Analysis Increased Model Complexity
More Parameters More Spatial Interactions More Complex Responses but still data limited ….
MORE MODELLING UNCERTAINTY
Visual Uncertainty Analysis Framework
Modular Modelling System - USGS
Deeper Valley Zone
Bedrock Depth (m)
5 4 3 2 1 0 Conceptualised Bedrock
Model Structures Visualisations of Models & Uncertainty Field Measurements Visualisations of Measurements Improved Representations of Hydrological Processors and Predictions
Freer, et al., Lancaster Univ., UK