Transcript No Slide Title

Model Predictive Uncertainty
Sensitivity analysis …..
GM Seam Inflows
3
Parameter Sets
Set 2
Flow (L/s)
Set 3
2
Set 5
Set 6
1
0
Jan-97 Feb-97 Mar-97 Apr-97 May-97 Jun-97
Jul-97 Aug-97 Sep-97 Oct-97 Nov-97 Dec-97 Jan-98 Feb-98 Mar-98 Apr-98 May-98 Jun-98
Date
Jul-98
98
-9
Ap 8
r-9
8
M
ay
-9
Ju 8
n9
Ju 8
l-9
8
A
ug
S 98
ep
-9
O 8
ct
-9
8
N
ov
-9
8
D
ec
-9
Ja 8
n9
Fe 9
bM 99
ar
-9
A 9
pr
M 99
ay
Ju 99
n9
Ju 9
l- 9
9
A
ug
-9
9
S
ep
-9
O 9
ct
-9
9
ar
98
7
7
7
20
b-
n-
-9
-9
-9
40
M
Fe
Ja
ec
ov
ct
97
7
7
50
D
N
O
p-
-9
l- 9
ug
Se
A
Ju
7
7
7
97
-9
-9
n-
ay
pr
97
97
-9
b-
n-
ar
Ju
M
A
M
Fe
Ja
Flow (L/s)
Permian Inflows
Parameter Sets
Set 1
Set 2
Set 3
30
Set 4
Set 5
Set 6
Set 7
Set 8
Set 9
10
0
Date
20
-9
Ap 8
r-9
8
M
ay
-9
Ju 8
n9
Ju 8
l-9
8
A
ug
S 98
ep
-9
O 8
ct
-9
8
N
ov
-9
8
D
ec
-9
Ja 8
n9
Fe 9
bM 99
ar
-9
A 9
pr
M 99
ay
Ju 99
n9
Ju 9
l- 9
9
A
ug
-9
9
S
ep
-9
O 9
ct
-9
9
40
98
50
ar
98
7
7
7
60
b-
n-
-9
-9
-9
70
M
Fe
Ja
ec
ov
ct
97
7
7
80
D
N
O
p-
-9
l- 9
ug
Se
A
Ju
7
7
7
97
-9
-9
n-
ay
pr
97
97
-9
b-
n-
ar
Ju
M
A
M
Fe
Ja
Flow (L/s)
Tertiary Sands Inflow
Parameter Sets
Set 2
Set 3
Set 4
Set 5
Set 6
Set 7
30
Set 8
Set 9
Curve 8
10
0
Date
Hydraulic property heterogeneity
correlation length
Hydraulic property correlation decreases with distance
correlation length
distance
Hydraulic property correlation decreases with distance
correlation length
variogram
distance
Hydraulic property measurement points
Hydraulic property correlation decreases with distance
correlation length
variogram
distance
Hydraulic property realisation
Hydraulic property realisation
constrained by point measurements
constrained by point measurements
Particularly useful in pathline analysis ...
...
Sensitivity analysis [calibrated model] …..
Estimated parameter values
p2
Objective function
minimum
p1
Estimated parameter values
p2
Objective function
minimum
Maximum probability
for p1 and p2
p1
Estimated parameter values
p2
Allowed
parameter
values
Maximum probability
for p1 and p2
p1
Estimated parameter values
p2
Allowed
parameter
values
Maximum probability
for p1 and p2
p1
Estimated parameter values – nonlinear case
p2
Allowed
parameter
values
Maximum probability
for p1 and p2
p1
value
Field or laboratory measurements and model output:-
Model output
calibration dataset
q2
q1
q3
etc
distance or time
prediction
value
Field or laboratory measurements and model output:-
Model output
calibration dataset
q2
q1
q3
etc
distance or time
Lower
predictive
limit
value
Field or laboratory measurements and model output:-
Model output
calibration dataset
q2
q1
q3
etc
distance or time
Upper
predictive
limit
value
Field or laboratory measurements and model output:-
Model output
calibration dataset
q2
q1
q3
etc
distance or time
Confidence
interval for
prediction
Estimated parameter values – nonlinear case
p2
Allowed
parameter
values
Maximum probability
for p1 and p2
p1
Estimated parameter values – nonlinear case
p2
knowledge
constraints
Allowed parameter
values
p1
A certain model prediction
p2
Increasing value
p1
Defining a confidence interval
p2
The critical
points
p1
value
Residuals
Model output
calibration dataset
q2
q1
q3
etc
distance or time
prediction
The variance of the residuals is:2

=  / (m - n)
m = number of observations
n = number of parameters
value
Field or laboratory measurements and model output:-
Model output
calibration dataset
q2
q1
q3
etc
distance or time
Confidence
interval for
prediction
value
Field or laboratory measurements and model output:-
Model output
calibration dataset
q2
q1
q3
etc
distance or time
Predictive
uncertainty
interval
Software for predictive uncertainty analysis
UCODE
• assumes model linearity
• only works with a few parameters
PEST
• full nonlinear predictive analysis
• unlimited number of parameters
For linear models…
Estimated parameter values:-
p2
Extreme values
of p1 and p2
p1
A simple lumped parameter model
A simple lumped parameter model
par1
par2
par5
par6
par3
par4
The covariance matrix of the
estimated parameter set is given by
C(p) = 2 (Mt QM)-1
For a nonlinear model replace M by
J, the Jacobian matrix.
C(p) = 2 (Jt QJ)-1
Let M (ie. green M) represent the action
of the model in predictive mode and o the
model outputs in predictive mode. Then
C(o) = M C(p)Mt
For a nonlinear model:C(o) = J C(p)Jt
Notice that predictions can be correlated.
value of prediction #1
Maximum probability
Bivariate probability density function.
For linear and nonlinear models…
PEST’s predictive analyzer
p2
Initial
parameter
estimates
The critical
point
p1
PEST’s predictive analyzer
p2
The critical
point
Initial
parameter
estimates
p1
Major problem with this approach
• assumes that there is an objective
function minimum
• assumes that this defines a unique set
of parameters
• thus it assumes that parameters are
“lumped” and that there aren’t many of
them
objective function contours
2.0
1.8
1.6
1.4
Log(T 2)
1.2
1.0
0.8
0.6
0.4
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
1.6
1.8
2.0
A Confined Aquifer
head
Fixed
Inflow
T1
Fixed head
T2
T3
objective function contours
2.0
1.8
1.6
1.4
Log(T 2)
1.2
1.0
0.8
0.6
0.4
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
1.6
1.8
2.0
Hillside and
Piezometers
Transmissivity distribution - I
100 m2/day
4000
3500
3000
2500
2000
1500
1000
500
0
0
1000
2000
3000
4000
5000
6000
Transmissivity distribution - II
360 m2/day
12 m2/day
4000
3500
3000
2500
2000
1500
1000
500
0
0
1000
2000
3000
4000
5000
6000
SNOW section
of the PERLND
module of HSPF
PWATER
section of the
PERLND
module of HSPF
PWATER
section of the
PERLND
module of HSPF
(continued)
Daily Flow
9000
8000
7000
6000
5000
4000
3000
2000
1000
0
1
649 1297 1945 2593 3241 3889 4537 5185 5833 6481 7129 7777 8425 9073 9721
1
627 1253 1879 2505 3131 3757 4383 5009 5635 6261 6887 7513 8139 8765 9391
9000
8000
7000
6000
5000
4000
3000
2000
1000
0
Monthly Volume
1.00E + 10
9.00E + 09
8.00E + 09
7.00E + 09
6.00E + 09
5.00E + 09
4.00E + 09
3.00E + 09
2.00E + 09
1.00E + 09
0.00E + 00
1
18
35
52
69
86
103 120 137 154 171 188 205 222 239 256 273 290 307
1
18
35
52
69
86
103 120 137 154 171 188 205 222 239 256 273 290 307
1.00E + 10
9.00E + 09
8.00E + 09
7.00E + 09
6.00E + 09
5.00E + 09
4.00E + 09
3.00E + 09
2.00E + 09
1.00E + 09
0.00E + 00
Exceedence fraction
1 .2
1
0 .8
0 .6
0 .4
0 .2
0
1 .2
1
2
3
4
1
2
3
4
5
1
0 .8
0 .6
0 .4
0 .2
0
5
In cases such as these (and most real-world
cases are the same), quantification of
predictive uncertainty must take place by other
means.
The linear method of confidence interval
estimation is impossible to apply because
parameter covariance matrices are singular
and the mathematics breaks down.
PEST’s ability to maximise/minimise a
prediction while still maintaining calibration
constraints can still be applied. This is an
excellent method to test the “wiggle room” of
a prediction. However quantification of
uncertainty limits becomes mathematically
more difficult.
Reality
Exit time = 3256
Exit point = 206
Calibration to 12 observations (no noise)
Exit time = 7122 [true=3256]
Exit point = 241 [true=206]
R e a lity
12 obs
Use predictive analysis to minimize travel time
K ranges from 1.9e-5 to 8813 m/day
Exit time = 280 [true=3256]
Exit point = 226 [true=206]
2 .5
2
1 .5
re a lity
2 0 0 p re d ictio n
1
0 .5
0
0
100
200
300
400
Mathematically, because of parameter
nonuniqueness, the parameter and
predictive uncertainty limits are huge. Limits
will then be set by parameter plausibility and
parameter relationships plausibility, rather
than simply by model-decalibration
considerations.
Special “calibration-constrained” monte
carlo methods are often used (see later).
Another major problem is this….
value
Field or laboratory measurements and model output:-
Model output
calibration dataset
q2
q1
q3
etc
distance or time
Predictive
uncertainty
interval
We know the residual variance when our
predictions are of the same type as the data
against which we calibrate. But what if we
are making a prediction of a different type or
at a different place? That is why we are
using a physically based model in the first
case. The “residual noise” can be enormous
(remember our travel time example from
earlier lecture).
Some examples…
A surface water modelling example
N O R T H C A R O L IN A
C o n te n tn e a C re e k
w a te rsh e d
N e u se R ive r b a sin
N C C o u n ty B o u n d a rie s
(7 7 k m 2 )
Sandy Run
(1 4 0 k m 2 )
M id d le S w a m p
L ittle C o n te n tn e a (4 7 0 k m 2 )
C o n te n tn e a
(2 6 0 0 k m 2 )
(1 4 5 0 0 k m 2 )
N e u se
Observed and modelled flows over
part of calibration period.
10000
1000
100
10
1
1 -Ja n -8 3
1-M ar-83
1-M ay -83
1 - J u l- 8 3
1-S ep-83
1-Nov -83
1 -Ja n -8 4
Observed and modelled monthly volumes
8 .E + 0 9
7 .E + 0 9
6 .E + 0 9
5 .E + 0 9
4 .E + 0 9
3 .E + 0 9
2 .E + 0 9
1 .E + 0 9
0 .E + 0 0
1970
1972
1974
1976
1978
1980
1982
1984
1986
Observed and modelled exceedence fractions
1
E xc e e d e n c e fr a c ti o n
0 .8
0 .6
0 .4
0 .2
0
10
100
1000
F l o w ( c u ft /s e c )
10000
800
TS S (m g /l)
600
400
200
0
1970
1975
1980
1984
1990
1995
1000
TS S (m g /l)
100
10
1
1970
1975
1980
1984
1990
1995
10000
80
8000
60
6000
40
4000
20
2000
0
0
1970
1975
1980
1984
1990
1995
M e a su re d flo w (c u ft /s e c )
TS S (m g /l)
100
100
TS S (m g /l)
1000
10
100
1
10
1976
1978
1979
1980
1980
1982
1983
1984
M e a su re d flo w (c u ft /s e c )
10000
1000
TS S (m g /l)
100
10
1
10
100
1000
Flo w (c u ft/se c)
10000
1000
TS S (m g /l)
100
10
1
1970
1975
1980
1984
1990
1995
1000
TS S (m g /l)
100
10
1
1970
1975
1980
1984
1990
1995
Points joined only as a method of
enhancing visual comparison of field
measurements with model outputs
1000
TS S (m g /l)
100
10
1
1970
1975
1980
1984
1990
1995
Observations used in calibration process
• Individual TSS values
• mean TSS (observed and modelled after interpolation)
• std dev of TSS (observed and modelled after interpolation
• bed sand, silt and clay unchanged over calibration period
Points joined only as a method of
enhancing visual comparison of field
measurements with model outputs
1000
TS S (m g /l)
100
10
1
1975
1980
1984
1990
1995
Measured and model-generated TSS timeinterpolated to measurement times
1000
TS S (m g /l)
100
10
1
1975
1980
1984
1990
1995
Measured and model-generated
TSS
1000
TS S (m g /l)
100
10
1
1976
1978
1979
1980
1980
1982
1983
1984
Measured and model-generated
TSS
1000
TS S (m g /l)
100
10
1
1976
1978
1979
1980
1980
1982
1983
1984
Aim of predictive analysis
Maximise/minimise a key model prediction
while maintaining the model in a calibrated
state.
In our case we will maximise/minimise the
total sediment mass carried by the creek
over the whole calibration period of 1970 to
1995.
Calibration limits
Objective function under calibration
conditions = 4.2E4
Model deemed to be uncalibrated when
objective function 4.8E4.
So very “tight” calibration constraints.
Maximum/minimum sediment mass
Maximum sediment mass = 1.5E6
tonnes
Minimum sediment mass = 7.9E5
tonnes
Minimum phi
Measured and model-generated
TSS
1000
TS S (m g /l)
100
10
1
1976
1978
1979
1980
1980
1982
1983
1984
Minimum phi
Minimized
prediction
Measured and model-generated
TSS
1000
TS S (m g /l)
100
10
1
1976
1978
1979
1980
1980
1982
1983
1984
Minimum phi
Minimized
prediction
Measured and model-generated
TSS
2500
TS S (m g /l)
2000
1500
1000
500
1976
1978
1979
1980
1980
1982
1983
1984
a groundwater modelling example
A hillside
A hillside and finite difference grid
Observation points
Transmissivity field:average=100m2/day
exponential variogram
X-correlation length = twice Y-correlation length
Recharge = 100mm/yr
Concentration of Effluent = 100 units
Effective porosity =5%
Long. dispersivity = 10m; Trans dispersivity = 1m
Log transmissivity distribution
average 100m2/day
green is higher
Piezometric surface
Contour interval = 5m
Solute concentration after 3 years leakage
Maximum concentration =
20 units
Remediation strategy: interception by pumping
Solute concentration after 3 years leakage
Maximum concentration =
20 units
Well pumps at
600m3/day
Solute concentration after 30 years remediation
Maximum concentration =
2.6 units /m3
Contaminant
outflow to river
is 83000 units
over 30 years
Well pumps at
600m3/day
However in real life we do not know
what is in the ground.
So we must calibrate a model.
What is the worst-case scenario;
ie. how ineffective could pumping
be while ensuring that our model
is still calibrated?
Parameterisation using pilot points
Pilot point
Observation
bore
Observed and model-generated water levels at bores
Contour interval = 2m
Concentration residuals
-0.23
-0.47
-0.29
-0.30
-0.30
-0.35
0.01
-0.05
-0.04
-0.08
-0.42
-0.02
-0.10
0.12
-0.34
Solute concentration after 30 years remediation
Maximum concentration =
1.6 units /m3
Contaminant
outflow to river
is 8.0106 units
over 30 years
Well pumps at
600m3/day
Log transmissivity distribution
green is higher
Contaminant
outflow to river
is 8.0106 units
over 30 years
Well pumps at
600m3/day
What is the best-case scenario;
ie. how effective could pumping
be?
Make sure that the model is still calibrated.
Observed and model-generated water levels at bores
Contour interval = 2m
Concentration residuals
-0.32
-0.39
-0.21
-0.40
-0.17
-0.32
-0.04
0.15
0.08
0.03
-0.21
-0.02
-0.57
-0.25
-0.32
Solute concentration after 10 years remediation
Maximum concentration =
5.9 units
(minimum contour at .01 units)
Contaminant
inflow to creek is
14400 units over
30 years
Well pumps at
600m3/day
Log transmissivity distribution
green is higher
Contaminant
inflow to creek is
14400 units over
30 years
Well pumps at
600m3/day
Fail-safe Remediation
• Plot worst-case river pollution against
pumping rate
• Choose minimum pumping rate that meets
regulatory requirements
Worst-case outflow
Results from individual
predictive analysis runs
M axim um R iver O utflow
10000000
1000000
100000
10000
400.00
800.00
1200.00
1600.00
Total pumping rate from two wells
2000.00
Solute concentration after 30 years of pumping
Add a second
bore
Worst-case outflow
10000000
M axim um R iver O utflow
One extraction bore
1000000
100000
Two extraction bores
10000
400.00
800.00
1200.00
1600.00
Total pumping rate from two wells
2000.00
This method of predictive uncertainty analysis has the
advantage that it is computationally reasonably
efficient.
However in some cases predictive uncertainty limits
could have been narrower if known or suspected
parameter relationships are enforced.
However this is very difficult where these relationships
are “smoothed” or modified by the regularisation
process which is necessary to achieve calibration.
This is an area of active research.
Calibration-constrained Monte-Carlo …..
GLUE Method (Lancaster University)
Estimated parameter values – nonlinear case
p2
Allowed
parameter
values
Maximum probability
for p1 and p2
p1
Estimated parameter values – nonlinear case
p2
Allowed
parameter values
based on linear
approximation
p1
Advantages:• does not rely on linearity assumption
• sometimes predictive probabilities can be
estimated
• robust
Disadvantages:• Only a few parameters can be examined – so
model may be over-simplified
• Hit to miss ratio probably extremely low
• Extremely computationally expensive
Bayes Theorem
p(,|yi)  p(yi| ,) p(,)
Bayes Theorem
p(,|yi)  p(yi| ,) p(,)
Markov-Chain Monte Carlo
p2
Allowed
parameter values
based on linear
approximation
p1
Advantages:• two orders of magnitude more efficient than GLUE
• Provides posterior parameter probability distribution
(from which predictive distribtution can be derived)
• robust
Disadvantages:• Only a few parameters can be examined – so
model may be over-simplified
• Computationally very expensive
“Warping”
A model grid
4000
D istance in m etres
3500
3000
2500
2000
1500
1000
500
0
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
Distance in metres
5500
6000
6500
7000
7500
8000
The model domain
Fixed head
4000
D istance in m etres
3500
Recharge = 1.0e-4 m/day
3000
2500
2000
1500
1000
Transmissivity = 100 m2/day
500
0
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
Distance in metres
5500
6000
6500
7000
7500
8000
Calculated heads
4000
D istance in m etres
3500
3000
2500
2000
1500
1000
500
0
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
Distance in metres
5500
6000
6500
7000
7500
8000
Observation bore locations
4000
D istance in m etres
3500
3000
2500
2000
1500
1000
500
0
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
Distance in metres
5500
6000
6500
7000
7500
8000
Pilot point locations
4000
D istance in m etres
3500
3000
2500
2000
1500
1000
500
0
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
Distance in metres
5500
6000
6500
7000
7500
8000
Methodology
• Generate a random field (rather, the log of a
random field).
• Multiply random field by field interpolated from
pilot points (use regularisation to make that field
as smooth as possible).
• Use multiplied field in model.
• Estimate “field multipliers” at pilot points through
calibration process (ie.estimate field multipliers
so as to minimise head residuals calculated
using multiplied field.)
Use of PEST’s regularisation mode essential
Generated random field
Calibrated multiplier field
Field used by model
Generated random field
Calibrated multiplier field
Field used by model
Calibration Process
Measurement Objective function
• comprised of differences between measured and
calculated heads at bores
• maximum permitted measurement objective function
supplied by user
Regularisation Objective function
• rises with heterogeneity of multipliers at pilot points
• weights assigned to homogeneity constraints
determined geostatistically
• regularisation process maximises homogeneity of
multiplier field
The following fields all calibrate the
model with an rms error of 0.12m
(ie. 12 cm).
Calibrated transmissivity field
Transmissivity ranges from 11 m2/day to 667 m2/day
Calibrated transmissivity field
Transmissivity ranges from 13 m2/day to 617 m2/day
Calibrated transmissivity field
Transmissivity ranges from 15 m2/day to 994 m2/day
Calibrated transmissivity field
Transmissivity ranges from 10 m2/day to 815 m2/day
Calibrated transmissivity field
Transmissivity ranges from 10 m2/day to 471 m2/day
Calibrated transmissivity field
Transmissivity ranges from 7 m2/day to 643 m2/day
Calibrated transmissivity field
Transmissivity ranges from 10 m2/day to 524 m2/day
Calibrated transmissivity field
Transmissivity ranges from 13 m2/day to 572 m2/day
Calibrated transmissivity field
Transmissivity ranges from 10 m2/day to 488 m2/day
Calibrated transmissivity field
Transmissivity ranges from 11 m2/day to 1809 m2/day
Calibrated transmissivity field
Transmissivity ranges from 7 m2/day to 722 m2/day
Study area
Model domain
Observed heads
Model grid
Pilot points
rms = 2in
The
heterogeneity
that MUST exist.
rms = 3.5in
The
heterogeneity
that MUST exist.
rms = 2.9in
The
heterogeneity
that MAY exist.
rms = 2.9in
The
heterogeneity
that MAY exist.
rms = 2.9in
The
heterogeneity
that MAY exist.
Generate field and “warp” them to
enforce calibration constraints…
565
593
675
506
…perform stochastic analysis…
80
1 .2
70
1
60
0 .8
50
F re q u e n cy
40
0 .6
30
0 .4
C u m u la tive
fre q u e n cy
20
0 .2
10
0
0
6 .2 5
1 3 1 .2 5
2 5 6 .2 5
3 8 1 .2 5
Exit points
…evaluate probabilities.
90
1 .2
80
1
70
60
0 .8
50
0 .6
40
30
S e rie s2
S e rie s1
0 .4
20
0 .2
10
0
3 1 6 2 .5
0
6 4 1 2 .5
9 6 6 2 .5
1 2 9 1 2 .5
Exit times
Objective function contours
p2
Objective function
minimum
Initial parameter estimates
p1
Objective function contours
p2
Objective function
minimum
Initial parameter estimates
p1
Objective function contours
p2
Objective function
minimum
Final parameter estimates
p1
N O R T H C A R O L IN A
C o n te n tn e a C re e k
w a te rsh e d
N e u se R ive r b a sin
N C C o u n ty B o u n d a rie s
(7 7 k m 2 )
Sandy Run
(1 4 0 k m 2 )
M id d le S w a m p
L ittle C o n te n tn e a (4 7 0 k m 2 )
C o n te n tn e a
(2 6 0 0 k m 2 )
(1 4 5 0 0 k m 2 )
N e u se
Observed and modelled flows over
part of calibration period.
10000
1000
100
10
1
1 -Ja n -8 3
1-M ar-83
1-M ay -83
1 - J u l- 8 3
1-S ep-83
1-Nov -83
1 -Ja n -8 4
Observed and modelled monthly volumes
8 .E + 0 9
7 .E + 0 9
6 .E + 0 9
5 .E + 0 9
4 .E + 0 9
3 .E + 0 9
2 .E + 0 9
1 .E + 0 9
0 .E + 0 0
1970
1972
1974
1976
1978
1980
1982
1984
1986
Observed and modelled exceedence fractions
1
E xc e e d e n c e fr a c ti o n
0 .8
0 .6
0 .4
0 .2
0
10
100
1000
F l o w ( c u ft /s e c )
10000
Parameter
LZSN
UZSN
INFILT
BASETP
AGWETP
LZETP
INTFW
IRC
AGWRC
2.0
2.0
0.0526
0.200
0.00108
0.50
10.0
0.677
0.983
Observed and modelled flows over
part of calibration period.
10000
1000
100
10
1
1 -Ja n -8 3
1-M ar-83
1-M ay -83
1 - J u l- 8 3
1-S ep-83
1-Nov -83
1 -Ja n -8 4
Observed and modelled monthly volumes
8.E+09
7.E+09
6.E+09
5.E+09
4.E+09
3.E+09
2.E+09
1.E+09
0.E+00
1970
1972
1974
1976
1978
1980
1982
1984
1986
Observed and modelled exceedence fractions
1
Exceedence fr action
0.8
0.6
0.4
0.2
0
10
100
1000
Flow ( c u f t/s ec )
10000
Parameter
LZSN
UZSN
INFILT
BASETP
AGWETP
LZETP
INTFW
IRC
AGWRC
Set 1 Set 2
2.0
2.0
2.0
1.79
0.0526 0.0615
0.200 0.182
0.00108 0.0186
0.50
0.50
10.0
3.076
0.677 0.571
0.983 0.981
Set 3
2.0
2.0
0.0783
0.199
0.0023
0.20
1.00
0.729
0.972
Set 4
2.0
2.0
0.0340
0.115
0.0124
0.72
4.48
0.738
0.986
Set 5
2.0
1.76
0.0678
0.179
0.0247
0.50
4.78
0.759
0.981
Set 6
2.0
2.0
0.0687
0.200
0.0407
0.50
2.73
0.320
0.966
Advantages:• simple to implement
• provides posterior parameter probability distribution (from
which predictive distribtution can be derived)
• can generate pre-warped parameter values according to
known prior probability and correlation relationships
• robust
Disadvantages:• can computationally expensive (but getting more
efficient)
• Convenient – but not quite theoretically correct
Some Conclusions…
Some conclusions
• The potential for model predictive uncertainty is
often very high – especially when a model needs to
predict something which is different from the data
used in its calibration.
• In most cases there is no such thing as “the”
model.
• Not only is “the” model just one of many models,
but the regularisation that allows uniqueness to
exist will probably introduce bias into predictions.
• Predictive uncertainty analysis should be an
essential part of model deployment
• In view of this, the separation between the
calibration and predictive process is artificial. A
model is never “calibrated”.
• Calibration is simply the imposition of a set of
constraints on parameter values; when making a
prediction, only use parameters which respect what
we know about the system, and which allow the
model to replicate past measurements
• In most cases (especially those involving
complexity) the level of parameter nonuniqueness
is still very high
• There is thus potential for high predictive
uncertainty as well.