Transcript Prioritized Observation Uncertainty Quantification (POUQ

Framework for Quantification and
Reliability Analysis for Layered
Uncertainty using Optimization:
NASA UQ Challenge
Anirban Chaudhuri, Garrett Waycaster, Taiki Matsumura,
Nathaniel Price, Raphael T. Haftka
Structural and Multidisciplinary Optimization Group, University of Florida
1
NASA Problem Description
• Combined aleatory and epistemic uncertainty
– Epistemic uncertainty: 31 θ’s (Sub-parameters)
– Aleatory uncertainty: 21 p’s (Parameters)
Parameters
Intermediate
Variables
Constraints
Worst case
scenario
Performance
Metrics
Design Variables
2
Toy Problem
True distribution of G1
Function of G
G1 = 5(- P1 + P2 - (P3 – 0.5))
G2 = 0.7 – P3
g1obs
3000
2500
2000
P1: Constant
P2: Normal distribution
P3: Beta distribution
No Intermediate variables
Symbol Category
p1
II
p2
III
p3
III
1500
1000
500
0
-25
-20
-15
Uncertainty Model
Δ=[0, 1]
Normal, -2≤E[p2]≤1, 0.5≤V[p2]≤1.1
Beta, 0.6 ≤ E[p3] ≤ 0.8, 0.02 ≤ V[p3] ≤ 0.04
-10
-5
0
5
10
15
20
True Value
p1=0.5
E[p2]=0, V[p2]=1
E[p3]=0.7, V[p3]=0.03
w(p) = max(g1, g2)
J1  E[ w( p )]
J 2  1  P[ w( p )  0]
3
Task A: Uncertainty Characterization
4
Assumption and Approaches
• Assumption
The distribution of each uncertain parameter is
modeled as a uniform distribution.
• Approaches
1)Bayesian based approach
2)CDF matching approach
3)Prioritized Observation UQ approach
5
Bayesian Based Approach
• Uncertainty models θ are updated by Bayesian inference
f ( | x1,obs ) 
L( | x1,obs ) P( )
 L( | x
1,obs
) P( )d
θ
: set of uncertain parameters
x1,obs
: 1st set of observations
P(θ)
: prior distribution
L(θ|x1,obs): likelihood function
f(θ|x1,obs) : posterior distribution
• Marginal distribution of each parameter θi is obtained by
integration
f (i | x1,obs ) 
 ..  ..
f ( | x1,obs )d1...di 1di 1...d8
• Each marginal distribution (posterior) is obtained by Markov
Chain Monte Carlo (MCMC) method as a sample distribution.
6
CDF Matching Approach
• Two sample Kolmogorov-Smirnov (KS) test forms the
basis of this method.
• Use given observations for an empirical CDF (eCDF).
• We can also get confidence bands (10-95%) for these
eCDFs.
• Optimize 𝜃s to match these eCDFs and the confidence
bands using modified KS statistic, Dn,n’.
– Sum of distances instead of maximum distance.
Dn,n '   Fn ( x)  Fn ' ( x)
x
eCDF using given
observations
1 n
Fn ( X )   I X i  x
n i 1
eCDF using a particular 𝜃
realization and generate n’
samples using the aleatory
uncertainty
7
CDF Matching Approach
• For 31 eCDFs (actual
eCDF+30 confidence
bands), optimize the 𝜃s.
• 10-95% confidence bands.
• Gives a refined range of 𝜃s
from the 31 possibly
optimal 𝜃s.
Minimize Dn,n '

Such that, loweri  i  upperi for i  1, 2...8
DIRECT optimizer is
used (Finkel et al.)
8
Prioritized Observation UQ
• Both performance metrics measure risk.
• Refining the UQ based on amount of risk attached to an
observation.
• Similar strategy as the CDF matching method except the
objective function is weighted modified KS statistic.
DWn ,n '   (WR * Fn ( x)  Fn ' ( x) )
x
• WR is weight of the observation according to the risk associated
with it.
– Could be decided based on the J2 value.
– Implementation is very expensive in order to find J2 using Monte Carlo.
– Exploring Importance sampling or surrogate based strategies for future
work.
9
Toy Problem Results:
Posterior Distributions using Bayesian approach
P1
P2 (mean)
10
10
8
8
• Initially, the mean and variance
P2 are most uncertain (wider
ranges).
40
6
6
4
4
20
2
2
0
• 20 observations of G1 are used.
P3 (mean)
60
0
0.5
1
0
-2
-1
0
1
0
0.6
P2 (variance)
0.7
0.8
P3 (variance)
600
50
40
400
30
20
• While MCMC reduced the range
of mean and variance of P2, the
ranges of other parameters
remain.
 makes sense!
200
10
0
0.5
1
0
0.02
0.03
0.04
Posterior distributions updated by using 20 observations
True Value
10
Toy Problem Results:
Reduced bounds using CDF matching
Epistemic
uncertainty
parameter
True
Value
Δp1
μp2
σ2p2
E[p3]
V[p3]
0.5
0
1
0.7
0.03
Given Prior
Reduced Bounds using
20 observations
(Median)
Reduction in
median range (%
of prior range)
[0, 1]
[-2, 1]
[0.5, 1.1]
[0.6, 0.8]
[0.02. 0.04]
[0.0556, 0.9444]
[-0.5, 0.5]
[0.7663, 1.0988]
[0.6037, 0.7963]
[0.0204, 0.0396]
11.1%
66.7%
44.6%
3.7%
3.7%
• Using 20 observations for G1.
• Maximum reduction in bounds for mean and variance of P2.
• Similar results as the Bayesian Approach.
11
Toy Problem Results:
Effects of Number of Observations
• Started with only 5 observations of G1 and then increased to 20.
• Bayesian approach: MCMC provided sets of 𝜃s.
• CDF matching: around 7000 𝜃s generated in the updated range.
Create eCDF
using 1000
samples
Each 𝜃, 1000 G1
samples generated
Perform KS test to see if the
hypothesis that the CDF is same
as eCDF of the given all 20
observations is rejected.
Bayesian approach
K-S test rejection
percentage
Prior
Posterior by 5
observations
60.8%
69%
CDF Matching approach
Posterior by 20 Posterior by 5
observations
observations
0.7%
85.3%
Posterior by 20
observations
13.8%
• Rejection rate is substantially reduced by both approaches when
20 observations are used.
12
NASA Problem Results:
Posterior Distribution using Bayesian approach
P1 (mean)
P1 (mean)
P1 (variance)
500
50
0
0.6
0.65
0.7 0.75
P4 (mean)
0.8
1
0
-5
0
P5 (mean)
5
1
0
0.6
2
1
1
2
3
P5 (variance)
4
2
0
-5
0
P2
5
20
0
1
2
Rho
3
4
5
0.65
0.7 0.75
P4 (mean)
0.8
0
0.02 0.025 0.03 0.035 0.04
P4 (variance)
4
2
0
-5
0
P5 (mean)
5
0
4
4
2
2
0
-5
0
P2
5
0
1
2
3
P5 (variance)
4
1
2
Rho
3
4
-0.5
0
0.5
1
5
20
10
0
500
100
0
0.02 0.025 0.03 0.035 0.04
P4 (variance)
0
P1 (variance)
10
0
0.5
1
0
-1
-0.5
Posterior by first 25 observations
0
0.5
1
0
0
0.5
1
0
-1
Posterior by all 50 observations
13
NASA Problem Results:
Reduced bounds using CDF matching
Using first 25 observations
Symbol
E[p1]
V[p1]
p2
E[p4]
V[p4]
E[p5]
V[p5]
ρ
Given Prior
[0.6, 0.8]
[0.02, 0.04]
[0, 1]
[-5, 5]
[0.0025, 4]
[-5, 5]
[0.0025, 4]
[-1, 1]
Uncertainty Model
[0.6012, 0.7444]
[0.0209, 0.0344]
[0.1173, 0.9983]
[-4.8148, 4.4444]
[0.0765, 3.9589]
[-4.4444, 0]
[0.6688, 3.7779]
[-0.6914, 0.8889]
Reduction in range (% of prior range)
28.4%
32.1%
12.4%
7.4%
2.9%
55.6%
22.2%
21%
Uncertainty Model
[0.6267, 0.7667]
[0.0231, 0.04]
[0.1296, 0.9979]
[-4.8148, 3.2922]
[0.1423, 3.9260]
[-4.4444, 0.0412]
[1.5571, 3.9424]
[-0.6667, 0.8916]
Reduction in range (% of prior range)
30%
15.6%
13.2%
18.9%
5.3%
55.1%
40.3%
22.1%
Using all 50 observations
Symbol
E[p1]
V[p1]
p2
E[p4]
V[p4]
E[p5]
V[p5]
ρ
Given Prior
[0.6, 0.8]
[0.02, 0.04]
[0, 1]
[-5, 5]
[0.0025, 4]
[-5, 5]
[0.0025, 4]
[-1, 1]
14
NASA Problem Results:
Effects of Number of Observations
• Started with 25 observations of x1 and then increased to 50.
• Bayesian approach: MCMC provided sets of 𝜃s.
• CDF matching: around 7000 𝜃s generated in the updated range.
For each 𝜃, 1000 p
samples generated
Perform KS test to see if the
hypothesis that the CDF is same
as eCDF of the given all 50
observations is rejected.
Create eCDF
using 1000
samples
Bayesian approach
K-S test rejection
percentage
Prior
Posterior by 25
observations
62.4%
51.8%
CDF Matching approach
Posterior by 50 Posterior by 25 Posterior by 50
observations
observations
observations
37.6%
41.9%
30.8%
• Rejection rate is reduced by both approaches as compared to the prior.
15
Task B: Sensitivity Analysis
16
Primary objectives
• Effect of reduced sub-parameter bounds on intermediate
variable uncertainty
• Fix parameter values without error in intermediate variables
• Effect of reduced bounds on range of values of interest, J1
and J2
• Fix parameter values without error in range of J1 or J2
17
Intermediate Variable Sensitivity
• Reduce
theanalysis
range ofbased
sub-parameters
byin25%
repeat
Sensitivity
on changes
the and
bounds
of the
a above
process.
variable, rather than its value.
− Reduce upper bound.
• Empirical estimate of p-box of intermediate variable, x.
− Increase lower bound.
− Using double-loop Monte Carlo simulation.
− Centered reduction.
− Sample sub-parameter values within the bounds and
• Average
change in
the area realizations.
of the p-box brought about by these
subsequent
parameter
three reductions is a measure of sensitivity of these bounds.
Ainitial
Arevised
18
J1 and J2 Range Sensitivity
-For J1 and J2, we use the range of values from Monte Carlo
simulation
-Surrogate models are used to reduce computation of J1 and
J2
-Parameters are ranked based on each parameter’s sensitivity
on J1 and J2 using a rank sum score
19
Fixing Parameter Values
• We use DIRECT global optimization to maximize the
remaining uncertainty (either p-box area or J1/J2 range)
while fixing a single parameter.
• We generate an initial large random sample of all
parameters and replace one parameter with a constant.
• We fix parameters where the optimized uncertainty measure
is close to the initial value
20
Toy Problem Results
Percent Change
Effect on J1
G1 = 5(- P1 + P2 - (P3 – 0.5))
G2 = 0.7 – P3
P1: Constant
P2: Normal distribution
P3: Beta distribution
Effect on J2
Parameter 1
43.0%
21.8%
Parameter 2
86.3%
44.1%
Parameter 3
85.8%
44.2%
• Monte Carlo simulation introduces some error, as should be
expected.
• Able to accurately rank the sensitivities of each of the
parameter bounds, and suggests fixing unimportant
parameters at reasonable values.
21
NASA Problem Results: Revised
uncertainty model
• Initial intermediate variable analysis:
- We are able to fix nine parameters: 2, 4, 5, 7, 8, 13, 14,
15, and 17.
• Based on their expected impact on both J1 and J2, we select
revised models for parameters 1, 16, 18, and 21.
Parameter
1
6
10
12
16
18
20
21
Percent Change in J1
40%
13%
6%
19%
25%
33%
24%
51%
Percent Change in J2
88%
48%
53%
47%
64%
89%
55%
115%
22
Tasks C & D: Uncertainty Propagation
& Extreme Case Analysis
23
Primary objectives
• Uncertainty Propagation
– Find the range of J1 and J2
• Extreme Case Analysis
– Find the epistemic realizations that yield extreme J1 and J2 values
– Find a few representative realizations of x leading to J2 > 0
24
Double Loop Sampling (DLS)
• Double Loop Monte Carlo Sampling (DLS)
– Parameter Loop – samples sub-parameters (epistemic
uncertainty)
•
31 distribution parameters
– Probability Loop – samples parameters (aleatory
uncertainty)
•
17 parameters (p’s)
– Challenges:
•
Computationally expensive
25
Efficient Reliability Re-Analysis (ERR)
(Importance Sampling Method)
• Full double loop MCS is infeasible.
– Black box function g = f(x,dbaseline) is computationally expensive
• Instead of re-evaluating the constraints at each epistemic
realization we weigh existing points based on likelihood.
– Not importance sampling in traditional sense (i.e. No “important”
region”).
• How do we handle fixed but unknown constants that lie
within given interval?
– Generate initial p samples over entire range, [0,1]
– Use narrow normal distribution as “true” pdf
• pi ~ N(θi,0.25θi)
[1] Farizal, F., and Efstratios Nikolaidis. "Assessment of Imprecise Reliability Using Efficient
26
Probabilistic Reanalysis." System 2013: 10-17.
Optimized Parameter Sampling
• Optimization was used to find the epistemic realizations (𝜃s)
corresponding to extreme values of J1 and J2.
– The optimization process was repeated 4 times to minimize / maximize
J1 and J2.
• The objective function was based on ERR.
– Computational cost is significantly reduced.
• We need to use a global optimizer that is not gradient based.
– DIRECT algorithm by Daniel E. Finkel.
• Hasn’t been implemented on NASA problem yet due to high
computational costs.
– Use of surrogates is being explored.
27
Validation of ERR Method on Toy
Problem
Method used
DLS
ERR
Optimization with
ERR
Range of J1
[-0.10, 4.71]
[-0.11, 4.50]
[-0.10, 5.22]
Range of J2
[0.23, 0.94]
[0.13, 0.92]
[0.11,1]
[-0.10, 4.82]
[0.22, 0.96]
An MCS was performed using the
epistemic realizations from the
optimization.
• This shows that ERR method performed well when compared
to the more expensive DLS method for the toy problem.
Results of DLS for NASA problem
• It was only possible to use a small number of samples due
to computational time required
– 400 samples of the epistemic uncertainty
– 1,000 samples of the aleatory uncertainty
Range of J1
Range of J2
Given uncertainty model
[0.02, 5.19]
[0.08, 0.70]
Updated uncertainty model
(after getting revised 𝜽
bounds)
[0.03, 1.11]
[0.16, 0.76]
• Results show a significant reduction in range of J1
– Can we trust these results with such a small sample size?
29
Results of ERR method: NASA problem
Method
Range of J1
Range of J2
DLS
[0.02, 5.19]
[0.08, 0.70]
ERR
[0, 2.72]
[0, 1]
DLS with x to g 5th order PRS
Surrogate
[11.01, 33.26]
[0.36, 0.78]
• ERR results didn’t correspond very well with the DLS
results.
30
Limitations of current Importance sampling
based approach
• Good agreement with double loop sampling results for the toy
problem but not for NASA problem.
• Hypothesized that poor performance of the importance
sampling based approach is due to:
– Difficulty in creating initial set of samples with good coverage in 21
dimensional space (limited samples).
– Fixed but unknown constant parameters that were modeled using
narrow normal distributions.
• Possible fix:
– Dimensionality reduction by fixing the parameters through sensitivity
analysis.
– Use of surrogates to reduce computational time.
31
Summary
• Uncertainty Quantification using a given set of samples was
successfully performed using a Bayesian approach and a
CDF matching approach.
• P-box / reduction in range was used as the criterion to
decided the sensitivity of the parameters.
• An importance sampling based approach was utilized for
uncertainty propagation and extreme case analysis.
• A simpler toy problem was used validate all our methods,
increasing our confidence in the methods.
32
Thank You
Questions??
33
Back-Up Slides
34
Reduced bounds using CDF matching
Epistemic
uncertainty
parameter
Δp1
μp2
σ2p2
E[p3]
V[p3]
Reduced
True
Given
Bounds using 5
Value
Prior
observations
(Median)
0.5
[0, 1]
[0.0556, 0.9444]
0
[-2, 1]
[-0.5, 0.5]
1
[0.5, 1.1]
[0.7663, 1.0988]
0.7
[0.6, 0.8]
[0.6037, 0.7963]
0.03 [0.02. 0.04] [0.0204, 0.0396]
Reduction
Standard
Standard
in median
deviation of deviation of
range (% of
lower bound upper bound
prior range)
11.1%
0.0425
0.0201
66.7%
0.1410
0.0078
44.6%
0.1382
0.0586
3.7%
0.0032
0.0073
3.7%
0.0014
0.0007
Variation in lower bounds
Variation in upper bounds
1
1
0.5
0.8
0
0.6
-0.5
0.4
-1
0.2
-1.5
0
p1
E[p2]
V[p2]
E[p3]
V[p3]
p1
E[p2]
V[p2]
E[p3]
V[p3]
• Repeated the process 50 times.
35
MCMC Implementation (Backup Slide)
• Metropolis MCMC is used
• 20 MCMC runs (m=20)
* Sources of noise in output
- Different starting points *
- 10,000 posterior samples (2n=10,000)
- First 5000 samples are discarded for accuracy
• Proposal distribution* is a normal distribution with the standard deviation of 10% of
the prior range.
• 1000 random samples* is generated to construct an empirical PDF of G1 to calculate
the likelihood
• Likelihood (empirical PDF) is calculated by the kernel density estimation
- MATLAB ksdensity
1
𝑓(𝑥) =
𝑛ℎ
𝑛
𝑖=1
𝑥 − 𝑥𝑖
𝐾
ℎ
𝐾=Φ
ℎ = 𝜎𝐺1
4
3𝑛
1/5
36
MCMC Convergence (Backup Slide)
(1) Discard the first n draws
(2) Use Glen and Rubin Multiple Sequence Diagnostic
Potential scale reduction factor 𝑅 =
𝑉𝑎𝑟(𝜃)
𝑊
where
𝑉𝑎𝑟 𝜃 =
𝑛−1
1
𝑊+ 𝐵
𝑛
𝑛
𝑊: Within chain variance
𝐵: Between chain variance
(2) If 𝑅 is close to 1 (say less than 1.1), MCMC can be considered to be converged
and the total (m x n) draws are combined as a one chain.
37
Correlations between sub-parameters
E[p1]
E[p1]
V[p1]
p2
E[p4]
V[p4]
E[p5]
V[p5]
ρ
V[p1]
0.02
p2
0.03
0.05
E[p4]
-0.02
0.08
-0.04
V[p4]
-0.03
0.00
-0.05
E[p5]
0.08
0.02
-0.06
0.03
0.00
0.00
0.00
V[p5]
-0.03
0.06
-0.06
0.04
0.05
0.04
0.00
ρ
-0.06
0.01
0.05
0.03
-0.03
0.00
-0.02
38
Task B Summary
• Evaluating sensitivity using p-box and range as a metric to quantify
changes
• Surrogate models are utilized to reduce the computational expense of
the double loop simulation
• Parameter values are fixed by optimizing the remaining uncertainty
using DIRECT global optimization
• Refined models are requesting based on the rank sum score of each
parameter for both values of interest, J1 and J2
• Though the Monte Carlo simulation and surrogate models introduce
errors through approximation, our simple toy problem suggests this
method is still adequate to provide rankings of parameter sensitivity
39
Other Methods That Were Tried…
• P-box Convolution Sampling
– Requires replacing distributional p-box with free p-box
• Failure Domain Bounding (Homothetic Deformations)
– NASA UQ toolbox for Matlab has steep learning curve
– Theoretical background is challenging
• Replacing x to g function with surrogates
– Requires 8 surrogates (one for each constraint function) in 5
dimensional space
– Exploration of functions indicates delta function type behavior
that is difficult to fit with surrogate
– Attempts at creating PRS and Kriging surrogates results in poor
accuracy
40
10 / 10
Importance Sampling Formulation
• Worst case requirement metric
J1  E  max g i   E  w  p, d baseline  
 1i 8 




w(p) f (p | θ) dp 

1

N

w(p)

f (p | θ)
g (p)dp
g (p)
f (p i | θ)
w(p i )

g (p i )
i 1
N
• Similarly, for probability of failure
1
J 2  1  P  w(p, dbaseline )  0 
N
N
 I  w  p , d
i 1
i
baseline
  0
f (pi | θ)
g (pi )
41
4 / 10
Sampling Distributions
• Sampling Distributions
– 19 p’s are bounded between 0 and 1 (Beta, Uniform, or
Constant).
• Uniform sampling distribution is used.
– 2 p’s are normally distributed and possibly correlated.
• Samples must cover a large range.
– -5 ≤ E[pi] ≤ 5
– 1/400 ≤ V[pi] ≤ 4
• Uncorrelated multivariate normal distribution with
mean of 0 and standard deviation of 4.5 is used.
• 8 constraint functions are evaluated for 1e6 realizations of p.
42
Epistemic Realizations Corresponding
to J1/J2 Extrema: Toy Problem
Min J1
Max J1
Min J2
Max J2
0.60
0.07
1.00
0.03
-2.00
1.00
-1.43
1.00
0.52
1.10
0.50
0.50
0.80
0.60
0.80
0.60
0.03
0.02
0.02
0.02
Given Uncertainty model
Updated Uncertainty model
400
400
300
300
200
200
100
100
0
0
2
4
6
0
J1
0
0.5
1
J1
120
150
100
80
100
60
40
50
20
0
0
0.2
0.4
J2
0.6
0.8
0
0
0.2
0.4
0.6
0.8
J2
44
NASA Problem ERR error
Percent error between MCS estimates for J1 and J2 using 1,000 p samples
and ERR estimates using 1e6 initial samples
Percent Error (%)
Percent Error in Mean
Percent Error at Max
Percent Error at Min
Max Percent Error
Worst-Case Requirement
Metric (J1)
75%
97%
70%
4,110%
Probability of Failure
(J2)
29%
84%
40%
5,670%
45