Transcript Document

Summary of Experimental
Uncertainty Assessment
Methodology
F. Stern, M. Muste, M-L. Beninati, W.E. Eichinger
7/18/2015
1
Table of Contents

Introduction

Test Design Philosophy

Definitions

Measurement Systems, Data-Reduction Equations,
and Error Sources

Uncertainty Propagation Equation

Uncertainty Equations for Single and Multiple
Tests

Implementation & Recommendations
Introduction

Experiments are an essential and integral tool for
engineering and science

Experimentation: procedure for testing or determination of a
truth, principle, or effect

True values are seldom known and experiments have errors
due to instruments, data acquisition, data reduction, and
environmental effects

Therefore, determination of truth requires estimates for
experimental errors, i.e., uncertainties

Uncertainty estimates are imperative for risk assessments in
design both when using data directly or in calibrating and/or
validating simulation methods
Introduction

Uncertainty analysis (UA): rigorous methodology for
uncertainty assessment using statistical and engineering
concepts

ASME (1998) and AIAA (1999) standards are the most
recent updates of UA methodologies, which are
internationally recognized

Presentation purpose: to provide summary of EFD UA
methodology accessible and suitable for student and
faculty use both in classroom and research laboratories
Test design philosophy

Purposes for experiments:





Type of tests:




Science & technology
Research & development
Design, test, and product liability and acceptance
Instruction
Small- scale laboratory
Large-scale TT, WT
In-situ experiments
Examples of fluids engineering tests:






Theoretical model formulation
Benchmark data for standardized testing and evaluation of facility biases
Simulation validation
Instrumentation calibration
Design optimization and analysis
Product liability and acceptance
Test design philosophy

Decisions on conducting experiments: governed
by the ability of the expected test outcome to
achieve the test objectives within allowable
uncertainties

Integration of UA into all test phases should be a
key part of entire experimental program




Test description
Determination of error sources
Estimation of uncertainty
Documentation of the results
Test design philosophy
DEFINE PURPOSE OF TEST AND
RESULTS UNCERTAINTY REQUIREMENTS
SELECT UNCERTAINTY METHOD
-
DESIGN THE TEST
DESIRED PARAMETERS (C D, C R,....)
MODEL CONFIGURATIONS (S)
TEST TECHNIQUE (S)
MEASUREMENTS REQUIRED
SPECIFIC INSTRUMENTATION
CORRECTIONS TO BE APPLIED
DETERMINE ERROR SOURCES
AFFECTING RESULTS
YES
ESTIMATE EFFECT OF
THE ERRORS ON RESULTS
IMPROVEMENT
POSSIBLE?
NO
UNCERTAINTY
ACCEPTABLE?
NO
YES
NO
NO TEST
IMPLEMENT TEST
START TEST
RESULTS
ACCEPTABLE?
NO
YES
YES
NO
CONTINUE TEST
PURPOSE
ACHIEVED?
MEASUREMENT
SYSTEM
PROBLEM?
SOLVE PROBLEM
YES
ESTIMATE
ACTUAL DATA
UNCERTAINTY
-
DOCUMENT RESULTS
REFERENCE CONDITION
PRECISION LIMIT
BIAS LIMIT
TOTAL UNCERTAINTY
Definitions

Accuracy: closeness of agreement between
measured and true value

Error: difference between measured and true value

Uncertainties (U): estimate of errors in
measurements of individual variables Xi (Uxi) or
results (Ur) obtained by combining Uxi

Estimates of U made at 95% confidence level
Definitions
 Bias error : fixed,
systematic
 k+ 1
 Bias limit B: estimate
 Precision limit P:
estimate of 
 Total error:  =  + 
 k+ 1
X
X true
Xk
X k+ 1
(a) two readings

FREQUENCY OF OCCURRENCE
random
k
k
of 
 Precision error :
 = total error
 = bias error
 = precision error



X
X true

MAGNITUDE OF X
(b) infinite number of readings
Measurement systems, data
reduction equations, & error sources

Measurement systems for individual variables Xi:
instrumentation, data acquisition and reduction procedures,
and operational environment (laboratory, large-scale
facility, in situ) often including scale models

Results expressed through data-reduction equations
r = r(X1, X2, X3,…, Xj)

Estimates of errors are meaningful only when considered
in the context of the process leading to the value of the
quantity under consideration

Identification and quantification of error sources require
considerations of:


Steps used in the process to obtain the measurement of the quantity
The environment in which the steps were accomplished
Measurement systems and data
reduction equations

Block diagram showing elemental error sources, individual
measurement systems, measurement of individual variables,
data reduction equations, and experimental results
ELEMENTAL
ERROR SOURCES
1
2
J
INDIVIDUAL
MEASUREMENT
SYSTEMS
X
1
B ,P
X
2
B ,P
X
J
B,P
MEASUREMENT
OF INDIVIDUAL
VARIABLES
1
1
2
2
J
r = r (X , X ,......, X )
1
2
J
r
B, P
r
r
J
DATA REDUCTION
EQUATION
EXPERIMENTAL
RESULT
Error sources

Estimation assumptions: 95% confidence level,
large-sample, statistical parent distribution
MODEL FIDELITY AND
TEST SETUP:
- As built geometry
- Hydrodynamic deformation
- Surface finish
- Model positioning
TEST ENVIRONMENT:
- Calibration versus test
- Spatial/temporal variations
of the flow
- Sensor installation/location
- Wall interference
- Fluid and facility conditions
CONTRIBUTIONS
TO ESTIMATED
UNCERTAINITY
SIMULATION TECHNIQUES:
- Instrumentation interference
- Scale effects
DATA ACQUISITION AND
REDUCTION:
- Sampling, filtering, and statistics
- Curve fits
- Calibrations
Uncertainty propagation equation


Bias and precision errors in the measurement of Xi
propagate through the data reduction equation r = r(X1, X2,
X3,…, Xj) resulting in bias and precision errors in the
experimental result r
A small error (Xi) in the measured variable leads to a small
error in the result (r) that can be approximated using
Taylor series expansion of r(Xi) about rtrue(Xi) as
 r = r ( X i )  rtrue ( X i ) =  X

i
dr
dXi
The derivative is referred to as sensitivity coefficient. The
larger the derivative/slope, the more sensitive the value of
the result is to a small error in a measured variable
Uncertainty propagation equation

Overview given for derivation of equation describing the error
propagation with attention to assumptions and approximations used to
obtain final uncertainty equation applicable for single and multiple
tests

Two variables, kth set of measurements (xk, yk)
r = r ( x, y)
k
xk = xtrue +  xk +  xk
xtrue
y
x
k
 x = r x;  y = r y
k
y
xk
r
xk  xtrue  + r  yk  ytrue  + R2
x
y
 r = rk  rtrue =  x  x +  x + y  y +  y
k
r
k

rtrue
1
sensitivity coefficients
k
k
yk
r k = r (xk , yk )
The total error in the kth determination of r
k
x
k
yk = ytrue +  yk +  yk
rk  rtrue =
y
x
k
 rk
rk
ytrue
Uncertainty propagation equation


We would like to know the distribution of r (called the parent
distribution) for a large number of determinations of the result r
A measure of the parent distribution is its variance defined as

2
r
 
1 N
2
= lim    rk 
N  N
 k =1

2

Substituting (1) into (2), taking the limit for N approaching infinity,
using definitions of variances similar to equation (2) for  ’s and  ’s and
their correlation, and assuming no correlated bias/precision errors
 2 =  x2 2 +  y2 2 + 2 x y   +  x2 2 +  y2 2 + 2 x y  
r
x
y
x
y
x
x
x y
3
 ’s in equation (3) are not known; estimates for them must be made
Uncertainty propagation equation

Defining
uc2 estimate for  2r


bx2 , by2 , bxy2
estimates for the variances and covariances (correlated bias errors) of
the bias error distributions

S xw , S y2 , S xy
estimates for the variances and covariances ( correlated precision
errors) of the precision error distributions
equation (3) can be written as
uc2 =  x2bx2 + y2by2 + 2 x ybxy + x2 Sx2 + y2 S y2 + 2 x y Sxy
Valid for any type of error
distribution

To obtain uncertainty Ur at a specified confidence level (C%), a
coverage factor (K) must be used for uc: U r = Kuc
 For normal distribution,
K is the t value from the Student t distribution.
For N  10, t = 2 for 95% confidence level
Uncertainty propagation equation

Generalization for J variables in a result r = r(X1, X2, X3,…, Xj)
J 1
J
J
J 1
J
J
U =  B + 2  i k Bik +  P + 2  i k Pik
2
r
i =1
2
i
2
i
i =
i =1 k =i +1
i =1
r
X i
2 2
i i
i =1 k =i +1
sensitivity coefficients
Example:
CD =
D
= CD D,  ,U , A
2
1 2 U A
J
U
2
CD
J
=  B +  i2 Pi 2
2
i
i =1
2
i
i =1
2
 C 
 C 
 C 
 C 
=  D  BD2 + PD2 +  D  B2 + P2 +  D  BU2 + PU2 +  D  BA2 + PA2
 D 
 U 
 A 
  
2




2


2


Uncertainty equations for single and
multiple tests
Measurements can be made in several ways:

Single test (for complex or expensive experiments): one set
of measurements (X1, X2, …, Xj) for r


According to the present methodology, a test is considered a single test if the entire
test is performed only once, even if the measurements of one or more variables are
made from many samples (e.g., LDV velocity measurements)
Multiple tests (ideal situations): many sets of
measurements (X1, X2, …, Xj) for r at a fixed test condition
with the same measurement system
Uncertainty equations for single and
multiple tests

The total uncertainty of the result
U r2 = B2r + P2r
4
Br : same estimation procedure for single and
multiple tests

Pr : determined differently for single and multiple
tests

Uncertainty equations for single and
multiple tests: bias limits
 Br :
J 1
J
J
B =  B + 2  i k Bik
Sensitivity coefficients
2
r
i =1
2
i
2
i
i =1 k =i +1
r
X i
i =
 B : estimate of calibration, data acquisition, data reduction, conceptual
i
bias errors for Xi.. Within each category, there may be several elemental
sources of bias. If for variable Xi there are J significant elemental bias
errors [estimated as (Bi)1, (Bi)2, … (Bi)J], the bias limit for Xi is calculated as
J
2
k =1
k
B =  Bi 
2
i
 B : estimate of correlated bias limits for X and X
ik
i
L
Bik =  Bi  Bk 
 =1
k
Uncertainty equations for single test:
precision limits
 Precision limit of the result (end to end):
Pr = tSr
t: coverage factor (t = 2 for N > 10)
Sr: the standard deviation for the N readings of the result. Sr must be
determined from N readings over an appropriate/sufficient time interval
 Precision limit of the result (individual variables):
J
Pr =  (  i Pi )2
i=1
Pi = ti Si
the precision limits for Xi
Often is the case that the time interval is inappropriate/insufficient and Pi’s
or Pr’s must be estimated based on previous readings or best available
information
Uncertainty equations for multiple tests:
precision limits
 The average result:
1
r=
M
M
r
k
k =1
 Precision limit of the result (end to end):
 M rk  r 2 
S r = 

M

1
 k =1

tS r
Pr =
M
t: coverage factor (t = 2 for N > 10)
Sr : standard deviation for M readings of the result
 The total uncertainty for the average result:

U = B + P = B + 2 Sr
2
r
2
r
2
r
2
r
M

2
 Alternatively Pr can be determined by RSS of the
precision limits of the individual variables
1/ 2
Implementation

Define purpose of the test

Determine data reduction equation: r = r(X1, X2, …, Xj)

Construct the block diagram

Construct data-stream diagrams from sensor to result

Identify, prioritize, and estimate bias limits at individual variable level


Uncertainty sources smaller than 1/4 or 1/5 of the largest sources are neglected
Estimate precision limits (end-to-end procedure recommended)



Computed precision limits are only applicable for the random error sources that were
“active” during the repeated measurements
Ideally M  10, however, often this is no the case and for M < 10, a coverage factor t =
2 is still permissible if the bias and precision limits have similar magnitude.
If unacceptably large P’s are involved, the elemental error sources contributions must
be examined to see which need to be (or can be) improved

Calculate total uncertainty using equation (4)

For each r, report total uncertainty and bias and precision limits
Recommendations




Recognize that uncertainty depends on entire testing process and that
any changes in the process can significantly affect the uncertainty of
the test results
Integrate uncertainty assessment methodology into all phases of the
testing process (design, planning, calibration, execution and post-test
analyses)
Simplify analyses by using prior knowledge (e.g., data base),
concentrate on dominant error sources and use end-to-end calibrations
and/or bias and precision limit estimation
Document:





test design, measurement systems, and data streams in block diagrams
equipment and procedures used
error sources considered
all estimates for bias and precision limits and the methods used in their estimation
(e.g., manufacturers specifications, comparisons against standards, experience, etc.)
detailed uncertainty assessment methodology and actual data uncertainty estimates
References







AIAA, 1999, “Assessment of Wind Tunnel Data Uncertainty,” AIAA
S-071A-1999.
ASME, 1998, “Test Uncertainty,” ASME PTC 19.1-1998.
ANSI/ASME, 1985, “Measurement Uncertainty: Part 1, Instrument
and Apparatus,” ANSI/ASME PTC 19.I-1985.
Coleman, H.W. and Steele, W.G., 1999, Experimentation and
Uncertainty Analysis for Engineers, 2nd Edition, John Wiley & Sons,
Inc., New York, NY.
Coleman, H.W. and Steele, W.G., 1995, “Engineering Application of
Experimental Uncertainty Analysis,” AIAA Journal, Vol. 33, No.10,
pp. 1888 – 1896.
ISO, 1993, “Guide to the Expression of Uncertainty in Measurement,",
1st edition, ISBN 92-67-10188-9.
ITTC, 1999, Proceedings 22nd International Towing Tank Conference,
“Resistance Committee Report,” Seoul Korea and Shanghai China.