Transcript ITRI投影片範本B

Discussion on DUT Uncertainty at
CCM-WGFF
Chun-Min Su
CMS/ITRI, Chinese Taipei
2015/7/20
Background (1)
• Paragraph N5 of MRA-D-04 says
“Contributions to the uncertainty stated on the
calibration certificate include the measured
performance of the device under test during its
calibration at the NMI or accredited laboratory.
CMC uncertainty statements anticipate this
situation by incorporating agreed-upon values for
the best existing devices.”
BED
Copyright 2012 ITRI 工業技術研究院
Background (2)
• Recommendations in DOCUMENT JCRB-8/9
“Uncertainty contributions of the device under
calibration or measurement” by Ad hoc JCRB
Working Group CMC Uncertainties:
1. It is recommended to exclude contributions to the
CMC uncertainty caused by the customer device
before or after its calibration or measurement at the
institute.
2. It is recommended to include contributions to the CMC
uncertainty caused by the best ordinarily available
customer device during its calibration or measurement
at the institute. In general, these values will be
published in Appendix C of the CIPM MRA.
For individual certificates the actual characteristics of
the customer device must be considered.
Copyright 2012 ITRI 工業技術研究院
Background (3)
ILAC-P14:12/2010
• ILAC Policy for Uncertainty in Cal. (Clause 5.4)
– Calibration laboratories shall provide evidence that
they can provide calibrations to customers in
compliance with 5.1 b) so that measurement
uncertainties equal those covered by the CMC. In the
formulation of CMC, laboratories shall take notice of
the performance of the “best existing device” which is
available for a specific category of calibrations.
– A reasonable amount of contribution to uncertainty
from repeatability shall be included and contributions
due to reproducibility should be included in the CMC
uncertainty component, when available. There should,
on the other hand, be no significant contribution to the
CMC uncertainty component attributable to physical
effects that can be ascribed to imperfections of even
the best existing device under calibration or
measurement.
Copyright 2012 ITRI 工業技術研究院
Background (4)
ILAC-P14:12/2010
– It is recognized that for some calibrations a “best
existing device” does not exist and/or contributions to
the uncertainty attributed to the device significantly
affect the uncertainty. If such contributions to
uncertainty from the device can be separated from
other contributions, then the contributions from the
device may be excluded from the CMC statement. For
such a case, however, the scope of accreditation shall
clearly identify that the contributions to the uncertainty
from the device are not included.
– NOTE: The term “best existing device” is understood
as a device to be calibrated that is commercially or
otherwise available for customers, even if it has a
special performance (stability) or has a long history of
calibration.
Copyright 2012 ITRI 工業技術研究院
Background (5)
ILAC-P14:12/2010
• ILAC Policy for Uncertainty in Calibration (Clause 6.4)
– Contributions to the uncertainty stated on the calibration certificate
shall include relevant short-term contributions during calibration
and contributions that can reasonably be attributed to the
customer’s device. Where applicable the uncertainty shall cover
the same contributions to uncertainty that were included in
evaluation of the CMC uncertainty component, except that
uncertainty components evaluated for the best existing device shall
be replaced with those of the customer’s device. Therefore,
reported uncertainties tend to be larger than the uncertainty
covered by the CMC. Random contributions that cannot be known
by the laboratory, such as transport uncertainties, should normally
be excluded in the uncertainty statement. If, however, a laboratory
anticipates that such contributions will have significant impact on
the uncertainties attributed by the laboratory, the customer should
be notified according to the general clauses regarding tenders and
reviews of contracts in ISO/IEC 17025.
• ILAC Policy for Uncertainty in Calibration (Clause 6.5)
– As the definition of CMC implies, accredited calibration laboratories
shall not report a smaller uncertainty of measurement than the
uncertainty of the CMC for which the laboratory is accredited.
Copyright 2012 ITRI 工業技術研究院
Background (6)
• At it 10th meeting, WGFF resolved
“DUT uncertainty will be agreed upon by the
WGFF, based on uncertainties of KC transfer
standards, and included in the CMC uncertainty
value.”
• The document “Proposed Definition of DUT
Uncertainty” was generated on Dec. 6, 2011 and
circulated among members of the WGFF
Copyright 2012 ITRI 工業技術研究院
Background (7)
The same values of DUT uncertainties for different instruments,
derived primarily from the KC reports, was proposed
Measurand / Device Under
Test
DUT
Uncertainty
(k=2)
Units
Comments
K1 Water flow
0.02
%
Based on CCM.ff-K1 report figures 5 to
8 and figures 19 to 22.
%
Based on CCM.ff-K2 report figures 2 and
4 to 7 showing repeated cals of TS at
NEL.
K2 Hydrocarbon liquid
flow
0.02
K3 Air speed
0.2+0.1/u(m/s)
%
Based on APMP.ff-K3 TS
reproducibility, see "Air Speed"
worksheet.
K5 High-pressure gas flow
0.05
%
Based on CCM.ff-K5b report table 2.
K6 Low-pressure gas flow
0.05
%
From Wright Flomeko 2007 paper on the
uncertainty of the CCM K6 TS.
Copyright 2012 ITRI 工業技術研究院
Background (8)
• Feedbacks to the Dec. 6, 2011 proposal
– The numbers suggested by WGFF are sensible values
but the inclusion of fixed figure is not acceptable.
– Development of guidance on the expression of
uncertainty in CMCs and how to include repeatability,
reproducibility and inter-comparison contributions to
uncertainty, and uncertainty budget template.
–…
Copyright 2012 ITRI 工業技術研究院
2015/7/20
9
Ongoing Discussion in WGFF
• “WGFF Guidelines for CMC Uncertainty and
Calibration Report Uncertainty”
– Jun. 11, 2012 version
• Read and discussed in 2012 WGFF meeting @ Colorado
Springs, CO, USA
• Agreed on establishment of the document and most of the
content
• Some revisions required and a review team formed
– Jul. 24, 2012 version
• Circulated among members of the review team
• Comments were given
– Sep. 19, 2012 version
• Circulated among members of the review team
• Comments being given
Copyright 2012 ITRI 工業技術研究院
2015/7/20
10
Introduction to the Guidelines (1)
• Define Calibration and Measurement Capability
uncertainty (UCMC) as the root-sum-of-squares
(RSS) of:
1. the base uncertainty of the reference standard based
on the GUM methodology;
2. the repeatability of calibration results measured using
the best existing device.
2
base
UCMC  k95uCMC  k95 u
2
repeat, BED
u
UCMC represents the 95 % confidence level uncertainty of the average of
n measurements from the reference standard. The number of
measurements n should match the normal procedures routinely followed
during calibration of a customer’s device.
Copyright 2012 ITRI 工業技術研究院
2015/7/20
11
How to Calculate UCMC?
• The best way is to use
the Welch-Satterthwaite
Method and a coverage
factor k95 that is based
on the effective degrees
of freedom
Copyright 2012 ITRI 工業技術研究院
To Welch-Satterthwaite or not to W-S?
“Comparison of W-S and t-Value Methods” prepared by Dr. John Wright
• 4 methods were compared
– Following Annex G of the GUM by using the W-S Method
2
2
UWS  k95uWS  k95 ubase
  repeat,
BED
– Using the approach that EURAMET proposed
t

2
Ut95 /2  2ut95 /2  2 ubase
  95  repeat, BED 
 2

2
– Using the formula
2
base
Ut68  2ut68  2 u

 t68
2
repeat, BED

2
– assuming that both components have infinite DoF
2
2
URSS  2 ubase
  repeat,
BED
Copyright 2012 ITRI 工業技術研究院
2015/7/20
13
To W-S or not to W-S: Comparison Result
Copyright 2012 ITRI 工業技術研究院
2015/7/20
14
To W-S or not to W-S: Recommendation
“Comparison of W-S and t-Value Methods” prepared by Dr. John Wright
• The Guidelines call for the use of the WelchSatterthwaite method or the Ut95/2 method
– The W-S method follows the specific GUM
recommendations
– The Ut95/2 method should be allowed since it is simple
to understand and the uncertainty values it produces
are conservative
Copyright 2012 ITRI 工業技術研究院
2015/7/20
15
Introduction to the Guidelines (2)
• CMC Uncertainty
– Base uncertainty
• uncertainty of the quantity of fluid contained within, delivered,
or passed through a device under test
• That each identified measurement, instrument, and influence
factor is assessed for repeatability, stability, reproducibility,
and historical performance is expected
– Repeatability of the BED
• By the nature of the test, the result will be a combination of the
repeatability of the chosen DUT and that of the facility
Copyright 2012 ITRI 工業技術研究院
2015/7/20
16
Introduction to the Guidelines (3)
• Procedure for Assessing Repeatability for a flow cal.
1.Establish the range of flows, temperatures, and other
conditions listed in the scope of the facility to be covered.
2.Define a number of test flows and conditions which within the
normal working practices, should demonstrate the least stable
conditions and one or two which should demonstrate the most
stable conditions. (The least stable results are generally found
at the highest or lowest ends of the reference standard range.)
3.Identify the BED to use for each test.
Copyright 2012 ITRI 工業技術研究院
2015/7/20
17
Introduction to the Guidelines (4)
4. Working within the operating procedures of the laboratory to
ensure each test can proceed without interruption within a
working day, perform multiple calibrations under stable flows
and conditions to produce performance indicator values for
the BED. The number of repeats should match the normal
practices used when calibrating a customer’s device. If
procedures dictate less than twenty points, the effect of the
smaller sample should be fully recognised in the analysis by
using the Welch-Satterthwaite or t-value approaches
described below.
Performance indicator may be error, meter factor, K-factor, discharge coefficient, etc.
Copyright 2012 ITRI 工業技術研究院
2015/7/20
18
Introduction to the Guidelines (5)

5. For each set of points, examine the data to determine the
distribution to establish if a ‘normal’ or Gaussian distribution
can be assumed.
6. If the distribution is approximately normal, calculate the
n standard deviation of the mean. If the sample size is twenty
or greater, the standard deviation of the mean can be
combined by RSS with the base uncertainty and expanded
using a coverage factor of 2. If the sample is smaller than
twenty then, 1) use the Welch-Satterthwaite method to
determine the degrees of freedom and the coverage factor
necessary to achieve a 95 % confidence level, or 2) multiply
the standard deviation by the 95 % confidence level t-value
for n-1, divide by 2 to give the standard uncertainty for RSS
with other components, and use a coverage factor of 2 on
the RSS to obtain the expanded uncertainty. If the
distribution of the results is not normal or Gaussian, then
further consideration and assessment is necessary to
determine the correct statistical approach.
Copyright 2012 ITRI 工業技術研究院
2015/7/20
19
Introduction to the Guidelines (6)
• Procedure for Assessing Repeatability for a volume cal.
1.Establish the volume range and type of volumetric equipment
covered in the scope of the calibration system.
2.Select the instruments (BED) to be tested.
3.Perform n tests of each instrument at the chosen volume, where
n is the number of repeated measurements normally performed
on a customer’s device.
4.Work within the operating procedures of the laboratory to
ensure each test can proceed without interruption within a
working day.
5.Take the standard deviation of the mean of the repeated
measurements and 1) use the Welch-Satterthwaite method to
determine the degrees of freedom and the coverage factor
necessary to achieve a 95 % confidence level , or 2) multiply
the standard deviation by the 95% confidence level t-value for
n-1, divide by 2 to give the standard uncertainty for RSS with
other components, and use a coverage factor of 2 for the
expanded uncertainty.
Copyright 2012 ITRI 工業技術研究院
2015/7/20
20
Note that
Introduction to the Guidelines (7)
• Reported Uncertainty for the Performance
Indicator (UPI)
– Include additional components over ubase due to:
• instrumentation and characteristics associated with the DUT
• fluid properties (if applicable)
• repeatability or short-term reproducibility for the customer’s DUT
2
2
2
UPI  2uPI  2 ubase
 uAI2  uprop
 urepeat
or reprod, DUT
Note: UPI must be > UCMC
Copyright 2012 ITRI 工業技術研究院
2015/7/20
21
Introduction to the Guidelines (8)
– Associated Instrumentation and Property Uncertainties
– Repeatability or Short-Term Reproducibility of the
Customer’s DUT
• Acceptable methods for quantifying ureprd,DUT or repeatability are (in
all cases, the method must be clearly stated in the report)
– The Welch-Satterthwaite method and effective degrees of
freedom applied to multiple measurements at each set point.
– The standard deviation of the mean of multiple measurements at
each set point (  n) with statistical corrections using the t-value
for the finite sample size.
– The standard deviation of multiple measurements at each set
point, with weighting based on the t-value at 95 % confidence for
the number of points and then divided by 2 to give the standard
uncertainty for RSS with other components.
Copyright 2012 ITRI 工業技術研究院
2015/7/20
22
Note that
Introduction to the Guidelines (9)
• Correlation Methods
– In some reference standards, two devices can be
installed in series and calibrated at the same time. This
allows the application of correlation techniques or
Youden analysis to separate the facility and BED
repeatability.
– Separating the repeatability by correlation methods is
acceptable and the facility portion can be used instead
of urepeat,BED to calculate UCMC. In this case, a statement
in the comments section is required, such as:
“Contributions to the uncertainty from the device are not
included.”
Copyright 2012 ITRI 工業技術研究院
2015/7/20
23
Thank you for your attention!
Copyright 2012 ITRI 工業技術研究院
2015/7/20
24