Transcript Slide 1
Establishing the Integrity
of Data:
Measurement Systems Analysis
prepared by
Jeffrey T. Luftig, Ph.D.
W. Edwards Deming Professor of Management
Lockheed Martin Engineering Management Program
University of Colorado at Boulder
Topics
Measurement Scales and Types of
Data
Establishing the
Reliability and Validity of
Instrumentation, or
Precision and Accuracy of
Instrumentation
Measurement Scales and Data
Measurement Scales and Data
Measurement as a Process
As in any process, regardless of the nature
of data collected or generated,
measurement systems must demonstrate
Stability through time, or control
Minimal variation as a proportion of
specifications, or capability
Minimal variation as a proportion of
process variation
Measurement as a Process
Standard
Product or
Process to be
Measured
MEASUREMENT
Procedure
Equipment
PROCESS
Operator
Ambient
Environmental
Characteristics
Measurement
Definition of Terms
Reference Value
The theoretically or agreed upon correct
value of the characteristic being measured,
traceable to some standard
Resolution
The smallest increment, or unit of measure,
available from a measurement process
Generally at least 1/10th of the specification
range
Definition of Terms
Precision
The degree of agreement (or variability)
between individual measurements or test
results from measuring the same
specimen(s)
Accuracy (Bias)
The difference between the average of the
measurement error distribution and the
reference value of the specimen measured
Precision
Precision vs. Accuracy
Accuracy
Definition of Terms
Repeatability
The variation in repeated measurements of the
same items with a single measurement system
Within appraiser/system variation
Reproducibility
The variation in the average measurements by
different appraisers or systems measuring the
same items
Between appraiser/system variation
Measurement Error
Distribution of repeated
measures on a single
specimen or part
Precision
- Repeatability
- Reproducibility
Accuracy
(Bias)
Reference Value
Terms
Linearity
The degree to which bias changes with
changes in the magnitude of the
characteristic measured
Stability
The dependability, or consistency of the
measurement process over time
Measurement
Systems Capability
The variability resulting from measurement
error must not exceed a significant
proportion of the intended specifications
said to be capable
In addition, it is not desirable for
measurement error to exceed a significant
proportion of the total process variability
Capability is not the same as acceptability,
acceptability must be determined on a case
by case basis
Measurement Systems Capability
Measurement
Error Distribution
LSL
2
2
E Rpt
Rpd
5.15E
(USL - LSL)
5.15 E
%R &R
100
USL LSL
USL
Measurement System Studies
Potential Studies
Assess potential of a measurement
system to be capable over the long term
10 parts measured 2–3 times by one or
more appraisers
A “quick and dirty” study to find out if you
are in the ballpark
Assesses repeatability and reproducibility
Often called an R&R study
Measurement System Studies
Potential Studies
Error Through Time
Tests of Between-Subjects Effects
Dependent Variable: ADAS ADA within Period
Source
Time Period
Specimen
Residual
Total
Type III Sum
of Squares
54.021
325.873
431.604
811.498
df
3
7
21
31
Mean Square
18.007
46.553
20.553
F
.876
Sig .
.469
.097
Sig .
.961
Bias Through Time
Tests of Between-Subjects Effects
Dependent Variable: MEASURE Gram Weight
Source
Time Period
Specimen
Residual
Total
Type III Sum
of Squares
21.844
925.719
1571.406
2518.969
df
3
7
21
31
Mean Square
7.281
132.246
74.829
F
Measurement System Studies
Potential Studies (continued)
Multiple Comparisons
Dependent Variable: Gram Weight
Tukey HSD
(I) Time of Day Parts
Were Measured
1 Early Morning
2 Late Morning
3 Early Afternoon
464.0
4 Late Afternoon
463.5
Mean
(J) Time of Day Parts
Difference
Were Measured
(I-J)
Std. Error
2 Late Morning
-.75
4.33
3 Early Afternoon
1.50
4.33
4 Late Afternoon
-.13
4.33
1 Early Morning
.75
4.33
3 Early Afternoon
2.25
4.33
4 Late Afternoon
.63
4.33
1 Early Morning
-1.50
4.33
Weight Values Repeated
Through Time4.33
2Mean
LateGram
Morning
-2.25
4 Late Afternoon
-1.63
4.33
1 Early Morning
.13
4.33
2 Late Morning
-.63
4.33
3 Early Afternoon
1.63
4.33
Estimated Means
Based on observed means.
463.0
462.5
462.0
461.5
Early Morning
Late Morning
Early Afternoon
Late Afternoon
Sig .
.998
.985
1.000
.998
.953
.999
.985
.953
.981
1.000
.999
.981
95% Confidence Interval
Lower Bound
Upper Bound
-12.81
11.31
-10.56
13.56
-12.18
11.93
-11.31
12.81
-9.81
14.31
-11.43
12.68
-13.56
10.56
-14.31
9.81
Gram
W eight
-13.68
10.43
a,b
Tukey HSD
-11.93
12.18
Time
of Day Parts 11.43
-12.68
Were Measured
N
-10.43
13.68
3 Early Afternoon
1 Early Morning
4 Late Afternoon
2 Late Morning
Sig .
Means for groups in homogeneous subsets are displayed.
Based on Type III Sum of Sq uares
The error term is Mean Square(Error) = 74.829.
a. Uses Harmonic Mean Sample Size = 8.000.
b. Alpha = .05.
Time of Day Parts Were Measured
8
8
8
8
Subset
1
461.63
463.13
463.25
463.88
.953
Results & Conclusions: Evaluating the
Precision & Accuracy of the Measurement
System
This result of the
previous analysis
allows us to
calculate the
average variance
of the repeated
measures, which
when we take its
square root gives
us the estimate of
the standard
deviation due to
measurement
error:
2 = 66.39
= 8.15
Results & Conclusions: Evaluating the
Precision & Accuracy of the Measurement
System
Using the estimate
of measurement
error, we can
calculate the
Precision-Tolerance
ratio, which in the
case of short-term
studies, should be
less than 10%.
Assuming the
engineering tolerance
for this process is
470 (USL) –
450(LSL) = 20:
2 = 66.39
= 8.15
P/T = Precision-Tolerance Ratio =
= 6() / USL-LSL
= 6(8.15) / 470 – 450
= 2.44
= 244% > 10% Requirement (S-T)
Results & Conclusions: Evaluating the
Precision & Accuracy of the Measurement
System
Likewise, we can
estimate the
Accuracy (amount of
Bias) in the scale by
calculating the
average of the
differences between
the Means of the
Repeated Measures
and the True Values
for the associated
specimens:
= -3.09
Estimate Bias at 3.09 Grams; as
compared to the Precision estimate,
this is arguably an inconsequential
value.
Measurement System Studies
Short-term Studies
25 parts measured 5-8 times by one or
more appraisers
A more thorough short-term assessment
Long-term Studies
8-10 parts measured 25+ times by one or
more appraisers
Assesses through time stability
Measurement System Studies
Long-term Studies
Measurement System Studies
Long-term Studies
Measurement Systems
Requirements
Summary
Regardless of the type of data gathered by an
instrument, and the assessment methodology
employed, the instrument or device utilized to
obtain criterion data must meet three
requirements before the experiment should
proceed:
The instrument must be precise or reliable;
The instrument must be accurate or valid; and
The instrument should be / must be operating in a state
of statistical control.
Sources and References
The material used in the PowerPoint presentations associated with this course was drawn from a number
of sources. Specifically, much of the content included was adopted or adapted from the following
previously-published material:
Luftig, J. A Quality Improvement Strategy for Critical Product and Process Characteristics. Luftig
& Associates, Inc. Farmington Hills, MI, 1991
Spooner-Jordan, V. Understanding Variation. Luftig & Warren International, Southfield, MI 1996
Luftig, J. and Petrovich, M. Quality with Confidence in Manufacturing. SPSS, Inc. Chicago, IL
1997
Littlejohn, R., Ouellette, S., & Petrovich, M. Black Belt Business Improvement Specialist
Training, Luftig & Warren International, 2000
Ouellette, S. Six Sigma Champion Training, ROI Alliance, LLC & Luftig & Warren, International,
Southfield, MI 2005
Luftig, J. An Overview of Total Quality Management, Luftig & Warren, International, 1992
Luftig, J. Dr. Deming’s Theory of Profound Knowledge as a Foundation for Strategic Planning
and Policy Deployment, Luftig & Warren, International, 1997
Luftig, J. and Jordan, V. Design of Experiments in Quality Engineering, McGraw-Hill/Irwin
Publishing Company, 1998
25