Transcript Six Sigma in Measurement Systems: Evaluating the Hidden Factory OK Inputs Operation Inspect First Time Correct NOT OK Rework Hidden Factory Scrap Time, cost, people Bill Rodebaugh Director, Six Sigma GRACE slide 1

Six Sigma in Measurement Systems:
Evaluating the Hidden Factory
OK
Inputs
Operation
Inspect
First Time
Correct
NOT
OK
Rework
Hidden Factory
Scrap
Time, cost, people
Bill Rodebaugh
Director, Six Sigma
GRACE
slide 1
Objectives

The Hidden Factory Concept




Review Key Measurement System metrics including
%GR&R and P/T ratio
Case Study at W. R. GRACE




What is a Hidden Factory?
What is a Measurement System’s Role in the Hidden
Factory?
Measurement Study Set-up and Minitab Analysis
Linkage to Process
Benefits of an Improved Measurement System
How to Improve Measurement Systems in an
Organization
slide 2
The Hidden Factory -- Process/Production
OK
Inputs
Operation
Inspect
First Time
Correct
NOT
OK
Rework
Hidden Factory
Scrap
Time, cost, people
•What Comprises the Hidden Factory in a Process/Production Area?
•Reprocessed and Scrap materials -- First time out of spec, not reworkable
•Over-processed materials -- Run higher than target with higher
than needed utilities or reagents
•Over-analyzed materials -- High Capability, but multiple in-process
samples are run, improper SPC leading to over-control
slide 3
The Hidden Factory -- Measurement Systems
OK
Sample
Inputs
Lab Work
Inspect
Production
NOT
OK
Re-test
Hidden Factory
Waste
Time, cost, people
•What Comprises the Hidden Factory in a Laboratory Setting?
•Incapable Measurement Systems -- purchased, but are unusable
due to high repeatability variation and poor discrimination
•Repetitive Analysis -- Test that runs with repeats to improve known
variation or to unsuccessfully deal with overwhelming sampling issues
•Laboratory “Noise” Issues -- Lab Tech to Lab Tech Variation, Shift to
Shift Variation, Machine to Machine Variation, Lab to Lab Variation
slide 4
The Hidden Factory Linkage




Production Environments generally rely upon inprocess sampling for adjustment
As Processes attain Six Sigma performance they begin
to rely less on sampling and more upon leveraging the
few influential X variables
The few influential X variables are determined largely
through multi-vari studies and Design of
Experimentation (DOE)
Good multi-vari and DOE results are based upon
acceptable measurement analysis
slide 5
Objectives

The Hidden Factory Concept




Review Key Measurement System metrics including
%GR&R and P/T ratio
Case Study at W. R. GRACE




What is a Hidden Factory?
What is a Measurement System’s Role in the Hidden
Factory?
Measurement Study Set-up and Minitab Analysis
Linkage to Process
Benefits of an Improved Measurement System
How to Improve Measurement Systems in an
Organization
slide 6
Possible Sources of Process Variation
Observed Process Variation
Actual Process Variation
Measurement Variation
Long-term
Short-term
Variation
Variation due
Variation due
Process Variation
Process Variation
w/i sample
to gage
to operators
Repeatability
 2Observed
Calibration
Stability
Linearity
  2 Actua l Pr ocess   2 Measuremen t System
 2 Measuremen t System   2 Re peatability   2 Re producibility
Pr ocess
We will look at “repeatability” and “reproducibility” as primary
contributors to measurement error
slide 7
How Does Measurement Error Appear?
Actual process variation No measurement error
LSL
Frequency
15
USL
10
5
0
30
40
50
60
70
80
90
100
110
Process
15
LSL
USL
10
Frequency
Observed process
variation With measurement error
5
0
30
40
50
60
70
Observ ed
slide 8
80
90
100
110
Measurement System Terminology





Discrimination - Smallest detectable increment between two measured values
Accuracy related terms
 True value - Theoretically correct value
 Bias - Difference between the average value of all measurements of a sample and the
true value for that sample
Precision related terms
 Repeatability - Variability inherent in the measurement system under constant
conditions
 Reproducibility - Variability among measurements made under different conditions
(e.g. different operators, measuring devices, etc.)
Stability - distribution of measurements that remains constant and predictable over time for
both the mean and standard deviation
Linearity - A measure of any change in accuracy or precision over the range of instrument
capability
slide 9
Measurement Capability Index - P/T

Precision to Tolerance Ratio
515
. *  MS
P/T 
Tolerance


Addresses what percent of the tolerance is taken up by
measurement error
Includes both repeatability and reproducibility


Usually expressed
as percent
Operator x Unit x Trial experiment
Best case: 10% Acceptable: 30%
Note: 5.15 standard deviations accounts for 99% of Measurement System (MS) variation.
The use of 5.15 is an industry standard.
slide 10
Measurement Capability Index - % GR&R
%R & R 




 Observed
 MS
x 100
Pr ocess Variation
Usually expressed
as percent
Addresses what percent of the Observed Process Variation is
taken up by measurement error
%R&R is the best estimate of the effect of measurement
systems on the validity of process improvement studies (DOE)
Includes both repeatability and reproducibility
As a target, look for %R&R < 30%
slide 11
Objectives

The Hidden Factory Concept




Review Key Measurement System metrics including
%GR&R and P/T ratio
Case Study at W. R. GRACE




What is a Hidden Factory?
What is a Measurement System’s Role in the Hidden
Factory?
Measurement Study Set-up and Minitab Analysis
Linkage to Process
Benefits of an Improved Measurement System
How to Improve Measurement Systems in an
Organization
slide 12
Case Study Background

Internal Raw Material, A1, is necessary for Final Product production






High Impact Six Sigma project was chartered to improve an important quality variable,
CTQ1
The measurement of CTQ1 was originally not questioned, but the team decided to study
the effectiveness of this measurement



Expensive Raw Material to produce – produced at 4 locations Worldwide
Cost savings can be derived directly from improved product quality, CpKs
Internal specifications indirectly linked to financial targets for production costs are used to
calculate CpKs
If CTQ1 of A1 is too low, then more A1 material is added to achieve overall quality – higher
quality means less quantity is needed – this is the project objective
The %GR&R, P/T ratio, and Bias were studied
Each of the Worldwide locations were involved in the study
Initial project improvements have somewhat equalized performance across sites. Small
level improvements are masked by the measurement effectiveness of CTQ1
slide 13
CTQ1 MSA Study Design (Crossed)
Site 1 Lab
Site 1 Sample 1 Site 1 Sample 2
Op 1 Op 2 Op 3
T1 T2
Site 2 Lab
Site 3 Lab
Site 4 Lab
Site 2 Sample 1…..
6 analyses/site/sample
2 samples taken from each site
2*4 Samples should be representative
Each site analyzes other site’s sample.
Each plant does 48 analyses
6*8*4=196 analyses
slide 14
Gage name:
Date of study:
Reported by:
Tolerance:
Misc:
Z-14 MSA
JULY 2002
All Labs
110
CTQ1 MSA Study Results (Minitab Output)
Surface Area
Components of Variation
Response By Sample
890
120
%Contribution
%Study Var
%Tolerance
Percent
100
80
840
60
790
40
20
740
0
Gage R&R
Repeat
Reprod
Sample
Part-to-Part
1
2
R Chart by Operator
Sample Range
100
CB1 CB2 CB3 LC1 LC2 LC3
V1
V2
V3 W1
4
5
6
7
8
Response By Operator
W2
890
W3
840
UCL=52.45
50
790
R=16.05
0
LCL=0
740
Oper
0
CB1 CB2 CB3 LC1 LC2 LC3 V1 V2 V3 W1 W2 W3
Xbar Chart by Operator
CB1 CB2 CB3 LC1 LC2 LC3
V1
V2
V3 W1
Operator*Sample Interaction
W2
UCL=851.5
850
Mean=821.3
800
Operator
900
W3
LCL=791.1
Average
900
Sample Mean
3
850
800
750
Sample
0
slide 15
1
2
3
4
5
6
7
8
CB1
CB2
CB3
LC1
LC2
LC3
V1
V2
V3
W1
W2
CTQ1 MSA Study Results (Minitab Session)
Source
DF
Sample
SS
MS
F
P
7
14221
2031.62
5.0079
0.00010
Operator
11
53474
4861.27
11.9829
0.00000
Operator*Sample
77
31238
405.68
1.4907
0.03177
Repeatability
96
26125
272.14
191
125058
Total
%Contribution
Source
VarComp
Total Gage R&R
617.39
90.11
Repeatability
272.14
39.72
Reproducibility
345.25
50.39
278.47
40.65
66.77
9.75
67.75
9.89
Operator
Operator*Sample
Part-To-Part
(of VarComp)
slide 16
Sample, Operator,
& Interaction are
Significant
CTQ1 MSA Study Results
Mean
Equal Variances Differences
within Groups
(Tukey Comp.)
Site
%GRR
P/T
Ratio
All
94.3
(78.6 – 100)*
116
16.05
No (0.004)
Only 1,2 No Diff.
Site 1
38.9
(30.0 – 47.6)
29
7.22
Yes (0.739)
All Pairs No Diff.
Site 2
91.0
(70.7 – 100)
96
17.92
Yes (0.735)
Only 1,2 Diff.
Site 3
80.0
(60.8 – 94.8)
79
20.37
Yes (0.158)
All Pairs No Diff.
Site 4
98.0
(64.8 – 100)
120
18.67
Yes (0.346)
Only 2,3 No Diff.
R-bar
*Conf Int not calculated with Minitab, Based upon R&R Std Dev
slide 17
DotplotsResults
of C16 by C17
CTQ1 MSA Study
(Minitab Output)
(group means are indicated by lines)
Dotplot of All Samples over All Sites
890
C16
840
790
slide 18
Site 3
WO SA
Site 2
VF SA
Site 1
LC SA
C17
CB SA
740
Site 4
CTQ1 MSA Study Results (Minitab Session)
Analysis of Variance for Site
Source
DF
SS
MS
F
P
3
37514
12505
26.86
0.000
Error
188
87518
466
Total
191
125032
Site
Individual 95% CIs For Mean
Based on Pooled StDev
Level
N
Mean
StDev
Site 1
48
824.57
15.38
Site 2
48
819.42
22.11
Site 3
48
800.98
20.75
Site 4
48
840.13
26.58
-+---------+---------+---------+----(---*---)
(---*---)
(---*---)
(---*---)
-+---------+---------+---------+-----
Pooled StDev =
21.58
795
810
825
Site and Operator are closely related
slide 19
840
Per
60
790
40
740
CTQ1
MSA Study Results (Minitab Output)
0
20
Gage R&R
Repeat
Reprod
Part-to-Part
X-bar
R of All Samples for All Sites
R Chart by Operator
Sample Range
100
CB1 CB2 CB3 LC1 LC2 LC3
V1
V2
V3 W1
W2
UCL=52.45
R=16.05
0
1
LCL=0
Discrimination
Index840is “0”,
however can
790
probably see
740
differences
of 5
Oper
0
CB1 CB2 C
Xbar Chart by Operator
CB1 CB2 CB3 LC1 LC2 LC3
V1
V2
V3 W1
O
W2
900
W3
UCL=851.5
850
Mean=821.3
800
LCL=791.1
Most850of the
samples are
800“noise”
seen as
Average
Sample Mean
900
750
Sample
0
slide 20
2
890
W3
50
Sample
1
Pe
50
CTQ10 MSA Study Results (Minitab Output)
Gage R&R
Repeat
Reprod
Part-to-Part
Sample Range
X-bar R ofRAll
Samples
for Site 4
Chart
by WO OP
70
60
50
40
30
20
10
0
W1
W2
W3
UCL=60.99
R=18.67
LCL=0
0
Xbar Chart by WO OP
Sample Mean
900
W1
W2
W3
UCL=875.2
850
Mean=840.1
LCL=805.0
800
0
•Mean
differences are seen in X-bar area
•Most of the samples are seen as “noise”
slide 21
Sampl
CTQ1 MSA Study ResultsR=17.92
– Process Linkage
0
LCL=0
760
Site
2
Example
0
LC OP
LC1
860
850
840
830
820
810
800
790
780
LC1
LC2
LC3
850
UCL=853.1
840
Mean=819.4
LCL=785.7
1000
1
1
1
900
1
1
4
800
6
6
222 4
6
1
MSA Study
820 Results with
810
800 Mean = 819.4
830
790
I and MR Chart for TSA (t)
0
Individual Value
LC OP*Sa
Average
Sample Mean
Xbar Chart by LC OP
Sample
11
22
22
6
2
55
6 6
662 62
2 22
UCL=899.2
Mean=832.5
2
5
1
1
LCL=765.8
1
700
Subgroup
0
100
g Range
150
1
100
1
200
300
400
1 1
Selected
Samples
are Representative
11
1
1
1
11
1
1
UCL=81.95
slide 22
1
2
3
2002 Historical
Process
Results with
Mean = 832.5
Perc
50
0
CTQ1 MSA
Study Results – Process Linkage
I and MR Chart for TSA (t)
Site 2 Example
810
760
Gage R&R
Sample Range
Individual Value
1000
Repeat
Reprod
Sample
Part-to-Part
2
3
4
1
R Chart by LC
1 1OP
1
1
1
100
900
1
800
50
6
222 4
6
6
LC3
22
22
6
2
1
1
1
700
0
0
100
2
810
760
300
LC OP
400
LC1
1
1
1
LC2
11
1
LC3
1
22
850
UCL=853.1
840
1
Mean=819.4
2
2
222
2
2
LC2
LC3
2002 Historical LC O
L
Process
L
L
Results with
Range = 25.08
Calc
for
pt7 to8 pt
4
5
6
LC OP*Sample Interaction
1
2
2 LCL=785.7
Average
1
11
LC1
8
MSA Study Results
with Range = 17.92,
LCL=765.8
Calc for Subgroup
Xbar Chart by LC
1 OP
860
850
100
840
830
820
50
810
800
790
0
780
7
Mean=832.5
0
150
6
UCL=899.2
R=17.92
LCL=0
200
55860
6 6
662 62
2 22
UCL=58.54
5
5
By LC OP
1LC2
LC1
4
Subgroup
MovingMean
Range
Sample
1
830
820
810
800
790
Sample
0
UCL=81.95
R=25.08
LCL=0
1
2
3
When comparing the MSA with process operation, a large
percentage of pt-to-pt variation is MS error (70%) --- a
back check of proper test sample selection
slide 23
CTQ1 MSA Study Results – Process Linkage
Site 2 Example

Key issue for Process Improvement Efforts is “When will we see
change?”





Initial Improvements to A1 process were made
Control Plan Improvements to A1 process were initiated
Site 2 Baseline Values were higher than other sites
Small step changes in mean and reduction in variation will achieve goal
How can Site 2 see small, real change with a Measurement System with
70+% GR&R?
Use Power and Sample Size Calculator with and without impact
of MS variation. Lack of clarity in process improvement work,
results in missed opportunity for improvement and continued
use of non-optimal parameters
slide 24
CTQ1 MSA Study Results – Process Linkage
Site 2 Example
2-Sample t Test
2-Sample t Test
Alpha = 0.05
Alpha = 0.05
Sigma = 22.23
Sample
Target
Actual
Difference
Size
Power
Power
2
2117
0.9000
4
530
6
Sigma = 6.67
Sample
Target
Actual
Difference
Size
Power
Power
0.9000
2
192
0.9000
0.9011
0.9000
0.9002
4
49
0.9000
0.9036
236
0.9000
0.9002
6
22
0.9000
0.9015
8
133
0.9000
0.9001
8
13
0.9000
0.9074
10
86
0.9000
0.9020
10
9
0.9000
0.9188
12
60
0.9000
0.9023
12
7
0.9000
0.9361
14
44
0.9000
0.9007
14
5
0.9000
0.9156
16
34
0.9000
0.9018
16
4
0.9000
0.9091
18
27
0.9000
0.9017
18
4
0.9000
0.9555
20
22
0.9000
0.9016
20
3
0.9000
0.9095
Simulated Reduction of Pt to Pt variation by 70% decreases
time to observe savings by over 9X.
slide 25
CTQ1 MSA Study Results – Process Linkage
Site 2 Example
Benefits of An Improved MS

Realized Savings for a Process Improvement Effort





More trust in all laboratory numbers for CTQ1
Ability to make process changes earlier with R-bar at 6.67


For A1, an increase of 1 number of CTQ1 is approximately $1 per ton
Change of 10 numbers, 1000 Tons produced in 1 month (832  842)
$1 * 10 * 1000 = $10,000
Previously, it would be pointless to make any process changes within the 22 point
range. Would you really see the change?
As the Six Sigma team pushes the CTQ1 value higher, DOEs and other
tools will have greater benefit
slide 26
Objectives

The Hidden Factory Concept




Review Key Measurement System metrics including
%GR&R and P/T ratio
Case Study at W. R. GRACE




What is a Hidden Factory?
What is a Measurement System’s Role in the Hidden
Factory?
Measurement Study Set-up and Minitab Analysis
Linkage to Process
Benefits of an Improved Measurement System
How to Improve Measurement Systems in an
Organization
slide 27
Measurement Improvement in the Organization

Initial efforts for MS improvement are driven on a BB/GB project basis




Intermediate efforts have general Operations training for lab personnel,
mostly laboratory management



Six Sigma Black Belts and Green Belts Perform MSAs during Project Work
Lab Managers and Technicians are Part of Six Sigma Teams
Measurement Systems are Improved as Six Sigma Projects are Completed
Lab efficiency and machine set-up projects are started
The %GR&R concept has not reached the technician level
Current efforts enhance technician level knowledge and dramatically
increase the number of MS projects




MS Task Force initiated (3 BBs lead effort)
Develop Six Sigma Analytical GB training
All MS projects are chartered and reviewed; All students have a project
Division-wide database of all MS results is implemented
slide 28
Measurement Improvement in the Organization

Develop common methodology for Analytical GB training
Six Sigma Step
Define
Measure
Analyze
Improve
Control
Action
 Target measurement
system for study
 Identify KPOVs
 Identify KPIVs
 Evaluate KPOV
performance
Typical Six Sigma Tools Used
Project Charter
 Measurement System
Analysis
 Reduce Reproducibility
 Reduce Repeatability
 Reduce Operator or
Instrument Bias
 Final Report
 Control Plan for KPIVs
slide 29
“Soft” tools: Process Map, Cause & Effect
Matrix, FMEA
“Stat” tools: Minitab Graphics, SPC,
Capability Analysis
Gage R&R, ANOVA, Variance Components,
Regression, Graphical Interpretation
“Soft” tools: Fishbone Diagram, Focused
FMEA
“Stat” tools: D-Study, t-Tests and
Regression, Design of Experiments
SPC, Reaction Plans, Control Plans, ISO
synergy, Mistake Proofing
Final Thoughts




The Hidden Factory is explored throughout all Six Sigma programs
One area of the Hidden Factory in Production Environments is
Measurement Systems
Simply utilizing Operations Black Belts and Green Belts to improve
Measurement Systems on a project by project basis is not the long term
answer
The GRACE Six Sigma organization is driving Measurement System
Improvement through:




Tailored training to Analytical Resources
Similar Six Sigma review and project protocol
Communication to the entire organization regarding Measurement System
performance
As in the case study, attaching business/cost implications to poorly performing
measurement systems
slide 30