Transcript Slide 1

Export Approval: #2090010808
“Proof” that Process Improvement Works!
Or How I Learned to Stop Worrying
and Love Quantitative Management …
©2008 BAE Systems.
Organizational Systemic Causal Analysis & Resolution Group
Metrics Analysis Group
(Individual Mileage Will Vary!)
Proof that Process Improvement Works!
4 Scenarios In Attaining High Maturity
The following set of scenarios are based on true stories!
These events actually occurred
within real-life organizations on real-life projects
that employed real-life people!
The measurements shown came from
actual reported data!
Project names have been changed to
protect the “innocent”!
©2008 BAE Systems.
Organizational Systemic Causal Analysis & Resolution Group
Metrics Analysis Group
2
Proof that Process Improvement Works!
(Setting the Stage – Our History with CMM/CMMI)
CMMI V1.2
2006
Level 5
CMMI V1.1
Attempted SWCMM Level 4 but
only achieved
SW-CMM Level 3
2003
SW-CMM
CMM/CMMI
SE-CMM
SW-CMM
SE-CMM
2002
2000
1998
SE-CMM
SW-CMM
SE-CMM
SW-CMM
1997
1996
SE-CMM
SW-CMM
SW-CMM
1994
1989
Level 1
SW-CMM
Level 2
1992
Level 3
1st SW-CMM Assessment
Level 4
Process Improvement Achievements
©2008 BAE Systems.
Organizational Systemic Causal Analysis & Resolution Group
Metrics Analysis Group
3
Proof that Process Improvement Works!
(Setting the Stage – The Measures)
Post Delivery Defect Density w/ Org Target Max
6.0
Below Org Target Max Goal
Near Org Target Max Goal
5.0
Slightly Above Org Target Max Goal
Well Above Org Target Max Goal
Org Target Maximum Goal
PDDD ( Defects Per KSLOC)
4.0
COCOMO II Productivity Factor - Unit Cost w/ Org Target Max
1.75
3.0
Below Org Target Max Goal
Near Org Target Max Goal
1.50
2.0
Slightly Above Org Target Max Goal
Well Above Org Target Max Goal
Org Target Max
Org Target Maximum Goal
1.0
.75
1.00
Feb-06
Nov-05
Aug-05
Mar-05
May-05
Dec-04
Jun-04
Sep-04
Mar-04
Dec-03
Jun-03
Sep-03
Mar-03
Dec-02
Jun-02
Sep-02
Mar-02
Dec-01
Jun-01
Sep-01
Mar-01
Dec-00
Jun-00
Sep-00
Mar-00
Dec-99
Jul-99
Sep-99
0.75
Apr-99
Oct-98
0.0
Jan-99
Unit Cost - (Person Hours Per SLOC)
1.25
0.50
Org Target Max
.300
0.25
©2008 BAE Systems.
Feb-06
Nov-05
Aug-05
May-05
Mar-05
Sep-04
Dec-04
Jun-04
Mar-04
Dec-03
Jun-03
Sep-03
Mar-03
Dec-02
Jun-02
Sep-02
Mar-02
Dec-01
Jun-01
Sep-01
Mar-01
Dec-00
Jun-00
Sep-00
Mar-00
Dec-99
Jul-99
Sep-99
Apr-99
Oct-98
Jan-99
0.00
Organizational Systemic Causal Analysis & Resolution Group
Metrics Analysis Group
4
Proof that Process Improvement Works!
Scenario 1) Project X
Project X’s initial attempt at implementing the company’s
standard engineering process set is abandoned after just
2 baseline releases due to a perceived decrease in its
Productivity measure.
However, instrumentation for Product Quality was left in
place when project attempted to return to its previous
development processes.
Project X’s subsequent release realizes an extremely poor
Product Quality measure and Project X decides it should
re-adopt the company’s standard development
processes, ultimately becoming a champion and pilot for
more modern development methodologies.
©2008 BAE Systems.
Organizational Systemic Causal Analysis & Resolution Group
Metrics Analysis Group
5
Proof that Process Improvement Works!
COCOMO II Productivity Factor - Unit Cost – Project X
1.75
Below Org Target Max Goal
Project X has
reasonable
productivity but
unknown product
quality
Unit Cost - (Person Hours Per SLOC)
1.50
1.25
Near Org Target Max Goal
Slightly Above Org Target Max Goal
Well Above Org Target Max Goal
Org Target Maximum Goal
1.00
Project X abandons new
process set due to
perceived impacts to
development team …
Project X undertakes
new process set
0.75
0.50
Org Target Max
.300
©2008 BAE Systems.
Organizational Systemic Causal Analysis & Resolution Group
Metrics Analysis Group
Feb-06
Nov-05
Aug-05
May-05
Mar-05
Dec-04
Sep-04
Jun-04
Mar-04
Dec-03
Sep-03
Jun-03
Mar-03
Dec-02
Sep-02
Jun-02
Mar-02
Dec-01
Sep-01
Jun-01
Mar-01
Dec-00
Sep-00
Jun-00
Mar-00
Dec-99
Sep-99
Jul-99
Apr-99
Oct-98
0.00
Jan-99
0.25
6
Proof that Process Improvement Works!
Post Delivery Defect Density – Project X
6.0
96.866
Project X abandons
new process set
PDDD ( Defects Per KSLOC)
5.0
… Project X realizes
unacceptable level
of product quality &
re-institutes new
process set
implementation
4.0
Below Org Target Max Goal
Near Org Target Max Goal
3.0
Slightly Above Org Target Max Goal
Well Above Org Target Max Goal
Org Target Maximum Goal
2.0
Project X undertakes
new process set
Org Target Max
1.0
©2008 BAE Systems.
Organizational Systemic Causal Analysis & Resolution Group
Metrics Analysis Group
Feb-06
Nov-05
Aug-05
May-05
Mar-05
Dec-04
Sep-04
Jun-04
Mar-04
Dec-03
Sep-03
Jun-03
Mar-03
Dec-02
Sep-02
Jun-02
Mar-02
Dec-01
Sep-01
Jun-01
Mar-01
Dec-00
Sep-00
Jun-00
Mar-00
Dec-99
Sep-99
Jul-99
Apr-99
Oct-98
0.0
Jan-99
.75
7
Proof that Process Improvement Works!
Scenario 2) Project Y
Project Y (a sustaining/maintenance effort) has difficulty
implementing the standard development processes
which have been designed primary for the needs of fullscale development efforts.
Subsequently, 4 consecutive releases for Project Y are
measured and 7 of 8 data points (4 of 4 data points on
Productivity and 3 of 4 data points on Product Quality)
miss the Organizational Target Goals.
Still, improvement trends in both measures, as well as the
evident balancing of Productivity vs. Product Quality
performance, prove that Project Y is operating at High
Maturity!
©2008 BAE Systems.
Organizational Systemic Causal Analysis & Resolution Group
Metrics Analysis Group
8
Proof that Process Improvement Works!
COCOMO II Productivity Factor - Unit Cost – Project Y
1.75
Below Org Target Max Goal
Near Org Target Max Goal
1.50
Unit Cost - (Person Hours Per SLOC)
Slightly Above Org Target Max Goal
High Maturity Projects
May
Well Above Org Target Max Goal
Miss Target Goals But
Org Target Maximum Goal
Continuous Improvement Is
Their True Focus! And …
1.25
1.00
0.75
0.50
Org Target Max
.300
©2008 BAE Systems.
Organizational Systemic Causal Analysis & Resolution Group
Metrics Analysis Group
Feb-06
Nov-05
Aug-05
May-05
Mar-05
Dec-04
Sep-04
Jun-04
Mar-04
Dec-03
Sep-03
Jun-03
Mar-03
Dec-02
Sep-02
Jun-02
Mar-02
Dec-01
Sep-01
Jun-01
Mar-01
Dec-00
Sep-00
Jun-00
Mar-00
Dec-99
Sep-99
Jul-99
Apr-99
Oct-98
0.00
Jan-99
0.25
9
Proof that Process Improvement Works!
Post Delivery Defect Density – Project Y
6.0
Below Org Target Max Goal
Near Org Target Max Goal
5.0
Slightly Above Org Target Max Goal
Well Above Org Target Max Goal
PDDD ( Defects Per KSLOC)
Org Target Maximum Goal
4.0
… High Maturity Projects
Learn to Balance Cost &
Quality Indicators
Attempting to Optimize
Contract Performance
3.0
2.0
Org Target Max
1.0
©2008 BAE Systems.
Organizational Systemic Causal Analysis & Resolution Group
Metrics Analysis Group
Feb-06
Nov-05
Aug-05
May-05
Mar-05
Dec-04
Sep-04
Jun-04
Mar-04
Dec-03
Sep-03
Jun-03
Mar-03
Dec-02
Sep-02
Jun-02
Mar-02
Dec-01
Sep-01
Jun-01
Mar-01
Dec-00
Sep-00
Jun-00
Mar-00
Dec-99
Sep-99
Jul-99
Apr-99
Oct-98
0.0
Jan-99
.75
10
Proof that Process Improvement Works!
Scenario 3) Project Z
Project Z is a Large Scale Development program with
numerous overlapping development efforts and does
experience some difficulty implementing the company’s
standard development processes.
Initially, customer / project management pushback is high
due to perceived added cost of the more rigorous
development, review, measurement and analysis
processes but the engineering team is persistent in
implementing higher maturity practices.
Ultimately, Project Z starts to realize increased Process
Capability through better analyses of measurements and
appropriate corrective actions.
©2008 BAE Systems.
Organizational Systemic Causal Analysis & Resolution Group
Metrics Analysis Group
11
Proof that Process Improvement Works!
COCOMO II Productivity Factor - Unit Cost - Project Z
1.75
Below Org Target Max Goal
Near Org Target Max Goal
1.50
Slightly Above Org Target Max Goal
Well Above Org Target Max Goal
Unit Cost - (Person Hours Per SLOC)
1.25
Org Target Maximum Goal
As High Maturity Projects
Stabilize Performance …
1.00
0.75
0.50
Org Target Max
.300
0.25
©2008 BAE Systems.
Organizational Systemic Causal Analysis & Resolution Group
Metrics Analysis Group
Feb-06
Nov-05
Aug-05
May-05
Mar-05
Dec-04
Sep-04
Jun-04
Mar-04
Dec-03
Sep-03
Jun-03
Mar-03
Dec-02
Sep-02
Jun-02
Mar-02
Dec-01
Sep-01
Jun-01
Mar-01
Dec-00
Sep-00
Jun-00
Mar-00
Dec-99
Sep-99
Jul-99
Apr-99
Jan-99
Oct-98
0.00
12
Proof that Process Improvement Works!
Post Delivery Defect Density - Project Z
6.0
Below Org Target Max Goal
Near Org Target Max Goal
5.0
Slightly Above Org Target Max Goal
Well Above Org Target Max Goal
Org Target Maximum Goal
PDDD ( Defects Per KSLOC)
4.0
… Continuous Improvement
Becomes An Engrained
Behavior !
3.0
2.0
Org Target Max
1.0
.75
©2008 BAE Systems.
Organizational Systemic Causal Analysis & Resolution Group
Metrics Analysis Group
Feb-06
Nov-05
Aug-05
May-05
Mar-05
Dec-04
Sep-04
Jun-04
Mar-04
Dec-03
Sep-03
Jun-03
Mar-03
Dec-02
Sep-02
Jun-02
Mar-02
Dec-01
Sep-01
Jun-01
Mar-01
Dec-00
Sep-00
Jun-00
Mar-00
Dec-99
Sep-99
Jul-99
Apr-99
Jan-99
Oct-98
0.0
13
Proof that Process Improvement Works!
Scenario 4) The Organization
As development and maintenance efforts like Projects X, Y
and Z adopt a more rigorous and uniform set of
engineering development and project management
processes, the Organization as a whole will also start to
realize increased Process Capability and Process
Stability in its Organizational Performance Measures.
Additionally, a standardized company-wide set of
engineering development and project management
processes reduces the impact of the “Learning Curve”
for engineering and support personnel who transfer
between projects, programs and lines of business.
©2008 BAE Systems.
Organizational Systemic Causal Analysis & Resolution Group
Metrics Analysis Group
14
Proof that Process Improvement Works!
As Organizational Process Capabilities Improve …
Mean Effort to Resolve DRs (Pri 1,2,3)
50
Attained SWCMM Level 4
Maturity Rating
Average Effort (Engineering Hours)
40
70 %
Reduction
over 4 Years!
Attained SWCMM Level 5
Maturity Rating
37.79
Alarming
Trend!
29.57
30
Attained CMMI
Level 5 Maturity
Rating
20
18.97
18.90
15.75
12.86
10
11.78
10.28
9.49
7.69
7.89
2003Q4
2004Q1
7.02
11.27
10.33
9.77
2005Q2
2005Q3
7.81
0
2002Q1
©2008 BAE Systems.
2002Q2
2002Q3
2002Q4
2003Q1
2003Q2
2003Q3
2004Q2
2004Q3
2004Q4
2005Q1
Organizational Systemic Causal Analysis & Resolution Group
Metrics Analysis Group
2005Q4
15
Proof that Process Improvement Works!
… And Process Variances Stabilize…
Variance in Effort to Resolve DRs (Pri 1,2,3)
Attained SWCMM Level 5
Maturity Rating
16000
Variance - Effort To Resolve DRs
14000
12000
10000
Attained SWCMM Level 4
Maturity Rating
Reduced variance in
process and product
measurements enables
more meaningful
analyses and allows
sustainable process
changes!
Attained CMMI
Level 5 Maturity
Rating
8000
6000
4000
2000
0
2002Q1
©2008 BAE Systems.
2002Q2
2002Q3
2002Q4
2003Q1
2003Q2
2003Q3
2003Q4
2004Q1
2004Q2
2004Q3
2004Q4
2005Q1
Organizational Systemic Causal Analysis & Resolution Group
Metrics Analysis Group
2005Q2
2005Q3
2005Q4
16
Proof that Process Improvement Works!
Organizational Performance Can Be Optimized!
Priority 1- 3 DR Analysis - Average Effort to Resolve
New Level of
Performance
Requires
Rebaselining
Avg Labor Hours :
SW CMM
Levels 4 & 5
Achieved
CMMI Level 5
Achieved
Operating at
SW CMM
Level 3
CY 2001
CY 2002
CY 2003
Jan - Apr 2004
Period of Performance
Pri 1 DR Avg Effort
©2008 BAE Systems.
Pri 2 DR Avg Effort
Pri 3 DR Avg Effort
Pri 1-5 DR Avg Effort
Organizational Systemic Causal Analysis & Resolution Group
Metrics Analysis Group
Power (Pri 1-5 DR Avg Effort)
17
Proof that Process Improvement Works!
All Inspections - Monthly Equivalent Critical Code Defect Discovery Counts
Defects Discovered Per Month
4,000
SW-CMM
Level 4
SW-CMM
Level 5
Checklist Inspections
CMMI
Level 5
Formal Inspections
3,000
Total Inspection Defects Discovered
SW-CMM
Level 3
2,000
1,000
Nov-05
Sep-05
Jul-05
May-05
Mar-05
Jan-05
Nov-04
Sep-04
Jul-04
May-04
Mar-04
Jan-04
Nov-03
Sep-03
Jul-03
May-03
Mar-03
Jan-03
Nov-02
Sep-02
Jul-02
May-02
Mar-02
Jan-02
Nov-01
Sep-01
Jul-01
May-01
Mar-01
All Inspections - Monthly Average Defect Discovery Per Inspection
50.0
Checklist Inspection Average
Formal Inspection Average
40.0
Total Defects Discovered
30.0
20.0
10.0
©2008 BAE Systems.
Organizational Systemic Causal Analysis & Resolution Group
Metrics Analysis Group
Nov-05
Sep-05
Jul-05
May-05
Mar-05
Jan-05
Nov-04
Sep-04
Jul-04
May-04
Mar-04
Jan-04
Nov-03
Sep-03
Jul-03
May-03
Mar-03
Jan-03
Nov-02
Sep-02
Jul-02
May-02
Mar-02
Jan-02
Nov-01
Sep-01
Jul-01
May-01
Mar-01
0.0
Jan-01
Average Defects Discovered Per Inspection
Jan-01
0
18
Proof that Process Improvement Works!
Post Delivery Defect Density Reduction Goal (2001-2005)
1.750
Recorded Quarter Defect Density
75 %
Reduction
over 5 Years!
Max. Post-Delivery Defect Density Reduction Goal
1.500
Post-Delivery Defect Density (Defects/KSLOC)
4-Quarter Moving Average
1.270
1.250
1.050
1.050
1.000
0.900
0.750
0.500
0.250
1.27
0.41
0.32
0.66
0.24
0.86
0.76
0.71
0.90
0.69
0.33
0.41
0.72
0.31
0.73
0.34
0.75
0.32
0.16
©2008 BAE Systems.
05
20
05
Q
4
20
05
Q
3
20
05
Q
2
20
04
Organizational Systemic Causal Analysis & Resolution Group
Metrics Analysis Group
Q
1
20
04
Q
4
20
04
Q
3
20
04
Q
2
20
03
Q
1
20
03
Q
4
20
03
Q
3
20
03
Q
2
20
02
Q
1
20
02
Q
4
20
02
Q
3
20
02
Q
2
20
01
Q
1
20
01
Q
4
20
Q
3
20
Q
2
B
as
el
in
e
01
0.000
19
Proof that Process Improvement Works!
Improved Defect Detection Enables …
Defect Detection Process Capability Comparison
Pre-1995 (w/ Ad Hoc Processes)
1995-1999 (w/ Peer Reviews)
1999-2003 (w/ Fagan Process)
2004-present (w/ Multiple Review
Processes)
Defect Detection Densities
direction of change
Requirements
Analysis
Prelim. Design
Detailed Design
Code/Unit Test
System
Integration
FAT
Post-Delivery
(first year)
Software Development Phases
©2008 BAE Systems.
Organizational Systemic Causal Analysis & Resolution Group
Metrics Analysis Group
20
Proof that Process Improvement Works!
… Improved Defect Prevention!
Defect Prevention Process Capability Comparison
Initial DP Activities Profile
Defect Detection Densities
direction of change
Progressing DP Activities Profile
Requirements
Analysis
Prelim. Design
Advanced DP Activities Profile
Detailed Design
Code/Unit Test
System
Integration
FAT
Post-Delivery
(first year)
Software Development Phases
©2008 BAE Systems.
Organizational Systemic Causal Analysis & Resolution Group
Metrics Analysis Group
21
Proof that Process Improvement Works!
When the Defect Well Starts to Dry Up …
Projects that have cut their teeth on quantitative management of defect
injections/detections must ultimately look for new ways to apply their
statistical tool set and quantitative management skills.
Integration & Testing Reliability Prediction is one such advanced model.
Reliability Prediction Model Calibration
Window 1
[Attem pt to Cover Diam onds w ith Boxes]
y ≈ λe-θt
Reliability Prediction Model Calibration
Window 2
[Attem pt to Minim ize Blue Area Under Curve]
Reliability Prediction Model Parameters
Weekly Reliability Threshold
Calculated Lambda - Method 1
Calculated Lambda - Method 2
Calculated Lambda - Method 3
Calibration Lambda
Calibration Theta
0.0500
0.7061
1.4123
1.0147
1.3500
0.0060
Sum of Squared Differences
0.0051
©2008 BAE Systems.
135
60
Lambda/Theta
Spin Controls
Organizational Systemic Causal Analysis & Resolution Group
Metrics Analysis Group
22
Proof that Process Improvement Works!
Key Take-Aways!
1. Starting Down the Process Improvement Path Can Be
Both Scary & Risky. Danger Lurks Around Every Bend.
Choose a Pathfinder That You Can Trust!
2. Every Perceived Failure Can Also Be Viewed as a
Potential Opportunity for Improvement. Think Positively!
3. Limiting Process Variance is the Key to Any Successful,
Repeatable, Reliable and Sustainable Process
Improvement Implementation. Don’t Just Focus on
Mean Values!
4. Institutionalization Requires Time & Energy. Build Up &
Maintain Momentum. Starting & Stopping & Restarting
Again Is A Very Poor Use of Project Funds!
©2008 BAE Systems.
Organizational Systemic Causal Analysis & Resolution Group
Metrics Analysis Group
23
Proof that Process Improvement Works!
Questions?
©2008 BAE Systems.
Organizational Systemic Causal Analysis & Resolution Group
Metrics Analysis Group
24
Proof that Process Improvement Works!
Contact Info
•
©2008 BAE Systems.
Kevin Domzalski
•
•
•
BAE Systems – C3I Systems
10920 Technology Place (MZ: 606-PI)
San Diego, CA 92127
•
Email: [email protected]
Organizational Systemic Causal Analysis & Resolution Group
Metrics Analysis Group
25
Proof that Process Improvement Works!
About the Presenter
•
Kevin Domzalski is a seasoned member of the Process Improvement
Group at BAE Systems (C3I Systems) headquartered in San Diego,
California, where he currently fulfills the role of Metrics Analysis Group Lead
•
Kevin joined the company in 1983 (or at least one of the precedent
companies that have now been formed into BAE Systems, Inc.) but took a
5-year hiatus between 1993 and 1998 to work as an automotive industry
engineering consultant
•
Kevin has developed and teaches many company courses as well as the
Systems Engineering Software Overview course at the University of
California at San Diego (UCSD) Extension Studies Program as a adjunct
faculty member
•
In addition, Kevin was awarded the PSM Users Group’s “Most Successful
PSM Implementation” award in July of 2005 having been nominated by
David Card, co-author of PSM and the ISO/IEC 15939 Standard
©2008 BAE Systems.
Organizational Systemic Causal Analysis & Resolution Group
Metrics Analysis Group
26
Proof that Process Improvement Works!
Acronyms
CMM®
CMMIsm
COCOMO
ISO/IEC
MAG
OSCAR
PSM
UCSD
©2008 BAE Systems.
Capability Maturity Model
Capability Maturity Model Integration
Constructive Cost Model
International Organization for Standardization /
International Electrotechnical Commission
Metrics Analysis Group
Organizational Systemic Causal Analysis &
Resolution
Practical Software (& Systems) Measurement
University of California San Diego
Organizational Systemic Causal Analysis & Resolution Group
Metrics Analysis Group
27