Transcript Slide 1

Identifying and Using a
Project’s Key Subprocess
Metrics
Jeff S. Holmes
BTS Fort Worth
Everyone Loves a Hero
Heroes Come Through!
 Firemen – Saves Baby in Burning House
 Policemen – Catches Bad Guy
 Athlete – Hits Game Winning Homerun
Software Engineering Heroes!
 All night coding!
 Debugging over the weekend!
 THIS SHOULD NOT BE THE NORM!
How To Minimize “Fire Drills”?
Preventative Actions
Proper wiring can prevent fires.
Locking your car can prevent theft.
Don’t get behind in the ball game.
Understand project status earlier.
Metrics, metrics, everywhere…
But What is Really Important?

Customer Wants
– Functionality
– Zero Defects
– On Time


What software metrics
map to these?
How can we optimize
these outputs?
BTS Fort Worth Approach





Selected DMAIC to Improve Process
Identified Project with Two Years of Data
Performed Statistical Analysis
Conducted Pilot
Currently in “Control” Phase
DMAIC : Define
Identify “what is important”
 BTS FW Monitors

(KLOC/Hour)*
 Quality (Post Release Defects/KLOC)
 Schedule Adherence
 Productivity

These are BTS FW “Big X’s”
DMAIC : Measure
Requirements
Resources

Software Development Life Cycle
The “Simple” View
Perfect
Software!
DMAIC : More Details
Requirements
Resources
Requirements
Perfect Requirements
Resources
Design
Perfect Design
Perfect Models
Resources
Perfect Code
Resources
Perfect Design
Perfect Models
Perfect Code
Code
Test
Perfect
Software!
DMAIC : Subprocess Identification

BTS FW Identified Following Subprocesses
 Planning
Phase
 Requirements Phase
 Design Phase
 Code Phase
 Test Phase
 Release Phase
 Code Inspections
DMAIC : Measured Data

BTS FW Uses Following Data:
 # Requirements
 # Developers on the project
 % Time in Planning
 % Time in Requirements
 % Time in Design
%
%
%
(Resources)
Time in Code
Time in Test
Time in Release
 Requirements Churn
 Actual Size (KLOC)
 Avg Defect Detection Rate (DDR) in Code Inspection
DMAIC : Data Sources

BTS FW Data
 DOORs for Requirements
 Project Plans # Developers
 ClearCase for Code Size
 Inspection
and % Times
Database for DDR
DMAIC : BTS Subprocess Metrics
Resources
Time in Phase
Planning
Req Count
Time in Phase
Requirements
Req Churn
Time in Phase
Design
Req Churn
KLOC
Time in Phase
Code
DDR
Code Inspections
Test
Release
Time in Phase
Time in Phase
DMAIC : BTS Subprocess Metrics
Resources
Time in Phase
Planning
Productivity (KLOC/Hour)
Req Count
Time in Phase
Requirements
Quality (PR Defects/KLOC)
Req Churn
Time in Phase
Design
Schedule Adherence
Req Churn
KLOC
Time in Phase
Code
DDR
Code Inspections
Test
Release
Time in Phase
Time in Phase
DMAIC : BTS FW Analysis

Project Data

Used Step-wise Linear Regression to
• 8 Releases since 2002
• Similar work
• “Stable” team
• Identify statistically significant factors
• Develop prediction formulas for “Big X’s”
CAUTION !!
The following slides contain statistics that
could be hazardous to your health!
Persons who suffer from narcolepsy or
“statisticitis” should consider leaving the
room.
DMAIC : Stepwise Linear
Regression


Describes the relationship between one
'predicted' variable and 'predictor'
variables
Goal – get the simplest equation with the
best predictive power for
• Productivity – KLOC/Hour
• Quality – Post Release Defect/KLOC
DMAIC : Standard Least Squares
Model accounts for
99.82% of variance.
DMAIC : Significant Effects
< 0.05 is
significant

Most significant effects
• % Time in Req
• Average DDR
• Interaction between % Time in Code and
Average DDR
DMAIC : Standard Least Squares
Model accounts for
90.62% of variance.
DMAIC : Significant Effects
< 0.05 is
significant

Most significant effects
• % Time in Requirements
• Interaction between % Time in Requirements
and Requirements Churn
DMAIC : Statistically Significant











# Requirements
# Developers on the project
% Time in Planning
% Time in Requirements
% Time in Design
% Time in Code
% Time in Test
% Time in Release
Requirements Churn
Actual Size (KLOC)
Avg Defect Detection Rate (DDR) in Code Inspection
DMAIC : Key Subprocess Metrics
Resources
Time in Phase
Planning
Productivity (KLOC/Hour)
Req Count
Time in Phase
Requirements
Quality (PR Defects/KLOC)
Req Churn
Time in Phase
Design
Schedule Adherence
Req Churn
KLOC
Time in Phase
Code
DDR
Code Inspections
Test
Release
Time in Phase
Time in Phase
DMAIC : Variation Analysis

Prediction formulas generated to identify:

NOTE: Prediction formula uses all effects from the
models, not just the significant ones. Formula added in
Percent Planning.
• Good and bad variance
• Most significant factors
DMAIC : Factor Weighting
Metric
10% Deviation in Each Subprocess Area
% Plan
10%
0
0
0
0
% Req
0
10%
0
0
0
% Code
0
0
10%
0
0
Req Churn
0
0
0
10%
0
Avg DDR
0
0
0
0
10%
Productivity
0
3%
31%
0
9%
Quality
14%
40%
0
6%
0
Cycle Time
0
3%
24%
0
9%
DMAIC : Factors’ Effects
Metric
LOC/Hr Effect
PR Defect/KLOC Effect
% Planning Time
N/A
More time = Less
defects/KLOC
% Req Time
More Time = Less
KLOC/Hr
Less time = More
defects/KLOC
% Code Time
More Time = Less
KLOC/Hr
N/A
Code Insp. DDR
Higher DDR = Less
KLOC/Hr
N/A
2SR Req. Churn
N/A
Less Churn = Less
defects/KLOC
DMAIC : BTS FW Limits
Metric
Average
Std Dev
Lower
Limit
Upper
Limit
% Planning Time
18 %
8%
10 %
26 %
% Req Time
15 %
8%
7%
23 %
% Code Time
21 %
12 %
11 %
33 %
Code Insp. DDR
1.89
.5
1.39
2.39
2SR Req. Churn
1.76
1.89
0
3.65
Green limit indicates direction a metric can deviate
from the average and have desired results.
Red indicates direction of undesired results.
Subprocess Metrics Notes


Initial Data Left Much to be “Desired”
Despite Poor Data, the Analysis
Identified:
• Which Metrics and Processes Are Significant
• Prediction Formulas Based on Project’s Data
• Insight into Factors’ Effects
• Limits for Monitoring the Factors
DMAIC : Pilot Confirmation




Used prediction formulas on other projects
Compared project actuals vs. predicted.
Used historical data from 5 projects.
Unable to compare predicted quality versus
actual. (Predicted LOC/Hr)
•

These projects have not been in field long enough for
CRUD to stabilize.
Interesting results found on predicted LOC/Hr.
DMAIC : Predicted vs. Actual
LOC/Hr
% Plan
% Req
% Code
Inspection DDR
Requirements Churn-2SR
Pred LOC/Hr
Pred PR Def/KLOC
Actual LOC/Hr
LOC/Hr Delta
Project A
Project B
Project C
Project D
Project E
0%
18%
-65%
10%
17%
-28%
10%
-97%
4%
17%
-348%
20%
0%
43%
25%
1.2
1.2
0
1.51
1.89
1
1.81
0
0
0
92.673688
4.789660
25.540537
2.288774
2.418825
7.7196696 1.33169549
28.1019536
3.39281 0.4010827
1.21
99%
1.79
63%
0.829
97%
2.039
11%
•Projects A, B, and C projects had huge deviations.
• Projects D and E were within 20%.
1.945
20%
DMAIC : Improve
Performance?



So what?
How do you use this information?
Does Project Management have confidence
in this analysis?
DMAIC : Applying Analysis


More emphasis on statistically significant
activities
Resulting in

Increased Productivity



On-Time Delivery
Desired Functionality Delivered
Improved Quality
DMAIC : Agile Processes

BTS FW Adopted Agile Practices

Iterative Development



Pair Programming


Prioritizes Requirements
Negates Requirements Churn
Optimizes Coding and Inspection Time
Minimal Documentation

Moves effort from non-statistical activities.
DMAIC : Agile Pilot Results

Productivity



Inspection Defect Detection Rate



0.00291 KLOC/Hr
20% improvement from 0.002399
1.18 Defects/Hr Detected
48% improvement from 0.8
Quality

0 Post Release Defects!
DMAIC : Agile Pilot Results

Customer Wants



Functionality – All functionality delivered
Zero Defects – No customer found defects
On Time – Product delivered 6 months early!
DMAIC : Agile Monitoring



Monitor Iterations Not Phases
Refactoring Subprocess Monitoring
Two Agile projects in-work now
DMAIC : Agile Monitoring

Monitoring





LOC per week
Defects caught per week by inspection
Defects caught per week by test
Time spent per week
Ratio of new work to correction work.
DMAIC : Agile Metrics
% Fault Work
20%
18%
16%
14%
12%
10%
8%
6%
4%
2%
0%
7/31/2005
8/7/2005
8/14/2005
8/21/2005
8/28/2005
Summary




Save your “heroes” for real crises.
Understand subprocesses
Monitor subprocesses
Seek to optimize key subprocesses
Recommendations



Examine current project data, it could
prove to be very valuable!
Improve data capture on important data.
Use the data as a guideline, but
experience can never be discounted.
THANK YOU!
Jeff S. Holmes
Principal Staff Software Engineer
Motorola Six Sigma Black Belt
Fort Worth BTS Development Team
Fort Worth, TX
817-245-7053
[email protected]