Transcript Predictive Models to Achieve Business Results

Predictive Models to Achieve
Business Results
Place your image on
top of this gray box.
If no graphic i
applicable, delete gray box
and notch-out
behind gray box, from
19th International Forum on
the Title Master
COCOMO and Software Cost Modeling
Cvetan Redzic, Michael Crowley, Nancy Eickelmann, Jongmoon Baik
Motorola, Inc.
October 26, 2004
1
Outline
• Overview
• Business Goals
• Models Used
– COQUALMO
– CoQ-DES
– MotoROI
• Primary Model Inputs
– CMM
– Life Cycle Scope
– PCE / PSE
• Results
– Cost
– Quality
2
Business Goal – Improved Customer Satisfaction
+
Satisfier
Features
3
Delighters
Attractive
2
Quality
1
Must Be
SW Quality
Type of needs
-
1.
2.
3.
Basic Expectations (Must Be)
Satisfier - Features
Delighters (Attractive)
Kano Analysis
3
Cause & Effect Diagram
Improved
Customer
Satisfaction
4
Integrating Predictive Models
• Models Used
– COQUALMO
– CoQ-DES
– MotoROI
5
COQUALMO
6
Combined COQUALMO Injection Factors
7
CoQ-DES
8
CoQ-DES Simulation
9
MotoROI
Input
Data
Validate
Inputs
Data
Valid?
No
Exit
Yes
Analyze
Consequences of
Software Failure
Analyze
Likelihood of
Software Failure
Risk Analysis &
Classification
Potential Maximum Return & ROI
Expected Return & ROI
Output Report
10
MotoROI - DOORS ROI Analysis
DOORS – ROI Analysis
CMM Level
Total SLOC
Scope of SLOC
Software Budget
Investment Budget
Investment Effectiveness
Allocated investment
Scope of Investment
Potential Maximum Return
Potential Maximum Return/Scope
Potential Maximum ROI
Expected Return
Expected ROI
% COPQ Savings
Report 1
Report 2
Report 3
Report 4
5
9,780,700
9,780,700
$37,209,669
$400,000
50%
$400,000
Requirements
$5,953,547
$2,232,580
1:1.6
$120,000
1:0.3
2%
3
9,780,700
9,780,700
$37,209,669
$400,000
100%
$400,000
Requirements
$13,023,384
$4,837,257
1:3.5
$520,000
1:1.3
4%
3
9,780,700
9,780,700
$37,209,669
$400,000
75%
$400,000
Requirements
$13,023,384
$4,837,257
1:3.5
$390,000
1:0.97
3%
3
9,780,700
9,780,700
$37,209,669
$400,000
50%
$400,000
Requirements
$13,023,384
$4,837,257
1:3.5
$260,000
1:0.7
2%
11
Model Integration - Primary Model Inputs
• CMM
• Life Cycle
• PCE / PSE
12
CMM – Process Maturity
Prevention Appraisal
Int Failure
Ext Failure TCoSQ
60
•
COQUALMO
– PMAT (process
maturity has the
greatest +/-impact) on
injection rates
Cost as a Percent of Development
50
40
• CoQ-DES
– Not Used directly but
is inherent in
organizational
calibration
30
20
• MotoROI
10
0
1
2
4
3
SEI CMM Level
5
– Process maturity as
represented by the
cost of quality/cost of
poor quality financial
structure is a primary
factor.
Knox Theoretical Model of TCOQ
(About 50% at CMM Level 3)
13
Life Cycle
Requirement
System
Test
Design
Component/Integration
Test
Testing
Inspections
Implementation
Unit Test
Code
• COQUALMO
– Req., Des., Imp., and Code
• CoQ-DES
– Full Life Cycle
• MotoROI
– Full Life Cycle or Individual Phases
14
PCE and PSE
Phase Containment Effectiveness
& Phase Screening Effectiveness
•
COQUALMO
– PCE and PSE as
evidenced by
injection and
removal rates
• CoQ-DES
– PCE and PSE as
evidenced by
injection and
removal rates
• MotoROI
– PCE for DP or PSE
for technology
effectiveness
15
Measuring and Monitoring Results
• Quality
• Cost
16
Quality - Sources of Variation
Actual vs. COQUALMO Estimate
Defect Injection in REQ, DESIGN & CODE
REQ
DESIGN
COQUALMO
For Release with about 100 Delta
KLOC, no significant difference
estimates & actuals in DI & DR
For large size Release over 100 Delta
KLOC, there is significant
difference b/w estimates & actuals
in DI & DR for Code
CODE
Average
Edge
Defect Removal in REQ, DESIGN & CODE
REQ
Calculated Chi-Square
Value
0.19
Chi-Square (2;0.05)
5.99
DES
0.57
CODE
11.97
REQ
Significance
No
No
Yes
DESIGN
COQUALMO
Average
CODE
Edge
17
Quality - Sigma Level
From PCE, SRE & CRUD data
Sigma Level:
Sigma Level by Release - As Is
Sigma Level
Defects per Million
Opportunities
DPMO = 1M * D/(N*O)
D = 2464 HS Faults
(from PCE)
N = 139,595 Delta LOC
DPMO = 1M * 2464/139,595
DPMO = 17651 - 3.61 s
GSR4
GSR4.1
GSR5.X
SL HS Faults
GSR6
H2
SL SRE PRs
GSR7
EDGE
SL CRUD
What is Sigma Level from release
perspective ?
Relatively stable across the
releases
GSR8
Stable processes
Need Leap improvement:
SEI CMM Level 5 TCM
18
Quality - SRE Goal Setting
19
Quality: As-Is Process
Defect Injection per KLOC - As Is
REQ
IDS
HLD
Average
LLD
LB
Cum. Defect Injection by Phase
CODE
REQ
IDS
UB
IDS
HLD
LLD
CODE
Average
PT
LB
FT
UB
LLD
Cum. Average
Defect Removal per KOLC by Phase - As Is
REQ
HLD
ST
LB
CODE
UB
Cum. Defect Removal per KOLC by Phase - As Is
P-R
CRUD
REQ
IDS
HLD
LLD
CODE
Cum. Average
PT
LB
FT
ST
P-R
CRUD
UB
20
Quality - Rayleigh Model Analysis
21
Quality - Impact of Tactical Changes
Forecast: CRUDoutcome
1,000 Trials
Frequency Chart
998 Display ed
.028
28
.021
21
.014
14
.007
7
.000
0
22
23
25
26
27
Certainty is 95.00% from 23 to 26
Monte-Carlo simulation,
to include uncertainty &
risks In the expert based
opinion
22
Quality - New Process Baseline
Defect Injection per KLOC by Phase - Should Be
REQ
IDS
HLD
LLD
Cum. Defect Injection per KLOC by Phase - Should Be
CODE
REQ
Average
LB
UB
IDS
HLD
LLD
CODE
Average
PT
FT
UB
LB
HLD
Cum. Average
ST
P-R
LLD
LB
CODE
UB
Cum. Defect Removal per KOLC by Phase - Should Be
Defect Removal per KOLC by Phase - Should Be
REQ
IDS
CRUD
REQ
IDS
HLD
LLD
CODE
Cum. Average
PT
FT
LB
UB
ST
P-R
CRUD
23
Cost - Vital X Monthly Review Charts
SLIM
GSR8 Program Re v ie w - GSR8
FTE Staff
8
8
6
6
Cum Eff SLOC
5
4
3
2
8
8
140
6
6
5
4
3
2
200
120
ppl
80
100
60
40
50
SLOC (thousands)
150
100
20
0
Jul
'03
1
Sep
3
Nov
5
Jan
'04
7
Mar
9
11
May
Jul
13
15
17
19
21
Sep Nov Jan
Mar May
'05
0
Jul
'03
1
Sep
3
Nov
5
Jan
'04
Cum All De fe cts
8
6
5
7
Mar
9
11
May
Jul
13
15
17
19
21
Sep Nov Jan
Mar May
'05
All De fe cts
4
3
2 1400
8
6
5
4
3
2 250
1200
200
1000
AD
600
150
CAD
800
100
400
50
200
0
0
1
3
5
7
9
11 13 15 17 19 21
Jul Sep Nov Jan Mar May Jul Sep Nov Jan Mar May
'03
'04
'05
Current Plan
Actuals
1
3
5
7
9
11 13 15 17 19 21
Jul Sep Nov Jan Mar May Jul Sep Nov Jan Mar May
'03
'04
'05
Current Forecast
Green Control Bound
Yellow Control Bound
Project: GSR8
24
Quality - Vital X Monthly Review Charts
Fault Injection & Removal vs. Baselines
Cum. Defect Injection by Phase
Defect Injection per KLOC by Phase - Should Be
REQ
IDS
HLD
Average
LLD
LB
CODE
UB
REQ
IDS
H2
IDS
HLD
LLD
Average
CODE
GSR8
PT
FT
UB
LLD
Cum. Average
LB
CODE
UB
Cum. Defect Removal per KOLC by Phase
Defect Removal per KOLC by Phase
REQ
HLD
ST
LB
P-R
CRUD
REQ
IDS
HLD
Cum. Average
LLD
CODE
LB
PT
Cum. GSR8
FT
ST
P-R
CRUD
UB
25
Quality - Vital X Monthly Review Charts
SRE
26
CRUD Goal Tracking
CRUD - Projected vs. Actual
Jan-04 Feb04
Mar- Apr-04 May- Jun-04 Jul-04 Aug04
04
04
Cum. CRUD
Sep04
Oct04
Nov04
Dec04
Goal
27
Summary
• Integrating predictive models provides multiple
views of project quality, cost and schedule
issues.
• More accurate predictions of defect injection are
possible
• More accurate predictions of defect removal are
possible
• More accurate predictions of overall staffing and
project cost are possible
28