Transcript Document 7196378

Towards COSYSMO 2.0
Future Directions and Priorities
CSSE Annual Research Review
Los Angeles, CA
March 17, 2008
Garry Roedler
Gan Wang
Jared Fortune
Ricardo Valerdi
Agenda
• Context setting
• Discussion on COSYSMO 2.0
improvements
• Prioritization exercise
8:15 – 9:00 am



Introductions
overview of the model
summary of COSYSMO 2.0 improvements
Garry Roedler
9:00 – 10:00 am


Reuse (overview only)
Integration between SwE & SysE
Jared Fortune,
Gan Wang
10:00 – 10:30 am
Break
10:30 – 11:00 am


Assumption of linearity in cost drivers
Cost drivers vs. scale factors
Gan Wang
11:00 – 12:00 pm


Recursive levels in the design
Risk modeling (overview only)
Ricardo Valerdi,
Garry Roedler
12:00 – 1:00 pm
Lunch
1:00 – 1:45 pm

Best practice guidance
Garry Roedler
1:45 – 2:30 pm

Modeling organizational factors in space
systems
Darryl Webb
2:30 – 3:00 pm
Break
3:00 – 4:00 pm


Cost driver impact survey results from Oct ‘07
Discussion & wrap-up
Gan Wang,
Garry Roedler
4:00 – 5:00 pm

Joint meeting w/ SoS cost estimation group
Jo Ann Lane
Context setting
How is Systems Engineering Defined?
•
•
•
Acquisition and Supply
•
– Supply Process
– Acquisition Process
Technical Management
•
– Planning Process
– Assessment Process
– Control Process
System Design
– Requirements Definition Process
– Solution Definition Process
Product Realization
– Implementation Process
– Transition to Use Process
Technical Evaluation
– Systems Analysis Process
– Requirements Validation Process
– System Verification Process
– End Products Validation Process
EIA/ANSI 632, Processes for Engineering a System, 1999.
Note: The requirements of EIA/ANSI 632 are addressed in ISO/IEC 15288, which was also used as a
Source for consistent definition in COSYSMO.
COSYSMO Origins
Current SE Standards
EIA-632 ISO/IEC 15288
Systems Engineering (SE)
(Warfield 1956,
1950
EIA 1999,
ISO/IEC 2002)
2000
Software Cost Modeling
(Boehm 1981)
COSYSMO
1980
(Humphrey 1989)
SW-CMM®
SE-CMM ®
CMMI®
SECM
1990
2000
*CMM and CMMI are registered trademarks of Carnegie Mellon University
Warfield, J. N., Systems Engineering, United States Department of Commerce PB111801, 1956.
Boehm, B. W., Software Engineering Economics, Prentice Hall, 1981.
Humphrey, W. Managing the Software Process. Addison-Wesley, 1989.
EIA/ANSI 632, Processes for Engineering a System, 1999
ISO/IEC 15288, System Life Cycle Processes, 2002.
COSYSMO Data Sources
Boeing
Integrated Defense Systems (Seal Beach, CA)
Raytheon
Intelligence & Information Systems (Garland, TX)
Northrop Grumman
Mission Systems (Redondo Beach, CA)
Lockheed Martin
Transportation & Security Solutions (Rockville, MD)
Integrated Systems & Solutions (Valley Forge, PA)
Systems Integration (Owego, NY)
Aeronautics (Marietta, GA)
Maritime Systems & Sensors (Manassas, VA;
Baltimore, MD; Syracuse, NY)
General Dynamics
Maritime Digital Systems/AIS (Pittsfield, MA)
Surveillance & Reconnaissance Systems/AIS
(Bloomington, MN)
BAE Systems
National Security Solutions/ISS (San Diego, CA)
Information & Electronic Warfare Systems (Nashua, NH)
SAIC
Army Transformation (Orlando, FL)
Integrated Data Solutions & Analysis (McLean, VA)
L-3 Communications
Greenville, TX
Modeling Methodology
3 rounds; > 60 experts
62 data points; 8 organizations
COSYSMO Scope
• Addresses first four phases of the system
lifecycle (adapted from ISO/IEC 15288)
Conceptualize
Develop
Oper Test Transition
to
& Eval
Operation
Operate,
Maintain,
or
Enhance
Replace
or
Dismantle
• Considers standard Systems Engineering
Work Breakdown Structure tasks (per
EIA/ANSI 632)
COSYSMO Operational Concept
# Requirements
# Interfaces
# Scenarios
# Algorithms
+
3 Adj. Factors
Size
Drivers
Effort
Multipliers
- Application factors
-8 factors
- Team factors
-6 factors
COSYSMO
Calibration
Effort
COSYSMO Model Form
E
PM NS

 14
 A    ( we,k  e,k  wn ,k  n ,k  wd ,k  d ,k )    EM j
 k
 j 1
Where:
PMNS = effort in Person Months (Nominal Schedule)
A = calibration constant derived from historical project data
k = {REQ, IF, ALG, SCN}
wx = weight for “easy”, “nominal”, or “difficult” size driver
 x = quantity of “k” size driver
Ex = represents diseconomies of scale
EM = effort multiplier for the jth cost driver. The geometric product results in an
overall effort adjustment factor to the nominal effort.
Size Drivers vs. Effort Multipliers
• Size Drivers: Additive, Incremental
– Impact of adding a new item inversely proportional to
current size
• 10 -> 11 rqts = 10% increase
• 100 -> 101 rqts = 1% increase
• Effort Multipliers: Multiplicative, system-wide
– Impact of adding a new item independent of current
size
• 10 rqts + high security = 40% increase
• 100 rqts + high security = 40% increase
Size Driver Weights
Easy
Nominal
Difficult
# of System Requirements
0.5
1.00
5.0
# of Interfaces
1.7
4.3
9.8
# of Critical Algorithms
3.4
6.5
18.2
# of Operational Scenarios
9.8
22.8
47.4
Cost Driver Clusters
UNDERSTANDING FACTORS
– Requirements understanding
– Architecture understanding
– Stakeholder team cohesion
– Personnel experience/continuity
COMPLEXITY FACTORS
– Level of service requirements
– Technology Risk
– # of Recursive Levels in the Design
– Documentation Match to Life Cycle Needs
OPERATIONS FACTORS
– # and Diversity of Installations/Platforms
– Migration complexity
PEOPLE FACTORS
– Personnel/team capability
– Process capability
ENVIRONMENT FACTORS
– Multisite coordination
– Tool support
Criteria
+ Matched driver polarity
+ Grouped by theme
+ Combined moderately
correlated parameters
Cost Driver Rating Scales
Very
Low
Low
Nominal
High
Very High
Requirements Understanding
1.87
1.37
1.00
0.77
0.60
3.12
Architecture Understanding
1.64
1.28
1.00
0.81
0.65
2.52
Level of Service Requirements
0.62
0.79
1.00
1.36
1.85
2.98
1.00
1.25
1.55
Migration Complexity
Extra
High
1.93
EMR
1.93
Technology Risk
0.67
0.82
1.00
1.32
1.75
2.61
Documentation
0.78
0.88
1.00
1.13
1.28
1.64
1.00
1.23
1.52
# and diversity of installations/platforms
1.87
1.87
# of recursive levels in the design
0.76
0.87
1.00
1.21
1.47
1.93
Stakeholder team cohesion
1.50
1.22
1.00
0.81
0.65
2.31
Personnel/team capability
1.50
1.22
1.00
0.81
0.65
2.31
Personnel experience/continuity
1.48
1.22
1.00
0.82
0.67
2.21
Process capability
1.47
1.21
1.00
0.88
0.77
0.68
2.16
Multisite coordination
1.39
1.18
1.00
0.90
0.80
0.72
1.93
Tool support
1.39
1.18
1.00
0.85
0.72
EMR = Effort Multiplier Ratio
1.93
Cost Drivers Ordered by Effort Multiplier Ratio (EMR)
Effort Profiling
Conceptualize
Develop
Operational
Test &
Evaluation
Transition to
Operation
Operate,
Maintain,
or Enhance
Life Cycle Phases/Stages
Technical
Management
System
Design
Product
Realization
Technical
Evaluation
EIA/ANSI 632
Acquisition &
Supply
Replace or
Dismantle
Impact
10 theses
Model
Academic Curricula
E

 14
PM NS  A    ( we,k  e,k  wn ,k  n ,k  wd ,k  d ,k )    EM j
 k
 j 1
Academic
prototype
Commercial Implementations
Policy & Contracts
Proprietary Implementations
SEEMaP
COSYSMO-R
Intelligence Community
Sheppard Mullin, LLC
SECOST
COSYSMO 2.0 Improvements
Recommended Improvements
(from user community)
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
Reuse
Integration of SwE & SysE estimation
Assumption of linearity in COSYSMO cost drivers
Effect of cost drivers and scale factors
Number of recursive levels of design
Risk modeling
Establishing best practice guidance
Consideration of SoS scope in COSYSMO Joint meeting
Estimation in Operation & Maintenance Phase
Deferred
Requirements volatility
1. Reuse
• Central question: What is the effect of reuse in
estimating systems engineering size/effort?
• Hypothesis: A COSYSMO reuse submodel will improve
the model’s estimation accuracy
• POC: Jared Fortune
• References
– Valerdi, R., Wang, G., Roedler, G., Rieff, J., Fortune, J.,
“COSYSMO Reuse Extension,” 22nd International Forum on
COCOMO and Systems/Software Cost Modeling, 2007.
2. Integration of SwE & SysE estimation
• Central question: What is the overlap between
COCOMO II and COSYSMO?
• Hypothesis: By identifying the WBS elements in
COSYSMO that overlap with the WBS in COCOMO II,
the systems engineering resource estimation accuracy
increases
• POC: Ricardo Valerdi
• References
– Valerdi, R., The Architect and the Builder: Overlaps Between
Software and Systems Engineering. (working paper)
3. Linearity in COSYSMO cost drivers
• Central question: How do we characterize the nonlinearity of cost drivers across the system life cycle?
• Hypothesis: Not all cost drivers have a constant impact
on systems engineering effort throughout the life cycle.
• POC: Gan Wang
• References
– Wang, G., Valerdi, R., Boehm, B., Shernoff, A., “Proposed
Modification to COSYSMO Estimating Relationship,” 18th
INCOSE Symposium, June 2008.
4. Effect of cost drivers and scale factors
• Central question: Can some of the cost drivers become
scale factors in the cost estimating relationship
calibrated by the new data set?
• Hypothesis: The current set of size and cost drivers are
too sensitive to small variations in rating levels.
• POC: Gan Wang
• References
– Wang, G., Valerdi, R., Boehm, B., Shernoff, A., “Proposed
Modification to COSYSMO Estimating Relationship,” 18th
INCOSE Symposium, June 2008.
Note from GJR: Potential relationship between this and improvement #7. Also, in reality, number of
recursive levels has a different impact on the effort than other cost drivers. This may be able to be
addressed by this improvement, by improvement #5, or could be a separate improvement depnding
on scope of each. Should be an item for discussion as we go through these.
5. Number of recursive levels of design
• Central question: How can the integration complexity of
system elements one layer below the system-of-interest
be operationalized?
• Hypothesis: The integration complexity of system
elements is a predictor of systems engineering effort.
• POC: John Rieff
• References
– Marksteiner, B., “Recursive Levels and COSYSMO”, October
2007. (working paper)
6. Risk Modeling
• Central question: How can risk associated with the
COSYSMO estimate be quantified?
• Hypothesis: The output generated by COSYSMO can be
quantified using probability distributions for better
assessment of the likelihood of meeting the estimate
• POC: John Gaffney (developer of COSYSMO-R)
• References
– Valerdi, R., Gaffney, J., “Reducing Risk and Uncertainty in
COSYSMO Size and Cost Drivers: Some Techniques for
Enhancing Accuracy,” 5th Conference on Systems Engineering
Research, March 2007, Hoboken, NJ.
7. Best practice guidance for use of Cost Drivers
• Central question: How can misuse of the COSYSMO
cost drivers be avoided?
• Hypothesis: By developing a best practice guide that
describes common pitfalls associated with COSYSMO
cost drivers, over-estimation can be reduced or avoided
• POC: Garry Roedler
• References
– COSYSMO User Manual
8. Consideration of SoS scope in COSYSMO
• Central question: How can COSYSMO be updated to
address system of systems effort estimation?
• Hypothesis: To be discussed in joint session
• POC: Jo Ann Lane
9. Estimation in Operation & Maintenance Phase
• Central question: How can we estimate systems
engineering effort in the Operate & Maintain phase?
• Hypothesis: Coverage of the Operate & Maintenance
phases will broaden to model’s life cycle coverage
• POC: Ricardo Valerdi
10. Requirements volatility
• Central question: How do we quantify the effects of
requirements volatility on systems engineering effort
throughout the life cycle?
• Hypothesis: Requirements volatility is a significant factor
for predicting systems engineering effort and can serve
as a leading indicator for project success
• POC: Ricardo Valerdi
• Feb 15, 2007 Workshop led by Rick Selby
– Identified critical success factors in: technical, product, process, people
– http://sunset.usc.edu/events/2007/ARR/presentations/RequirementsVol
atilityWorkshopSummaryARR2007.ppt
– Loconsole, A., Borstler, J., “An industrial case study on requirements
volatility measures,” 12th Asia-Pacific Software Engineering
Conference, 2005.
Prioritization Exercise
• Factors to Consider
–
–
–
–
Availability of data
Impact on total cost of ownership
Frequency of use
Compatibility with other models (i.e., COCOMO
family, PRICE-H, etc.)
– Addressal of future trends (Volatility, Uncertainty,
Scalability)
– Factor interactions
Recommended Improvements
(from user community)
1.
Reuse (completed and approved for V2.0 baseline – see
minutes from workshop at PSM User Conference)
2.
3.
4.
5.
6.
Integration of SwE & SysE estimation
Assumption of linearity in COSYSMO cost drivers
Effect of cost drivers and scale factors
Number of recursive levels of design
Risk modeling (completed and approved for V2.0 baseline –
see minutes from workshop at PSM User Conference)
7.
8.
9.
10.
Establishing best practice guidance
Consideration of SoS scope in COSYSMO Joint meeting
Estimation in Operation & Maintenance Phase
Deferred
Requirements volatility
Priority
Improvement
2
3
4
5
7
8
Availability
of Data
M
L
L
L
H
L
Impact on
TOC
H
H
H
H
M
M
Frequency
of Use
H
H
H
H
H
L
Compatible
with Models
H
L
L
M
L
M
Address
Trends
L
L
L
M
M
M
Factor
Interactions
L
H
H
L
H
L
M+
M
M
M
M+
L+
Priority
1
H
6
H
9
10
L
L