Transcript Requirements Volatility - Software Engineering II
University of Southern California
Center for Systems and Software Engineering
Requirements Volatility
Dr. Mauricio E. Peña February 19, 2014
1
University of Southern California
Center for Systems and Software Engineering
Agenda
• • • • • • • •
Introduction Definitions Requirements development and management Requirements metrics Causes of requirements volatility Impacts of requirements volatility Mitigation approaches Summary
2
University of Southern California
Center for Systems and Software Engineering
Importance of Understanding Requirements Volatility
• • •
Requirements volatility has been identified by numerous research studies as a risk factor and cost-driver of systems engineering projects 1 Requirements changes are costly, particularly in the later stages of the lifecycle process because the change may require rework of the design, verification and deployment plans 2 The Government Accountability Office (GAO) concluded in a 2004 report on the DoD ’s acquisition of software-intensive weapons systems that missing, vague, or changing requirements are a major cause of project failure 3
System developers often lack effective methods and tools to account for and manage requirements volatility
Source: 1- Boehm (1991), 2- Kotonya and Sommerville (1995), 3- GAO-04-393
3
University of Southern California
Center for Systems and Software Engineering
Requirements Volatility is Expected
• •
Changes to requirements are a part of our increasingly complex systems & dynamic business environment
– –
Stakeholders needs evolve rapidly The customer may not be able to fully specify the system requirements up front
–
New requirements may emerge as knowledge of the system evolves
–
Requirements often change during the early phases of the project as a result of trades and negotiations Requirements volatility must be anticipated and managed
Sources; Kotonya and Sommerville (1995); Reifer (2000)
4
University of Southern California
Center for Systems and Software Engineering
Requirements Definitions
“Requirements are intended to change vague desires into explicit and unambiguous statements of what the customers want ” 1 “A requirement is a capability that a system must supply or a quality that a system must possess in order to solve a problem or achieve an objective within the system ’s conceptual domain” 2 “A requirement is: (1) A condition or capability needed by a user to solve a problem or achieve an objective (2) A condition or capability that must be met or possessed by a system or system component to satisfy a contract, standard, specification, or other formally imposed documents ” 3
Source: 1- Weinberg, G. (1983), 2- Costello and Liu (1995); 3- IEEE (1990)
5
University of Southern California
Center for Systems and Software Engineering When is that Systems Engineer going to tell me the S/W requirements?
6
University of Southern California
Center for Systems and Software Engineering
Requirements Categories
1. Customer or operational requirements
effectiveness define the expectations of the system in terms of operational scenarios and environments, mission objectives, and measures of
2. Functional requirements
of tasks or activities specify what has to be done in terms
3. Performance requirements
define how well or to what extent the activities must be accomplished
4. Design requirements
dictate how to build products and what manufacturing processes to follow
5. Derived requirements
requirements; and are spawned from higher-level
6. Allocated requirements
: high level-requirements are partitioned into multiple lower-level requirements
Source: DoD Systems Management College (2001)
7
University of Southern California
Center for Systems and Software Engineering
Requirements Volatility Definitions
• • • •
Requirements volatility
is typically defined as the change in requirements (added, deleted, and modified) over a given time interval Also known as:
Requirements creep:
An increase in scope and number of system requirements
Requirements churn:
Instability in the requirements set – requirements are modified or re-worked without necessarily resulting in an increase in the total number of requirements
Source: MIL-STD-498
8
University of Southern California
Center for Systems and Software Engineering
Requirements Management
• •
The Systems Engineering Capability Model (EIA, 2002) defines five activities that influence the generation and evolution of requirements 1
– – – – –
Problem refinement Requirements analysis Requirements quality Requirements evolution and maintenance Feedback and verification Requirements definition and analysis are sometimes collectively referred to as
requirements engineering
:
–
The use of systematic and repeatable techniques to identify, document, develop and maintain a complete and consistent requirement set 2
Source: 1- EIA (2002); 2- Kotonya and Sommerville (1998)
9
University of Southern California
Center for Systems and Software Engineering
Context Diagram – Requirements Analysis Process
Source: Systems Engineering Handbook. INCOSE Version 3.2
10
University of Southern California
Center for Systems and Software Engineering
Requirements Development
The customer ’s needs and objectives are developed into operational concepts to satisfy them System functional and performance requirements are developed to define what the system must do and under what environments and constraints it must perform Once the baseline system requirements and system architecture are established, they are decomposed through successively lower levels.
Source: DoD Instruction 5000.02 (2008)
11
University of Southern California
Center for Systems and Software Engineering
Requirements Baseline and Change Control
• •
A
Requirements baseline
is established when the functional requirements and characteristics of the system are formally documented and placed under configuration control
– –
After the System Requirements Review or equivalent milestone Modifications to the baseline are requested through Engineering Change Proposals (ECP) Typically, a formal management approval structure is established to prioritize the ECPs, evaluate their impact, and make decisions regarding their approval and implementation
–
Other change documents used to control the requirements baseline are Requests for Deviation and Requests for Waivers
–
RDWs propose a departure from the baseline by allowing the acceptance of products that are non-conformant or do not meet the requirements as stated
Source: DoD Instruction 5000.02 (2008)
12
University of Southern California
Center for Systems and Software Engineering
Systems Engineering Reviews and Acquisition Lifecycle Phases
(Program Initiation)
Sys Tech Requirements Item Design Rqmts.
Detailed Design Rqmts.
Source: DoD Instruction 5000.02 (2008)
13
University of Southern California
Center for Systems and Software Engineering
Requirements and Evolutionary Acquisition Process
Source: DoD Instruction 5000.02 (2008)
14
University of Southern California
Center for Systems and Software Engineering
Requirements Base Measures
Base Measures
# of Requirements # of requirements defects
Measurement Method
Count the # of requirements in the applicable specification (# of shalls/wills/musts) Count the # of defects # of requirements changes Impact of each change Count the # of changes per category (added, deleted or modified) from Engineering Change Proposals (ECP), requirements database, etc.
Estimate effort-hours per change (sometimes estimated in ECPs, budgeted hours vs. actual hours)
Source: Systems Engineering Leading Indicators Guide, Version 2.0, 2010
15
University of Southern California
Center for Systems and Software Engineering
Requirements Derived Measures
% Requirements Growth =
((# of requirements in current baseline - # of requirements in previous baseline) ( # of requirements in previous baseline) X 100
% Requirements Modified =
(# of requirements modified / total # of requirements) X 100
Source: Systems Engineering Leading Indicators Guide, Version 2.0, 2010
16
University of Southern California
Center for Systems and Software Engineering
Requirements Volatility Metrics (1 of 2)
Program “A” Requirements Volatility
Volatility expressed as a % of the total number of requirements
Systems Engineering Leading Indicators Guide, Version 2.0, 2010
17
University of Southern California
Center for Systems and Software Engineering
Requirements Volatility Metrics (2 of 2)
Program “B” Requirements Volatility
Volatility metrics track the number and type of changes over time
Source: Hammer, T., Huffman, L., and Rosenberg, L. (1998).
18
University of Southern California
Center for Systems and Software Engineering
Periodic and Cumulative Measures
19
University of Southern California
Center for Systems and Software Engineering
Observations on the Behavior of Volatility
1. Requirements volatility is caused by an identifiable set of project and organizational factors 2. The level of requirements volatility is a function of the system life cycle phase 3. Requirements volatility leads to an increase in project size and cost 4. The cost and effort impact of a requirements change increases the later the change occurs in the system life cycle 5. The impact of requirements volatility varies depending on the type of change: added, deleted, or modified
Sources: Kotonya and Sommerville (1995); Hammer et al. (1998); Malaiya and Denton (1999); Stark et al. (1999); Houston (2000); Zowghi and Nurmuliani (2002); Ferreira (2002); Kulk and Verhoef 2008 20
University of Southern California
Center for Systems and Software Engineering
Expected Level of Volatility Across Lifecycle Phases (Survey)
don't know don't know 20 don't know <5% 5-10% 10-20% don't know <5% 5-10% <5% 15 <5% 10 5 0 >20% Conceptualize 10-20% 5-10% 5-10% >20% 10-20% >20% Development Operational Test & Evaluation Expected Requirements Volatility (%) Transition to Operation 21
University of Southern California
Center for Systems and Software Engineering
Impact of Hardware/Software Project Breakdown on Expected Volatility
Operational Test & Evaluation Lifecycle Phase
100% 80% 60%
Transition to Operation Lifecycle Phase
100% 80%
Respondents from S/W intensive projects tend to expect more volatility later in the lifecycle
60% 40% 40% 20% 20% 0% <5% 5-10% 10-20% >20% Expected Requirements Volatility (%) 75% Hardware, 25% Software 25% Hardware, 75% Software 50% Hardware, 50% Software 0% <5% 5-10% 10-20% >20% Expected Requirements Volatility (%) 75% Hardware, 25% Software 25% Hardware, 75% Software 50% Hardware, 50% Software
Source: LAI Knowledge Exchange / CSSE ARR Survey (2010)
22
University of Southern California
Center for Systems and Software Engineering
Sample Volatility Profiles
Source: Practical Software & Systems Measurement Conference Survey (2010)
23
University of Southern California
Center for Systems and Software Engineering
Requirements Trends as a Leading Indicator of Project Performance
•
Evaluates trends in the growth, change, completeness and correctness of the system requirements.
•
It helps to determine the stability and completeness of the system requirements which could potentially impact project performance Requirements Growth Trends Corrective Action Taken Planned # of Requirements Actual # of Requirements Projected # of Requirements CDR SRR PDR Time
Source: Systems Engineering Leading Indicators Guide, Version 2.0, 2010
24
University of Southern California
Center for Systems and Software Engineering
Causes of Requirements Volatility
• •
External factors
– – – – –
Shifting customer priorities and needs Changes in the political and business environment Addition or change of stakeholders Development of new technologies Changes in co-dependent systems Internal factors
– –
Deficient requirements development processes Lack of experienced systems engineering resources applied to requirements analysis
–
Poor initial understanding or interpretation of customer needs by the development team
–
Changes in organizational structure and policies
Sources: Kotonya and Sommerville (1995); Houston (2000); Zowghi and Nurmuliani (2002); Kulk and Verhoef 2008; Ferreira, S., Collofello, J., Shunk, D., and Mackulak, G. (2009)
25
University of Southern California
Center for Systems and Software Engineering
Systems Dynamics View of the Causes of Requirements Volatility
6 Contextual / Environmental Changes 7 Experienced Staff 8 Changes in COTS Products 9 Changes in Org / structure/ Process 1 Poor Understanding of the System & Customer Needs 2 Customer-driven Scope Change 3 Changes in Co-dependent Systems 4 Requirements Process Maturity 5 Technology Maturity Requirements Volatility
Sources: Kotonya and Sommerville (1995); Hammer et al. (1998); Malaiya and Denton (1999); Stark et al. (1999); Houston (2000); Zowghi and Nurmuliani (2002); Kulk and Verhoef 2008; Ferreira, S., Collofello, J., Shunk, D., and Mackulak, G. (2009)
26
University of Southern California
Center for Systems and Software Engineering
Impacts of Requirements Volatility
• • • • • • •
Several research studies have found that requirements volatility is positively correlated with increase in The functional size of the project (due to discarded lines of code and additional requirements) Engineering effort Cost (often >20%) and schedule duration Changes in requirements result in additional rework and increased defect density The impact of changing requirements is much greater the later the change occurs in the project lifecycle Removing a requirement may not necessarily result in a net decrease in engineering effort and cost
Sources: Kotonya and Sommerville (1995); Hammer et al. (1998); Malaiya and Denton (1999); Stark et al. (1999); Houston (2000); Zowghi and Nurmuliani (2002); Kulk and Verhoef 2008; Ferreira, S., Collofello, J., Shunk, D., and Mackulak, G. (2009)
27
University of Southern California
Center for Systems and Software Engineering
Systems Dynamics View of the Impact of Requirements Volatility
Quality Rework / Defects Project Cost +/ Requirements Volatility +/ Number of System Requirements +/ Project Effort +/ +/ Project Schedule +/ Customer Satisfaction
Sources: Kotonya and Sommerville (1995); Hammer et al. (1998); Malaiya and Denton (1999); Stark et al. (1999); Houston (2000); Zowghi and Nurmuliani (2002); Kulk and Verhoef 2008; Ferreira, S., Collofello, J., Shunk, D., and Mackulak, G. (2009)
28
University of Southern California
Center for Systems and Software Engineering
Cost Commitment on Projects
% 1 00 C om m itm e nt to Te chnology, Configuration, P erform ance, Cost, etc.
75 Cost Incurre d S ys tem -S pecific Know ledge 50 25 E ase of C ha nge N E E D Conceptua l Prelim inary De sign Deta il De sign and Developm e nt Construction a nd/or P roduction S ys tem Use , P haseout, and D is pos al
Changes to the System are more difficult to implement the later they occur in the lifecycle
Source: Blanchard and Fabrycky (1998)
29
University of Southern California
Center for Systems and Software Engineering
Expected Cost Penalty due to Requirements Volatility (Survey Results)
11.0
10.0
3.0
2.0
1.0
0.0
9.0
8.0
7.0
6.0
5.0
4.0
Conceptualize Development Operational Test & Evaluation Transition to Operation
Source: COCOMO Forum Survey (2010) and PSM Workshop (2011)
30
University of Southern California
Center for Systems and Software Engineering
Expected Cost Penalty per Change Category (Survey Results)
Source: COCOMO Forum Survey Exercise (2010) and PSM Workshop (2011)
31
University of Southern California
Center for Systems and Software Engineering
Requirements Volatility: A Sizing and Cost Estimation Challenge
• • • •
Our dynamic competitive environment leads to rapid changes in objectives, constraints and priorities Requirements are often emergent instead of pre specifiable These trends have lead to the increasing use of incremental and evolutionary development strategies Counting the number of source lines of code (SLOC) delivered may underestimate the size of the system
–
Portions of the code may have been modified or deleted due to changing requirements
–
These modified or discarded lines would not result in an increase in the final SLOC count
Source: AFCAA Software Cost Estimation Metrics Manual Version 0.82 (draft)
32
University of Southern California
Center for Systems and Software Engineering
Requirements Volatility and Parametric Cost Estimation
• •
The Constructive Cost Model (COCOMO II) incorporated the impact of requirements volatility on the effective size of software systems using an adjustment factor called REVL (Requirements Evolution and Volatility) REVL is also known as the “breakage parameter,” defined as the % of code discarded due to changes in requirements.
Size
1
REVL
100
Size D
Where,
Size D
is the initial equivalent size of the software product adjusted for reuse
Source: Boehm et al. (2000).
33
University of Southern California
Center for Systems and Software Engineering
Requirements Volatility Expected Ranges
Rating Level
Very Low Low Nominal High Very High
RVOL Range
0-6% 6-12% 12-24% 24-48% >48%
Source: AFCAA Software Cost Estimation Metrics Manual Version 0.82 (draft)
RVOL Average
3% 9% 18% 36% 72% 34
MM nom nom
2 .
.
8 1 1 .
.
20 04
i
4 1
W i
Center for Systems and Software Engineering
Requirements Volatility in Ada COCOMO (1 of 2)
•
Ada process model attempted to reduce the diseconomies of scale (exp = 1.20) of the COCOMO embedded mode
MM nom
2 .
8
KDSI
1 .
20
Where,
MM non
= number of man-months required to develop a nominal software product KDSI = thousands of delivered source instructions
Source: Ada COCOMO and the Ada Process Model (Boehm and Royce, 1989)
35
• • • University of Southern California
Center for Systems and Software Engineering
Requirements Volatility in Ada COCOMO (2 of 2)
Improvements in diseconomies of scale resulting from the Ada process model by 1. Early software architecture definition 2. Mitigation of risks by PDR 3. Requirements stabilization by PDR; and 4. The team ’s experience with the Ada process model
MM nom
2 .
8
KDSI
1 .
04
i
4 1
W i
Each of these four elements is rated using a scale from 0.00 to 0.05
The ratings are summed to determine Σ
Source: Ada COCOMO and the Ada Process Model (Boehm and Royce, 1989)
36
University of Southern California
Center for Systems and Software Engineering
Ada COCOMO Σ factor: Requirements Volatility
No Changes Small Non Characteristic Rating Scale critical Changes Frequent non-critical Changes Occasional Moderate Changes Frequent Moderate Changes Many Large Changes System requirements baselined, under rigorous change control Level of uncertainty in key requirements areas, mission, user interfaces, hardware, other interfaces Organizational track record in keeping requirements stable Use of incremental development to stabilize requirements System architecture modularized around major sources of change .00
Fully Very Little Excellent Full Fully .01
Mostly Little Strong Strong Mostly .02
Generally Some Good Good Generally
Source: Ada COCOMO and the Ada Process Model (Boehm and Royce, 1989)
.03
Some Moderate Some Some .04
Little Considerable Significant Weak Little Little .05
None Extreme Very Weak None None 37
University of Southern California
Center for Systems and Software Engineering
Actions to Mitigate Requirements Volatility
1. Improve communication between stakeholders and development team 2. Improve the capability and maturity of the requirements development process 3. Additional training / more experienced personnel 4. Increase the use of prototypes and models to validate requirements 5. Improve technology insertion processes 6. Increase management oversight (i.e. metrics, reviews) 7. Establish a more strict change control process
38
University of Southern California
Center for Systems and Software Engineering
Mitigation Factor Effectiveness: Survey Results
Source: CSSE ARR Survey (2010)
39
University of Southern California
Center for Systems and Software Engineering
Characteristics of Good Requirements (1 of 3)
1. Necessary
• Avoid specification of design details & redundant requirements
2. Implementation Independent
• Specify “what” is to be done, not “how” to do it
3. Clear and Concise
• • Don ’t use ambiguous terms; avoid compound requirements with multiple “shall statements” and conjunctions such as “and”, “or” Example – is the following a good requirement?
•
“The car shall accelerate from 0 to 60 MPH in 5 seconds and break to a standstill in 10 seconds under extreme conditions ”
Source: Systems Engineering Handbook. INCOSE Version 3.2
40
University of Southern California
Center for Systems and Software Engineering
Characteristics of Good Requirements (2 of 3)
4. Complete
• Provide all the information necessary for the developer to understand what you mean • Do you have any TBXs (to be determined, to be confirmed)?
5. Consistent
• • Is the requirement self-consistent?
Do you have conflicting statements?
6. Achievable
• Engage the developer/designer in the requirements definition to ensure they concur that the requirement can be met
Source: Systems Engineering Handbook. INCOSE Version 3.2
41
University of Southern California
Center for Systems and Software Engineering
Characteristics of Good Requirements (3 of 3)
7. Traceable
• Are there any orphan requirements (i.e. they don ’t trace to a higher level requirement)?
8. Verifiable
– Avoid subjective terms such as “provide support,” and “user friendly ” • – Look out for conditions that are not clearly achieved such as “minimize,” “to the degree possible,” etc.
Example:
1. The aircraft should be as light as possible 2. The wet weight of the aircraft shall not exceed 2000 lbs.
Source: Systems Engineering Handbook. INCOSE Version 3.2
42
University of Southern California
Center for Systems and Software Engineering
Summary
• • • •
Requirements changes are unavoidable in today ’s rapidly changing environment Some level of volatility is healthy and is part of the system development process However, excessive volatility and late requirements changes can have a very detrimental effect on the project – this is a risk that must be addressed and mitigated Requirements volatility should be anticipated, managed, and accounted for in cost and size estimates
43
University of Southern California
Center for Systems and Software Engineering
References
• Air Foce Cost Analysis Agency Software Cost Estimation Metrics Manual, Draft Version 0.82 (2011) • Boehm, B. and Royce, W. (1989). Ada COCOMO and the Ada Process Model, TRW Defense Systems Group • Boehm, B. (1991). “Software Risk Management: Principles and Practices.” IEEE Software. Vol 8 (No.1).pp 32-41 • Boehm, B., Abts, C., Brown, A.W., Chulani, S., Clark, B., Horowitz, E., Madachy, R., Reifer, D.J., and Steece, B. (2000). Software Cost Estimation with COCOMO II. Prentice Hall • EIA (2002). EIA-731.1 - Systems Engineering Capability Model • Ferreira, S., Collofello, J., Shunk, D., and Mackulak, G. (2009). “Understanding the effects of requirements volatility in software engineering by using analytical modeling and software process simulation.” The Journal of Systems and Software. Vol. 82, pp 1568-1577.
• General Accounting Office (2004). Stronger Management Practices are Needed to Improve DOD’s Software-intensive Weapon Acquisitions (GAO-04-393). Defense Acquisitions.
• Hammer, T., Huffman, L., and Rosenberg, L. (1998). “Doing requirements right the first time.” Crosstalk, the Journal of Defense Software Engineering. Pp 20-25.
• Houston, Dan X. (2000). A Software Project Simulation Model for Risk Management, Ph.D. Dissertation, Arizona State University • IEEE (1990). IEEE Standard Glossary of Software Engineering Terminology (ANSI), [1-55937-067-X] [SH13748-NYF], IEEE • INCOSE (2010). Systems Engineering Handbook. INCOSE Version 3.2
• Kotonya, G., Sommerville, I., (1998). Requirements Engineering: Processes and Techniques. John Wiley and Sons, Ltd.
• MIL-STD-498. 1994. Software Development and Documentation. U.S. Department of Defense.
• Nguyen, V. and Boehm, B. (2010). A COCOMO Extension for Software Maintenance. 25th International Forum on COCOMO and Systems/Software Cost Modeling • Reifer, Donald J. 2000. “Requirements Management: The Search for Nirvana.” IEEE Software. Vol 17 (No. 3), pp 45-47 • Roedler, G. and Rhodes, D. (2007). Systems engineering leading indicators guide. Version 1. Massachusetts Institute of Technology, INCOSE, and PSM • Weinberg, G. (1993). Quality Software Management: Volume 2 First-Order Measurement. Dorset House.
• Zowghi, D. and Nurmuliani, N. (2002). A Study of the Impact of Requirements Volatility on Software Project Performance. Proceedings of the Ninth Asia Pacific Software Engineering Conference 44