Transcript Shrinking the Cone of Uncertainty with Continuous Assessment
University of Southern California
Center for Systems and Software Engineering
Shrinking the Cone of Uncertainty with Continuous Assessment
Pongtip Aroonvatanaporn CSCI 510 Fall 2011 October 5, 2011 (C) USC-CSSE 10/5/11 1
University of Southern California
Center for Systems and Software Engineering
Outline
• • • • •
Introduction
– Motivation – Problems
Related Works Proposed Methodologies Conclusion Tool Demo 10/5/11 (C) USC-CSSE 2
University of Southern California
Center for Systems and Software Engineering
Motivation
• • • •
The Cone of Uncertainty Exists until the product is delivered, or even after The wider, the more difficult to ensure accuracies and timely deliveries Focus on uncertainties of team aspects from product design onwards
– COCOMO II space – Many factors before that (requirements volatility, technology, etc.)
10/5/11 (C) USC-CSSE 3
University of Southern California
Center for Systems and Software Engineering
Motivation
•
Key principles of ICSM
– Stakeholder satisficing – Incremental and evolutionary growth of system definition and stakeholder commitment – Iterative system development and definition – Concurrent system definition and development – Risk management •
10/5/11 COCOMO II
– COCOMO II space is in the development cycle – Influences on estimations and schedules [Construx, 2006] • Human factors: 14x • Capability factors: 3.5x
• Experience factors: 3.0x
(C) USC-CSSE 4
University of Southern California
Center for Systems and Software Engineering
Motivation
• •
Standish CHAOS Summary 2009 Surveyed 9000 projects Delivered with full capability within budget and schedule Cancelled Over budget, over schedule, or undelivered
32% 24% 44%
68% project failure rate 10/5/11 (C) USC-CSSE 5
University of Southern California
Center for Systems and Software Engineering
Terms and Definitions
• •
Inexperienced
– Inexperienced in general – Experienced, but in new domain – Experienced, but using new technology
Continuous Assessment
– Assessments take place over periods of time – Done in parallel with process, instead of only at the end – Widely used in education – Used in software process measurement [Jarvinen, 2000]
10/5/11 (C) USC-CSSE 6
University of Southern California
Center for Systems and Software Engineering
The Problem
• •
Experienced teams can produce better estimates
– Use “yesterday’s weather” – Past projects of comparable sizes – Past data of team’s productivity – Knowledge of accumulated problems and solutions
Inexperienced teams do not have this luxury No tools or data that monitors project
’
s progression within the cone of uncertainty 10/5/11 (C) USC-CSSE 7
University of Southern California
Center for Systems and Software Engineering
Problems of Inexperience
• • • • •
Imprecise project scoping
– Overestimation vs. underestimation
Project estimations often not revisited
– Insufficient data to perform predictions – Project’s uncertainties not adjusted
Manual assessments are tedious
– Complex and discouraging
Limitations in software cost estimation
– Models cannot fully compensate for lack of knowledge and understanding of project
Overstating team’s capabilities
– Unrealistic values that do not reflect project situation – Teams and projects misrepresented (business vs. technical)
10/5/11 (C) USC-CSSE 8
University of Southern California
Center for Systems and Software Engineering
Imprecise Project Scoping
• •
Based on CSCI577 data, projects either significantly overestimate or underestimate effort
– Possibly due to: • Unfamiliarity with COCOMO • Inexperienced
Teams end up with inaccurate project scoping
– Promise too much
10/5/11 (C) USC-CSSE 9
University of Southern California
Center for Systems and Software Engineering
Overestimation
• • • •
Estimate is too high to achieve within available resources Need to reduce the scope of the project
– Re-negotiate requirements with client – Throw away some critical core capabilities
Lose the expected benefits Often do not meet client satisfactions/needs 10/5/11 (C) USC-CSSE 10
University of Southern California
Center for Systems and Software Engineering
Underestimation
• • •
Estimate is lower than actual Project appears that it can be done in with less resources
– Clients may ask for more capabilities – Teams may end up promising more
As project progresses, team may realize that project is not achievable
– If try to deliver what was promised, quality suffers – If deliver less that what was promised, clients suffer
10/5/11 (C) USC-CSSE 11
University of Southern California
Center for Systems and Software Engineering
Project Estimations Not Revisited
• • • •
At the beginning, teams do not always have the necessary data
– No “yesterday’s weather” – High number of uncertainties
Initial estimates computed are not accurate If estimates are readjusted, no problem Reality is, estimates are left untouched 10/5/11 (C) USC-CSSE 12
University of Southern California
Center for Systems and Software Engineering
Estimations in ICSM
• • •
Estimates are “supposedly” adjusted during each milestone reviews
– Reviewed by team – Reviewed by stakeholders
Adjustments require necessary assessments to become more accurate Without assessments, adjustments are made with no directions 10/5/11 (C) USC-CSSE 13
University of Southern California
Center for Systems and Software Engineering
Manual Assessments are Tedious
• • • • •
Complex process Time consuming Require experienced facilitator/assessor to perform effectively Often done by conducting various surveys, analyze the data, and determine weak/strong points
– Repeated as necessary
Discouraging to the teams 10/5/11 (C) USC-CSSE 14
University of Southern California
Center for Systems and Software Engineering
Size Reporting
• •
How to accurately report progress
– By developer’s status report?
– By project manager’s take?
Report by size is most accurate
– Counting logical lines of code is difficult – Even with tools support, a labor intensive task to report accurately
10/5/11 (C) USC-CSSE 15
University of Southern California
Center for Systems and Software Engineering
Limitations in Software Cost Estimation
• • •
Little compensation for lack of information and understanding of software to be developed The “Cone of Uncertainty”
– There’s a wide range of products and costs that the project can result in – Not 100% sure until product is delivered
Designs and specifications are prone to changes
– Especially in agile environment
10/5/11 (C) USC-CSSE 16
University of Southern California
Center for Systems and Software Engineering
Overstating Team’s Capabilities
• • •
Unrealistic values
– COCOMOII parameters – Do not reflect project’s situation
Business vs. Technical
– – – –
Clients
– want the highest value
Sales
– want to sell products
Project Managers
– want the best team
Programmers
– want the least work
Is it really feasible?
– Provide the evidence – From where?
10/5/11 (C) USC-CSSE 17
University of Southern California
Center for Systems and Software Engineering
Competing Project Proposals
• • •
Write project proposals to win Often overstate and overcommit yourselves
– Put the best, highly-capabled people on the project – Have high experienced teams – Keep costs low – Any capabilities are possible
This may only be true at the time of writing
– What about when project really begins?
10/5/11 (C) USC-CSSE 18
University of Southern California
Center for Systems and Software Engineering
Ultimate Problem
• • •
Developers rather spend time to develop rather than
– Documenting – Assessing – Adjusting
Not as valuable to developers as to other stakeholders In the end, nothing is done to improve 10/5/11 (C) USC-CSSE 19
University of Southern California
Center for Systems and Software Engineering
The Goal
• • •
Develop a framework to address mentioned issues Help unprecedented projects track project progression Reduce the uncertainties in estimation
– Achieve eventual convergence of estimate and actual
Must be quick and easy to use (C) USC-CSSE 10/5/11 20
University of Southern California
Center for Systems and Software Engineering
Benefits
• • • •
Improve project planning and management
– Resources and goals – Ensure the accuracy of estimation – Determine/confirm project scope
Improved product quality control
– Certain about amount of work required – Better timeline – Allows for better work distribution
Actual project progress tracking
– Better understanding of project status – Actual progress reports
Help manage realistic schedule and deliveries 10/5/11 (C) USC-CSSE 21
University of Southern California
Center for Systems and Software Engineering
Outline
• • • • •
Introduction Related Works
– Assessment – Sizing – Management
Proposed Methodologies Conclusion Tool Demo 10/5/11 (C) USC-CSSE 22
University of Southern California
Center for Systems and Software Engineering
• • • •
IBM Self-Check
[Kroll, 2008] A survey-based assessment/retrospective Method to overcome common assessment pitfalls
– Bloated metrics, Evil scorecards, Lessons forgotten, Forcing process, Inconsistent sharing
Reflections by the team for the team Team choose set of core practices to focus assessment on
– Discussions triggered by inconsistent answers between team members – Develop actions to resolve issues
10/5/11 (C) USC-CSSE 23
University of Southern California
Center for Systems and Software Engineering
Software Sizing and Estimation
• •
Agile techniques
– Story points and velocity [Cohn, 2006] – Planning Poker [Grenning, 2002]
Treatments for uncertainty
– PERT Sizing [Putnam, 1979] – Wideband Delphi Technique [Boehm, 1981] – COCOMO-U [Yang 2006]
10/5/11 Require high level of expertise and experience (C) USC-CSSE 24
University of Southern California
Center for Systems and Software Engineering
Project Tracking and Assessment
PERT Network Chart
[Wiest, 1977]
GQM
[Basili, 1995]
Goal Question Metric Objective Answer Measurement
• Identify critical paths • Nodes updated to show progress • Grows quickly • Becomes unusable when large, especially in
smaller agile environments
• Captures progress from conceptual, operational, and qualitative levels • Align with organization/team • Only useful when
used correctly Burn Charts
[Cockburn, 2004] • Effective in tracking progress • Not good at
responding to major changes Architecture Review Board
[Maranzano, 2005]
10/5/11
• • Reviews to validate feasibility of architecture and design • Increases the likelihood of project success • Adopted by software engineering course
Stabilize team, reduce knowledge gaps, evaluate risks (C) USC-CSSE 25
University of Southern California
Center for Systems and Software Engineering
Outline
• • • • •
Introduction Related Works Proposed Methodologies
– Project Tracking Framework – Team Assessment Framework
Conclusion Tool Demo 10/5/11 (C) USC-CSSE 26
University of Southern California
Center for Systems and Software Engineering
Project Tracking Framework
[Aroonvatanaporn, 2010]
Integrating the Unified Code Count tool and COCOMO II model
– Quickly determine effort based on actual progress – Extend to use Earned-Value for percent complete
PM NS E
A
Size E
0 .
91 0 .
01
j
5 1
SF j i n
1
EM i
Hypotheses:
H1 10/5/11
Adjusted with
REVL (C) USC-CSSE 27
University of Southern California
Center for Systems and Software Engineering
Size Counting
• • • •
COCOMO uses size to determine effort Use of the Unified Code Count tool Allows for quick collection of SLOC data
– Then fed to the COCOMO model to calculate equivalent effort
Collected at every build
– Depends on iteration length
10/5/11 (C) USC-CSSE 28
10/5/11
University of Southern California
Center for Systems and Software Engineering
Project Tracking Results
[Aroonvatanaporn, 2010] Initial estimate
~50 %
Initial estimate Adjusted estimate Accumulated effort Accumulated effort Adjusted estimate
~18% (C) USC-CSSE 29
University of Southern California
Center for Systems and Software Engineering
Team Assessment Framework
• •
Similar to the approach of
– IBM Self-Check • Use survey-based approach to identify
inconsistencies
team members and
knowledge gaps
among – Inconsistencies/uncertainties in answers – Use conflicting questions to validate consistencies
Two sources
for question development
10/5/11 (C) USC-CSSE 30
University of Southern California
Center for Systems and Software Engineering
Question Development
Team’s assessment performed by each team member Strengths, weaknesses, issues, etc.
Survey questions • •
Team members assess and evaluate their own team
– Operational concept engineering – Requirements gathering – Business case analysis – Architecture development – Planning and control – Personnel capability – Collaboration ICSM
These questions focus on resolving team issues and reducing knowledge gaps 10/5/11 (C) USC-CSSE 31
University of Southern California
Center for Systems and Software Engineering
Adjusting the COCOMOII Estimates
• • • •
Answering series of questions is more effective than providing metrics
[Krebs, 2008]
Framework to help adjust COCOMO II estimates to reflect reality Questions developed to focus on
– Team stabilization and reducing knowledge gaps – Each question relate to COCOMO II scale factors and cost drivers
Two approaches to determine relationship between question and COCOMO II parameters
– Finding correlation – Expert advice
10/5/11 (C) USC-CSSE 32
University of Southern California
Center for Systems and Software Engineering
Example Scenario
COCOMO II ACAP
:
PCAP
:
APEX
:
PLEX
:
LTEX
: HI LO NOM HI NOM HI NOM NOM + 50% HI NOM + 50%
Survey
Have sufficiently talented and experienced programmers and systems engineering managers been identified?
Discussion
• Where do we lack in experience?
• How can we improve?
10 4 9 Average
:
Deviation
: 7.7
3.2
10/5/11 (C) USC-CSSE 33
University of Southern California
Center for Systems and Software Engineering
Outline
• • • • •
Introduction Related Works Proposed Methodologies Conclusion
– Past 577 data – Summary
Tool Demo 10/5/11 (C) USC-CSSE 34
University of Southern California
Center for Systems and Software Engineering
COCOMO II Estimation Range
Team provided range vs. COCOMO II built-in calculation
– Data from Team 1 of Fall 2010 – Spring 2011 semesters Team’s pessimistic
10/5/11
1500 1000 500 0 3500 3000 2500 2000 Valuations Most likely Team’s optimistic Foundations Rebaseline Foundations
Phase (C) USC-CSSE
COCOMO II pessimistic Development Iteration 1 COCOMO II optimistic Development Iteration 2
35
University of Southern California
Center for Systems and Software Engineering
CSCI577 Estimation Errors
• •
Data from Fall 2009 – Spring 2010 semesters Fall 2010 – Spring 2011 will be collected after this semester ends
4500 4000 3500 3000 2500 2000 1500 1000 500 0
Architected Agile Teams
2 4 7
Team
9
10/5/11
11 Actual Effort Valuation Foundations RDC IOC1 IOC2
(C) USC-CSSE
3000 2500 2000 1500 1000 500 0
NDI/NCS Teams
5 6
Team
12 Actual Effort Valuation Foundations Development
36
University of Southern California
Center for Systems and Software Engineering
Conclusion
• • •
This research focuses on improving team performance and project outcomes
– Tracking project progress – Synchronization and stabilization of team – Improving project estimations
Framework to shrink the cone of uncertainty
– Less uncertainties in estimations – Less uncertainties within team – Better project scoping
The tool support for the framework will be used to validate and refine the assessment framework 10/5/11 (C) USC-CSSE 37
University of Southern California
Center for Systems and Software Engineering
Outline
• • • • •
Introduction Related Works Proposed Methodologies Conclusion Tool Demo
– What is the tool?
– What does it support?
10/5/11 (C) USC-CSSE 38
University of Southern California
Center for Systems and Software Engineering
Tool Support for Framework
• •
Develop with IBM Jazz
– Provides team management – Provides user management – Support for high collaborative environment
Potentials
– Extensions to Rational Team Concert – Support for other project management tools
10/5/11 (C) USC-CSSE 39
University of Southern California
Center for Systems and Software Engineering
Tool Support
• • •
Tool will be used throughout the project life cycle Used for:
– Tracking project progress – Project estimation – Team assessment
Frequency
– Start after prototyping begins – Done every two weeks?
10/5/11 (C) USC-CSSE 40
University of Southern California
Center for Systems and Software Engineering
Different Project Types
• •
Architected Agile
– Track through development of source code
NDI/NCS
– Utilize Application Points
10/5/11 (C) USC-CSSE 41
University of Southern California
Center for Systems and Software Engineering
Tool
10/5/11 (C) USC-CSSE 42
University of Southern California
Center for Systems and Software Engineering
Tool
10/5/11 (C) USC-CSSE 43
University of Southern California
Center for Systems and Software Engineering
Tool
10/5/11 (C) USC-CSSE 44
University of Southern California
Center for Systems and Software Engineering
Tool
10/5/11 (C) USC-CSSE 45
University of Southern California
Center for Systems and Software Engineering
Publications
Aroonvatanaporn, P., Sinthop , C., and Boehm, B. “Reducing Estimation Uncertainty with Continuous Assessment: Tracking the ‘Cone of Uncertainty’.” In
Proceedings of the IEEE/ACM International conference on Automated Software Engineering
, pp. 337-340. New York, NY, 2010.
10/5/11 (C) USC-CSSE 46
University of Southern California
Center for Systems and Software Engineering References
Basili , Victor R. “Applying the goal/question/metric paradigm in the experience factory”. In
Software Quality Assurance and Measurement: Worldwide Perspective
, pp. 21-44. International Thomson Computer Press, 1955. Boehm, B.
Software Engineering Economics
. Prentice-Hall, 1981.
Boehm, B. and Lane, J. “Using the incremental commitment model to integrate systems acquisition, systems engineering, and software engineering.”
CrossTalk
, pp. 4-9, October 2007. Boehm, B., Abts, C., Brown, A.W., Chulani, S., Horowitz, E., Madachy, R., Reifer, D.J., and Steece, B.
Software Cost Estimation with COCOMO II
. Prentice-Hall, 2000.
Boehm, B., Port, D., Huang, L., and Brown, W. “Using the Spiral Model and MBASE to Generate New Acquisition Process Models: SAIV, CAIV, and SCQAIV.”
CrossTalk
, pp. 20-25, January 2002 Boehm, B. et al. “Early Identification of SE-Related Program Risks.” Technical Task Order TO001, September 2009. Cockburn, A. “Earned-value and Burn Charts (Burn Up and Burn Down).
Crystal Clear,
Addison-Wesley, 2004. Cohn, M.
Agile Estimating and Planning.
Prentice-Hall, 2006. Construx Software Builders, Inc. “10 Most Important Ideas in Software Development”. http://www.scribd.com/doc/2385168/10 Most-Important-Ideas-in-Software-Development Grenning, J. Planning Poker, 2002. http://www.objectmentor.com/resources/article/PlanningPoker.zip
10/5/11 (C) USC-CSSE 47
University of Southern California
Center for Systems and Software Engineering References
IBM Rational Jazz. http://www.jazz.net
Jarvinen, J.
Measurement based continuous assessment of software engineering process.
PhD thesis, University of Oulu, 2000 Koolmanojwong, S.
The Incremental Commitment Spiral Model Process Patterns for Rapid-Fielding Projects.
PhD thesis, University of Southern California, 2010 Krebs, W., Kroll, P., and Richard, E. “Un-assessment – reflections by the team, for the team.”
Agile 2008 Conference,
2008. Kroll, P. and Krebs, W. “Introducing IBM Rational Self Check for Software Teams, 2008”. http://www.ibm.com/developerworks/rational/library/edge/08/may08/kroll_krebs Maranzano, J.F., Rozsypal, S.A., Zimmerman, G.H., Warnken , P.E., and Weiss, D.M. “Architecture Reviews: Practice and Experience.”
Software, IEEE
, 22: 34-43, March-April, 2005.
Putnam, L. and Fitzsimmons, A. “Estimating Software Costs.”
Datamation
, 1979.
Standish Group. CHAOS Summary 2009. http://standishgroup.com
Unified Code Count. http://sunset.usc.edu/research/CODECOUNT/ USC Software Engineering I Class Website. http://greenbay.usc.edu/
10/5/11 (C) USC-CSSE 48
University of Southern California
Center for Systems and Software Engineering
References
Wiest, J.D. and Levy, F.K.
A Management Guide to PERT/CPM.
Prentice-Hall, Englewood Press, 1977.
Yang, D., Wan, Y., Tang, Z., Wu, S., He, M., and Li, M. “COCOMO-U: An Extension of COCOMO II for Cost Estimation with Uncertainty.”
Software Process Change,
2006, pp.132-141
10/5/11 (C) USC-CSSE 49