Transcript Software Metrics in Motorola by Daskalantonakis

Software Metrics in Motorola
by Daskalantonakis
• Software Metric:
– a method of quantifying the “extent” to which a software i)
process, ii) product, or iii) project possesses a certain attribute,
including:
• formula for computing the metric value
• chart for presenting the metric value
• guideline to use and interpret the the metric values, (in context)
(what metric scale is definition --- nominal, ordinal, interval or ratio?)
• Software Metric must be defined with intended goal and
the attributes of interest (Basili’s G-Q-M).
• Implementation of software metrics is complex and involves
multiple dimensions (discussed later)
Some Pre-requisites (tools & processes)
• Tools to help collect and analyze data should
exist
– cost accounting system
– software configuration management system
– problem reporting-corrective action recording
system
• Processes are documented and really in use
(executed) for those processes that are
associated with the metric ---- remember SEI ‘s
process improvement cycle where process must be
in use before you can control and improve? ---
(5) Multi-dimensional Views of Metrics
• 1) Usefulness/Utilization (for a metric to be useful ---it must be):
–
–
–
–
precisely defined and easy to understand
objective
cost effective (value of information > cost of getting it)
informative
• 2) Types of Metric
– process
– product
– project
• 3) Metric audience and users needs to be considered
–
–
–
–
–
software users
senior management
software project managers
Software engineers
software process engineers, quality assurance engineers, management staff
(5) Multi-dimensional Views of Metrics
• 4) Must Address Metric Users’ Needs:
– gain metric consensus/acceptance
– provide training and consulting support
– provide tools support for data gathering, analysis, etc.
• 5) Must Account for Levels (organizational) of Measurement
–
–
–
–
–
company wide
divisional or product group
project
sub-component
individual unit
Software Metric Initiatives in Motorola
• Better understand the software development process ---- to
improve --- not just to collect information
– productivity
– quality
– cycle time
• Two areas of focus for the metric initiatives
– a) process improvement
– b) project control
Note: the goal is NOT just to measure, but to “improve” through
measurement, analysis and feedback --- a bit like the goals of SEI
• A company-wide Metric Working Group established to
– Define metrics
– Set up processes to deploy the metrics
Metrics for Process Improvement
•
With management support, a Quality Policy for Software Development
(QPSD) establish 8 areas of focus for measurement:
1.
2.
3.
4.
5.
6.
7.
8.
•
Delivered defects
Total effectiveness through the process
Estimation accuracy
Adherence to schedule
Number of open customer problems
Time that problem remain “open”
Cost of non-conformance
Software reliability
debated these 8 focus
areas for a year
7 “Goals” for measurement:
–
–
–
–
–
–
–
1) improve project planning
2) increase defect containment
3) increase software reliability
4) decrease software defect density
5) improve customer service
6) reduce cost of non-conformance (to requirements)
7) increase software productivity
Specific Metric Definitions using G-Q-M approach
1.
Goal : Improve Project Planning
–
Question: How accurate is schedule estimation?
•
–
Question: How accurate is effort estimation?
•
2.
Metric: Schedule Estimation Accuracy = (actual proj. duration) / (estimated proj. dur.)
Metric : Effort Estimation Accuracy = (actual project effort)/(estimated proj. effort)
Increase Defect Containment
–
–
3.
Total pre-release Defect Containment Effectiveness
Defect Containment Effectiveness for each Phase
Increase Software Reliability
–
Failure Rate (in terms of execution time)
Decrease Defect Density (all normalized with “size” of product)
4.
–
–
–
–
–
–
In-Process Faults (errors and defects)
In-Process Defects
Total Released Defects
Total Released Defects caused by the delta incremental release
Total Customer Found defects
Customer Found Defect caused by the delta incremental release
Specific Metric Definitions by G-Q-M (cont.)
5. Improve Customer Service (using month as the interval)
–
–
–
–
6.
Reduce Cost of Non-Conformance
–
7.
Total New Problems (post release ones) Opened in a month
Total Problems that are still Open at end of month
Mean Age of Openness of Problems that are still Open at end of
month
Mean Age of Openness of Problems that are Closed at end of month
Cost (in dollars) spent in fixing post-release problems per month
Increase Software Productivity
–
–
Total Software Productivity in terms of software size per effort unit
Software Productivity (Incremental)
These metrics were shown in “10-up” management reporting charts
Metrics for Controlling Current (on-going) Project
• In-process control: ability to make decisions regarding the
current project based on current or projected “status”
– much of the data collected for the metrics used for earlier process
improvement can also be used for current in-progress project &
additionally others include:
• Life cycle phase and schedule tracking by date
• Cost value tracking: estimated cost, budgeted, actual, earned value
where earned value = (total budgeted cost of the completed activities)/
(total budgeted cost)
• Requirements changes over time tracking
• Design progress tracking in terms of number of requirements designed
• Fault “type” tracking to provide feedback for testing and support
• Estimated remaining defects (using Rayleigh curve)
• Review/inspection effectiveness tracking
• problem severity/priority of both closed and open problems
Software Metric Infrastructure at Motorola
• Metrics Working Group - across company (lasted several
years) and performed the following activities:
–
–
–
–
–
–
–
Clarifying metric definitions, documentation, etc.
Provide training and consulting
Guide in specifying measurement automation and associated tools
Provide support for analysis of data and recommendations making
Championing some process (e.g. Defect Prevention Process)
Guide the actual implementation of software metrics
Aid in assessing software measurement technology (e.g. customer
satisfaction measurement)
• Metric Users Group
– feedback and share experiences
• Software Metric Symposium
Experiences and Lessons Learned at Motorola
•
Start with a small set of, possibly “imperfect,” metrics to address areas
of improvement and evolve
•
As data are collected, start using them for in-progress project control.
•
Keep the metric data close to the source where the context of the
information is well understood
•
Software project requires more than one single metric; a set of metrics is
needed
•
Cost of software metrics initiative is about 1% of the total software
resources
•
There is a marked change in attitudes towards quality and processes.
Remember CMM Level 4&5 ? ---- this is the type of measurements
that must be carried out - - well defined and deployed metrics
Conclusion
• Motorola has been successful in implementing
a company wide software metric
– “minimal” resources
– several areas ( cost reduction, improved estimation,
ship-acceptance criteria, customer sat., etc.)
benefited
• Metrics can only show potential or real
“problem areas” -- YOU must take the
corrective actions !
Remember CMM level 5? ---- able to take actions - - improvements through change management processes