Transcript SE 532 Software Quality Management

Project Measurement
Source: Practical Software Measurement
John McGarry, et.al.
Measurement
“If you can’t measure it,
you can’t manage it”
Tom DeMarco
6/19/2007
SE 652- 2007_6_19_PSMp1.ppt
2
Fundamentals
Don’t try to measure everything
Align measures with:
Project goals & risks (basic survival mode)
Process improvement areas (continual improvement mode)
Define measurement program up front
Monitor continuously & take action where needed
6/19/2007
SE 652- 2007_6_19_PSMp1.ppt
3
Applications
Improve accuracy of size & cost estimates
Improve quality
Understand project status
Produce more predictable schedules
Improve organizational communication
Faster, better informed management decisions
Improve software processes
6/19/2007
SE 652- 2007_6_19_PSMp1.ppt
4
Basic In-Process Measurement Examples
Schedule
Earned Value vs. Planned Value
Schedule Variance
Development
Task completion
Actual code completed vs. planned
Project End Game
Defect Creation vs. Closure
Variations: severity
System Test
% Testing Complete
Variations: passed, failed, blocked
Test Time / Defect
Test Coverage (vs. requirements, white box code coverage)
6/19/2007
SE 652- 2007_6_19_PSMp1.ppt
5
Process Improvement Measurement Examples
Quality
Defect density
Post Deployment defect density
Inspection Effectiveness
Defects / inspection hour
Estimation Accuracy
6/19/2007
SE 652- 2007_6_19_PSMp1.ppt
6
Case Study 1: The mature product
Project characteristics
Release ‘N’ of successful product
Mature technology
Well understood requirements
Tenured team
Realistic schedule, adequate staff & budget
Project Goals?
Project Risks?
Development Strategy?
Measurement Plan?
6/19/2007
SE 652- 2007_6_19_PSMp1.ppt
7
Case Study 2: The Schedule Challenged
Project
Project characteristics
Significant amount of new capability
“Immovable” end date
Mature technology
Tenured team
Adequate staff & budget
Inadequate schedule
Project Goals?
Project Risks?
Development Strategy?
Measurement Plan?
6/19/2007
SE 652- 2007_6_19_PSMp1.ppt
8
Case Study 3: The Technology Challenged
Project
Project characteristics
Several new “bleeding edge” technologies
Tenured team
Adequate staff & budget
Schedule constrained
Project Goals?
Project Risks?
Development Strategy?
Measurement Plan?
6/19/2007
SE 652- 2007_6_19_PSMp1.ppt
9
Why Measure?
Support short & long term decision making
Mature software organization (CMMI level?) uses measurement to:
Plan & evaluate proposed projects
Objectively track actual performance against plan
Guide process improvement decisions
Assess business & technical performance
Organizations need the right kind of information, at the right time to make
the right decisions
6/19/2007
SE 652- 2007_6_19_PSMp1.ppt
10
Measurement in Software Lifecycle
Plan
Do – carry out change
Check – observe effects of change
Act – decide on additional areas for improvement
Repeat
Considerations: Cost, schedule, capability, quality
6/19/2007
SE 652- 2007_6_19_PSMp1.ppt
11
Measurement Psychological Effects
Measurement as measures of individual performance
Hawthorne Effect
Measurement Errors
Conscious: rounding, pencil whipping (ie. False data entry)
Unintentional: inadvertent, technique (ie. Consistent)
6/19/2007
SE 652- 2007_6_19_PSMp1.ppt
12
Use of Measures
Process Measures – time oriented, includes defect levels, events & cost elements
Used to improve software development & maintenance process
Product Measures – deliverables & artifacts such as documents
includes size, complexity, design features, performance & quality levels
Project Measures – project characteristics and execution
includes # of developers, cost, schedule, productivity
Resource Measures –resource utilization
includes training, costs, speed & ergonomic data
6/19/2007
SE 652- 2007_6_19_PSMp1.ppt
13
Glossary
Entity - object or event (e.g. personnel, materials, tools & methods)
Attribute - feature of an entity (e.g. # LOC inspected, # defects found, inspection time)
Measurement - # and symbols assigned to attributes to describe them
Measure – quantitative assessment of a product/process attribute (e.g. defect density, test pass
rate, cyclomatic complexity)
Measurement Reliability – consistency of measurements assuming nochange to method/subject
Software validity – proof that the software is trouble free & functions correctly (ie. high quality)
Predictive validity – accuracy of model estimates
Measurement errors – systematic (associated with validity) & random (associated w/ reliability)
Software Metrics – approach to measuring some attribute
Defect – product anomaly
Failure – termination of product’s ability to perform a required function
6/19/2007
SE 652- 2007_6_19_PSMp1.ppt
15
PSM Measurement Process
Measurement Plan
Information need – e.g.:
What is the quality of the product?
Are we on schedule?
Are we within budget?
How productive is the team?
Measurable Concept
Measured entities to satisfy need (abstract level: e.g. productivity)
Measurement Construct
What will be measured? How will data be combined? (e.g. size, effort)
Measurement Procedure
Defines mechanics for collecting and organizing data
Perform Measurement
Evaluate Measurement
6/19/2007
SE 652- 2007_6_19_PSMp1.ppt
16
Measurement Construct
Decision Criteria
Indicator
Analysis Model
Derived Measure
Derived Measure
Measurement Function
Base Measure
Measurement method
Attribute
6/19/2007
Base Measure
Measurement method
Attribute
SE 652- 2007_6_19_PSMp1.ppt
17
Indicator
Derived Measure
Attributes
Base Measure
Base Measure
Attribute
Attribute
Derived Measure
Attribute
Distinguishable property or characteristic of a software entity
(Entities: processes, products, projects and resources)
Qualitative or Quantitative measure
6/19/2007
SE 652- 2007_6_19_PSMp1.ppt
18
Indicator
Derived Measure
Base Measure
Base Measure
Base Measure
Attribute
Attribute
Derived Measure
Measure of an attribute (one to one relationship)
Measurement method
Attribute quantification with respect to a scale
Method type
Subjective (e.g. high, medium, low), Objective (e.g. KLOC)
Scale
Ratio
Interval
Ordinal
Nominal
Unit of measurement
e.g. hours, pages, KLOC
6/19/2007
SE 652- 2007_6_19_PSMp1.ppt
19
Indicator
Derived Measure
Indicator
Derived Measure
Base Measure
Base Measure
Attribute
Attribute
Derived Measure
Derived Measure
Function of 2 or more base measures
Measurement Function
Algorithm for deriving data (e.g. productivity = KLOC/developer hours)
Indicator
Estimate or Evaluation
Analysis Model
Algorithm / calculation using 2 or more base &/or derived measures +
Decision Criteria
Numerical thresholds, targets, limits, etc.
used to determine need for action or further investigation
6/19/2007
SE 652- 2007_6_19_PSMp1.ppt
20
Indicator
Measurement Construct
Examples
Derived Measure
Base Measure
Base Measure
Attribute
Attribute
Derived Measure
Productivity
Attributes:
Base Measures:
Derived Measure:
Analysis Model:
Indicator:
Hours, KLOC
Effort (count total hrs), Size (KLOC counter)
Size / Effort = Productivity
Compute Mean, compute std deviation
Productivity: mean w/ 2  confidence limits
Quality
Attributes:
Base Measures:
Derived Measures:
Indicator:
measured defect rate
6/19/2007
Defects, KLOC
# Defects (count defects), Size (KLOC counter)
# Defects / Size = Defect Rate
Defect rate control: baseline mean, control limits &
SE 652- 2007_6_19_PSMp1.ppt
21
Indicator
More Measurement Construct
Examples
Derived Measure
Base Measure
Base Measure
Attribute
Attribute
Derived Measure
Coding
Base Measure:
Schedule (w.r.t. coded units)
Derived Measure:
Planned units, actual units
Analysis Model:
Subtract units completed from planned units
Indicator:
Planned versus actual units complete + variance
6/19/2007
SE 652- 2007_6_19_PSMp1.ppt
22
Indicator
Class Measurement Construct
Examples
Derived Measure
Base Measure
Base Measure
Attribute
Attribute
Derived Measure
Coding
Base Measure:
Derived Measure:
Analysis Model:
Indicator:
6/19/2007
SE 652- 2007_6_19_PSMp1.ppt
23
Measurement Planning
Identify Candidate Information Needs
Project Objectives
Cost, schedule, quality, capability
Risks
Prioritize
One approach: probability of occurrence x project impact = project exposure
e.g.
Schedule
Budget
Reliability
Dependencies
Product Volatility
6/19/2007
SE 652- 2007_6_19_PSMp1.ppt
24
PSM Common Information Categories
Schedule & Progress
Resources & Cost
Product Size & Stability
Product Quality
Process Performance
Technology Effectiveness
Customer Satisfaction
6/19/2007
SE 652- 2007_6_19_PSMp1.ppt
25
PSM Common Information Categories
Measurement Concepts
Schedule & Progress
- milestone dates/completion, EV/PV
Resources & Cost
- staff level, effort, budget, expenditures
Product Size & Stability
- KLOC/FP, # requirements, # interfaces
Product Quality
- defects, defect age, MTBF, complexity
Process Performance
- productivity, rework effort, yield
Technology Effectiveness - requirements coverage
Customer Satisfaction
- customer feedback, satisfaction ratings,
support requests, support time, willingness to repurchase
6/19/2007
SE 652- 2007_6_19_PSMp1.ppt
26
Select & Specify Measures
Considerations
Utilize existing data collection mechanisms
As invisible as possible
Limit categories & choices
Use automated methods over manual
Beware of accuracy issues (e.g. timecards)
Frequency needs to be enough to support ongoing decision making
(alternative: gate processes)
6/19/2007
SE 652- 2007_6_19_PSMp1.ppt
27
Measurement Construct
Measurement Construct
Information Need
Measurable Concept
Relevant Entities
Attributes
Base Measures
Measurement Method
Type of Method
Scale
Type of Scale
Unit of Measurement
Derived Measures
Measurement
Function
Indicator
Analysis Model
Decision Criteria
6/19/2007
SE 652- 2007_6_19_PSMp1.ppt
28
Project Measurement Plan Template
(from PSM figure 3-10, p 56)
Introduction
Project Description
Measurement Roles, Responsibilities & Communications
Description of Project Information Needs
Measurement Specifications (i.e. constructs)
Project Aggregation Structures
Reporting Mechanisms & Periodicity
6/19/2007
SE 652- 2007_6_19_PSMp1.ppt
29
Cycle 1 Measurement Plan
Identify cycle 1 risks & information needs
Pick 4 measures & create measurement constructs
Document in a measurement plan per previous slide
6/19/2007
SE 652- 2007_6_19_PSMp1.ppt
30