Transcript Why Software Metric Programs Fail

ผลผลิต โปรแกรมคอมพิวเตอร์ ไม่ มมี าตรฐาน
Why Software Metric Programs Fail
Jim Pederson
Software Process Improvement Network (SPIN)
April 26, 2001
Examples of Software Models
Success Models
• Win-Win • Business Case Analysis
Process Models
• Software Warranties • QFD
• 10X • Six Sigma
• Award Fees . IKIWISI
• JAD
•Spiral
•Waterfall
•Open Source
•Business Process Reengineering
•CMM’s • Peopleware
•COCOMO
•IPT’s • Agile
• COCOTS
•Evolutionary
•UML
•CORBA •COM
•Requirements
•Architectures
•Product Lines
•OO Analysis & Design
•Domain Ontologies
•COTS • GOTS
• Checkpoint
• System Dynamics . Metrics . ilities
• Simulation and Modeling . Environment
Property Models
• Some model assumptions conflict
Product Models
“Software Engineering” Definition
- Based on Webster definition of “engineering”
• The application of science and mathematics by
which the properties of software are made useful
to people
• Includes computer science and the sciences of
making things useful to people
– Behavioral sciences, economics, management
sciences
I. What Is Software Engineering?
• Is it :
–
–
–
–
–
Programming ?
Computer science ?
Engineering ?
A real discipline ?
A professional activity ?
Some Basic Definitions
• Software -- Computer programs,
procedures, and possibly associated
documentation and data pertaining to the
operation of a computer system.
• Engineering -- Application of systematic,
disciplined, quantifiable approach to some
process.
Resulting Definition Of
Software Engineering
• Definition -- The application of systematic,
disciplined, quantifiable approaches to
software development, operation and
maintenance.*
* - IEEE Standard 610.12-1990, IEEE Standard Glossary of Software
Engineering Terminology, IEEE Standards Collection Software
Engineering, IEEE (1997).
Why do we need it ?
• The “Software Crisis”
– First identified at NATO Conference,
Garmisch, Germany, 1968
– Characterized by software which is
•
•
•
•
of poor quality
over budget
late
Naur & Randell (1969)
Why Software Projects Fail
- Average overrun: 89.9% on cost, 121% on schedule, with 61% of content
A Hegelian View of Software Engineering
Evolution
Theses
Engineer
Software
like
Hardware
Many defects
Formality,
Waterfall
Compliance
Software
Value-Add
PlanDriven
Software
Maturity
Models
COTS
Integrated
Sw-Systems
Engineering
Soft
SysE
Process Overhead
Syntheses
Scalability,
Risk Mgmt.
Productivity;
Reuse;
Objects;
Peopleware
Risk Mgmt.
Domain Engr.
Risk-Based
Agile/Plan
-Driven
Hybrids;
Model-Driven
Development
Value-Based
Methods;
Collaboration;
Global
Development;
Enterprise
Architectures
Scalability
Prototyping
Antitheses
Software
Differences,
Engineer
Shortages
Software
as Craft
1950's
1960's
Global
Systems
of
Systems
Time to Market,
Rapid Change
1970's
1980's
1990's
Agile
Methods
2000's
2010's
Autonomy; BioComputing
What are Metrics and Why do Them?
• Supports Decision Making (or at least they should)
– Must be Timely and Representative
– Must be accepted by decision maker(s)
• Comparison to Expectation
– May be either stated or inferred
– At higher levels, expectations must be quantitatively defined
• Are frequently done to impress
– Assessments
– Internal Initiatives
– Customers
A Layered Architecture View of
Software Metrics
What’s Needed to do Metrics
• Analysis Layer
– Method or process of using data to make decisions
• Presentation Layer
– Metric Outputs (Charts and Reports)
• Data Layer
– Underlying data that supports the presentation layer
• Process Layer
– The process that the data represents
– Must also address the capture of data from the process
Metric Analysis Layer
• Typically driven by deviation from expected outcome
– Process Capability vs. Tolerance vs. “Seat of the Pants”
• Deviations should be documented and managed
– Analysis, Disposition, Implementation, Follow-up (Plan-Do-Check-Act)
• Various Quality Tools should be used to support analysis
– Fishbone, Correlation Analysis, DOE (in a couple of rare cases)
• Unique metrics can be used to assess corrective action
• In SEI model, SEPG would play active role.
– Relates project to organizational process (DPP, PCM)
• Common or Repetitive Analysis can flow back to Presentation Layer
• Willingness must exist on the decision maker(s) to use data
– Decision maker generally refers to project or functional management.
– Could also refer to work team
Metric Presentation Layer
• Different Presentation for different purposes and customers
– Project Metrics (most common and easiest to do)
– Organizational Metrics (integrates multiple projects)
– Process Metrics (crosses project or domains and is ongoing)
• Expected Outcomes should be depicted along with actual data
– Initially this is an intuitive process (extrapolate to end point)
– Defining expected outcomes requires historical process knowledge
• Application of SPC is difficult and somewhat overrated
–
–
–
–
Common SPC applications represent continuous processes
Software data varies around the project lifecycle
Some sub-process data is fairly continuous but for short periods of time
Some data (primarily cost related) is self normalizing but is driven from
lower level data that isn’t.
– Generally avoid percentage based tolerances
Metric Presentation Layer (cont.)
• Deciding What to Measure
– It’s difficult to know what’s worth measuring until you measure it
– Don’t define metrics based on what is easy to measure
– Establish metrics based on value to the whole
• Interesting Observations
– Staffing level is a simple metric that is highly predictive of project
outcomes and also shows interaction between projects.
– Cost metrics must be closely linked to meaningful schedules to have
substance (earned value).
– Early schedule slippages don’t recover – they generally get worse.
– Number or iterative releases is the key variable in modeling project
performance.
– Most common corrective action is to re-plan project.
– Bad project outcomes are frequently linked directly to bad estimating.
– Estimating, planning, and status monitoring (metrics) are intertwined.
Metric Data Layer
• Almost always underestimated and under scoped
• From management perspective, this often can be equated to the
“then a miracle occurs” block on a flow chart.
• Some aspects of data collection can be done manually but this is
generally undesirable
– Person dependent
– Recurring costs
– Inaccurate data that is difficult to reproduce
• Metric data automation needs to be approached as an IT
development effort
• Use of commercial tools doesn’t avoid this problem at all – it can
make it even more complex in some respects.
Metric Data Layer – Data Sources/Types
• Schedule
– Generally tracked in MS Project which is easily integrated
– Quality of schedule data is frequently poor
• Cost
– Project costs are generally captured in separate cost accounts
– Progress should link to quantitative schedule data but frequently doesn’t
• Defects and Change requests
– Universally kept but frequently doesn’t cover much of the life cycle
• Requirements / Design
– Either tracked in COTS tool or spreadsheet
– Can supersede certain documents by handling content as data
• Configuration
– Several common commercial tools
• Testing
– Various COTS tools and test files - can be treated as data to some
extent
Metric Data Layer – Data Model
Estimating
Project Tracking
Project Schedule (Generally MS Project)
Financial Performance Data (Cost Accounting System)
Requirements (RM Tool)
Design Definition (RM and Modeling Tools)
Coding/Implementation (CM Tool)
Test Definition (RM Tool)
Test Files and Results
Change Requests / Defects
Change Management
Metric Process Layer
• The data is only as good as the underlying process
– From an development perspective, and application (data layer) can
only model the process.
– If the process is highly inconsistent, it will greatly limit the availability
and validity of metrics data and outputs.
• Metrics can show process variation
– Look there first when investigating variations
• Process also limits the degree to which metric data can capture
and reflect the entire life cycle
• Metrics need to match the process
– Sophisticated metrics and unsophisticated process will prove to be
meaningless and a waste of time
Software Metrics
State of the Practice
Current State of the Practice
• Wide gap exists between theory and practice
– Theoretical side seems to be focused on academic credibility
– Practitioner issues are driven by organizational politics and culture
• Gap is caused by failure to address the whole problem
– Few people will be equally comfortable with all layers
– Move away from the comfort zone
Theoretical Perspective
• Driven by Academia, Consultants, and specialists from larger
companies (primarily aerospace)
– Focuses on presenting and analyzing data with emphasis on
application of quality tools and statistical methods
– Texts and presentation material on the subject tend to assume that
the data exists
• Tool features are heavily influenced by these specialists leading
to increased complexity and underused features
• Cultural and Human Factor issues have been under-addressed
– Depth of problem and relation to upper management values
• Theoretical perspective is weak in addressing many practical
issues that impact metric deployment
– Reflects a lack of “real world” experience
Practitioner Perspective
• Generally someone within in an organization that has knowledge of
process, statistics, and data.
– Breadth of knowledge makes those to choose from limited
• Generally not the project manager of functional manager
– Delegated task
– Practitioner generally lacks adequate authority to act on data
• Practitioner faces numerous constraints
–
–
–
–
Process limitations
Lack of data infrastructure
Poor tool integration
Interface problems between functional group
• Relevancy of effort is an ongoing issue
– Decision maker interest in and use of data
– Maintenance of data on the part of developer population
– Little victories and value of simplicity
The Use and Abuse of Metrics
(Common Interpretational Issues)
Common Misapplications – Product Size
• Size is a key metric because it drives effort and is a normalizing
data element for other metrics (I.e. defects per KSLOC)
• Size can be measured different ways
– SLOC, Function Point, Feature Point, other
• Measuring size is inherently difficult
– Expertise, existing documentation, software logic
• New and modified effort is what’s important (I.e. new/modified
SLOC) – not the total size
• Relatively few organizations can measure new and modified output
• Measuring new and modified output is always subject to a certain
amount of inaccuracy
• Size assessment can tend to reward inefficient design and
implementation in that bigger is generally interpreted as better.
Common Misapplications - Defects
• Measurement of defects is meaningless apart from some
assessment of size or effort (normalization)
• Process characteristics frequently limit this metric to the end of the
project life cycle
– Hard to use as a predictive metric
– Always favors the later part of LC regardless of process sophistication
• What’s included?
– Peer Reviews, Design Reviews, developer found, documentation
• Lower than expected totals can be deceiving
– Not found, Not documented, behind schedule, plan inaccurate
• Higher than expected totals can also be deceiving
– Higher capture rate, greater than planned complexity, scope growth
• True assessment of defects should consider phase found and
severity
Common Misapplications - Progress
• Progress can be assessed either from schedule or financial data
• Financial progress (earned value) should be linked directly to
schedule but frequently isn’t. (also referred to as Earned Value)
– The less tangible the linkage, the more arbitrary the metric
• Schedule progress can be approached two ways
– Very detailed schedule (inch stones)
– General schedule supported by detailed measure to assess progress
• Either way, maintaining detailed progress information is difficult and
needs the active support of the entire team
– Everyone performing work must contribute data
– Picking “low hanging fruit” can bias metric (avoid straight line projections)
• Progress metrics are the simplest to present and are probably the
most common type of metric.
• They are also the hardest to support with data and can easily
become highly arbitrary.
Common Misapplications - Productivity
• Can be based either on dollars or hours
– Hours is better because they remain constant
• Undefined size growth can make productivity appear poor
– Size is difficult to assess during implementation
– Unclear requirements can make size very difficult to predict up front.
• Inefficient design (code bloat) can make productivity appear better
than it really is
• Resource utilization has a sharp impact on productivity
–
–
–
–
Functions/objects are normally assigned to individuals
Work scope by function or object is rarely constant
Resources don’t directly expand or contract to fit work scope
Significant productivity increases (as much as 4x) have been realized
simply by adding and/or redistributing work
• Never assume full time resource availability
– Unplanned tasks
– Maintenance, Level of Effort, Overhead
Common Misapplications - Rework
• Can be calculated a variety of ways
–
–
–
–
Actual hours by problem report
Estimated hours by PR based on PR attributes
Separate rework charge number
Tasks relative to schedule milestones (waterfall model)
• None of these methods is particularly accurate and all are subject
to manipulation.
• Defining rework can be contentious
– Non-functional vs. non-optimized
– Classification of unplanned builds / releases
• Some development strategies can significantly increase rework
while improving overall efficiency
– Re-use, auto code, auto translation
Common Misapplications - Process
• Separating impact of process change from other project factors is
extremely difficult.
– Readily subject to manipulation
– Too many independent variable
– ANOVA or DOE is the only way to do this convincingly
• Some process changes can be valuable but meaningless
– Should start with overall process performance and drill down
– Don’t look for target of opportunity and try to make it fit
• Most processes are not continuous
– Either start and stop or are cyclical
• Process metrics cross domains and projects
– Origin of data is not homogenous
– Be mindful of potential sub-populations
Common Misapplications – Giving Up
• Application of software metrics is difficult
– Subject to inaccuracy
– Subject to manipulation
– Easy to become sarcastic about overall value
• Yet not everything is difficult
– Relatively simple yet meaningful metrics exist
• Problems are not insurmountable
– Knowing potential problems helps you avoid them
• Appearance of use vs. actual deployment
– Striving for appearance ruins credibility
• Consistency vs. Accuracy
– Consistency is at least as important as accuracy
• Breadth of usage
– Data issues tend to be self correcting when data is used
Conclusions on Why Metric Programs
Succeed and Fail
Why Metric Initiatives Fail
• Failure to adequately address Data Infrastructure
–
–
–
–
Assumption that data exists when it doesn’t
Failure to acknowledge resource requirements to manage data
Deployment of Commercial Tools without adequate design effort
No method to meaningfully assess product size (new and modified) or
track effort with enough detail to use
• Relationship of Process to Data not understood
–
–
–
–
Limitations on availability of data
Limitations on span of life cycle that data can represent
Assumption that process is consistent
Potential of tools to drive process behavior (both good and bad)
Why Metric Initiatives Fail (cont.)
• Failure to overcome Cultural Barriers
– Engineering / Developer population inherently hard to manage
– Value placed on reacting to problems (fire fighting) as opposed to
avoiding them in the first place.
– Quantitative management not an organizational value
– General fear of metrics (difficult to implement)
• Management Support Issues
–
–
–
–
–
–
–
Lack of top down support (upper management)
Primary focus on certification instead of deployment
Lack of demonstrated support (middle and lower management)
Unwillingness to manage employees to implement process
Failure to do adequate planning (generally goes back to estimating)
Unwillingness to use data to make decisions
Re-focus of metric issues on data and presentation (excuses)
How Metric Initiatives Succeed
• Define Success to match Organizational Goals
– Know what you want out get out of metrics
– Establish Reasonable Expectations
– Limited successes are better than large scale failure
• Match Metrics to Process
– Process must be defined and relatively consistent as a starting point
– Take process compliance seriously
– Don’t define a more sophisticated process than you intend to
implement
• Have a Plan
–
–
–
–
Identify ALL tasks and develop a schedule
Assign resources and manage dependencies
Manage like a real project
Don’t forget sustaining effort to maintain metric applications
How Metric Initiatives Succeed (cont.)
• Create a Common Vision
– Must extend from top management through all management levels
and technical leads
– Any break in the chain could cause entire effort to fail
– Reinforce positive behavior – create enthusiasm
– Forecast projects as opposed to sell them
• Create Ownership
–
–
–
–
–
–
Use of specialist is OK but don’t push the effort entirely off-line
Use the data to make decisions, question rationale of decisions
Don’t commit to the impossible to get work or maintain staffing
Encourage work team involvement
Assess reactive vs. non-reactive activities
Develop culture that values planning and quantitative decision making