Transcript INFO 636 Software Engineering Process I Prof. Glenn Booker Week 9 – Quality Management INFO636 Week 9 www.ischool.drexel.edu.

INFO 636
Software Engineering Process I
Prof. Glenn Booker
Week 9 – Quality Management
INFO636 Week 9
1
www.ischool.drexel.edu
Software Quality Management
• The PSP focuses on managing defects to
produce a quality product
• This, in turn, helps improve productivity
– Productivity still falls as the size of a system
increases, due to its increasing complexity
• To improve further, must improve your
process quality
INFO636 Week 9
2
www.ischool.drexel.edu
What is Quality?
• Crosby considered software quality to be
“conformance to requirements”
• Meeting user needs is often key to
determine quality
• We can distinguish between the quality of
the product created, and the quality of the
process used to create it
INFO636 Week 9
3
www.ischool.drexel.edu
Product Quality
• A software product must provide the
functions defined in its requirements, and
must be a functioning application
– Beyond that, how sensitive the user is to other
aspects of usability, reliability, compatibility,
etc. varies wildly
– Poor quality products tend to have a lot of
focus on testing to find defects
INFO636 Week 9
4
www.ischool.drexel.edu
Process Quality
• Process quality focuses on meeting
the developer’s needs
• A quality process supports creating good
products
– Process consistency and repeatability are
generally critical
– Efficiency and usability are less obvious
objectives
INFO636 Week 9
5
www.ischool.drexel.edu
Cost of Quality
• Quality is a big cost issue for software
• Each defect has costs for
–
–
–
–
–
–
–
–
detecting it
finding the source
determining a solution
fixing the problem (req’ts, design, and/or code)
reviewing the fix
testing the fix to see if it works
testing the fix to check for regression problems
updating documentation to reflect the fix
INFO636 Week 9
6
www.ischool.drexel.edu
Cost of Quality
• The relative cost of finding and correcting
a defect increases through the life cycle
– During requirements or design phases,
a defect has a relative cost of about 1
– Coding and testing phases, cost 10-40
– After release to the customer, cost 100-1000
INFO636 Week 9
7
www.ischool.drexel.edu
Cost of Quality
• Many of you also noted that the time to fix
defects found in testing or later is much
higher than those found earlier (see
pp. 276-7)
• PSP experience has shown that reviews
are twice as efficient at finding defects as
testing
INFO636 Week 9
8
www.ischool.drexel.edu
Testing Schedules
• Many organizations focus on getting into
system testing quickly, as a sign of
progress
• However, more time in requirements and
design would produce higher quality code
which needs less testing time
INFO636 Week 9
9
www.ischool.drexel.edu
Cost of Defect Removal
• Typical developers inject (create) about
100 defects per thousand LOC
– About half are found by compiler
– The rest need to be found by inspection or
testing
• Using the PSP, about 70% of defects
entering inspection are found
INFO636 Week 9
10
www.ischool.drexel.edu
Cost of Defect Removal
• Inspection and review average about ½
hour per defect, whereas testing averages
8-10 hours per defect – so the cost benefit
of inspection becomes clear
INFO636 Week 9
11
www.ischool.drexel.edu
An Example
• Consider a 50,000 LOC project
– About 5000 defects are initially created, but
2500 are found by the compiler (and are
presumably fixed immediately)
– Without reviews or inspections, those 2500
defects would take at least 2500*8 = 20,000
hours to find and fix using testing alone
INFO636 Week 9
12
www.ischool.drexel.edu
An Example
– Using inspections, we would expect to
find 1750 of those defects, and fix them
in ½ hour each for 875 hours
– The remaining 750 defects would be found by
testing, and fixed in 750*8=6000 hours
• Total defect removal time with inspection
and test is 6,875 hours, versus 20,000
hours for testing alone
– That saves $1.3M at $100/hr
INFO636 Week 9
13
www.ischool.drexel.edu
Cost of (Low) Quality
• Poor quality can be measured to
determine its impact on the project
– Failure cost = the cost of finding and repairing
failures
– Appraisal cost = the cost of conducting
reviews (called), and testing which finds
no defects
– Prevention cost = the cost of preventing the
failures from occurring
INFO636 Week 9
14
www.ischool.drexel.edu
Cost of Quality (COQ)
• For the PSP we simplify these to
– Failure cost = time for compile and test
– Appraisal cost = time for reviews and
inspections
– Prevention costs are low, since we have few
projects to examine for defect trends
• Based on this, define COQ measures
INFO636 Week 9
15
www.ischool.drexel.edu
COQ Measures
– Failure COQ = 100*(compile time + test
time)/(total devel time)
– Appraisal COQ = 100*(design review time +
code review time)/(total devel time)
– Total COQ = Failure COQ + Appraisal COQ
– Appraisal cost as % of total quality =
100*(appraisal COQ)/(total COQ)
– A/FR ratio = Appraisal COQ/(Failure COQ)
INFO636 Week 9
16
www.ischool.drexel.edu
COQ Measures
• The purpose of these measures is to help
look for trends in quality over several
projects
– Hope to see higher ‘appraisal cost as % of
total quality’ over time
– This should lead to lower defect rates for later
projects
INFO636 Week 9
17
www.ischool.drexel.edu
Quality Strategy
• Important to keep product and process
measures separate – both are important
• Avoid productivity measures by
themselves – they can help with planning,
and be compared to other measures, but
don’t use them as a sole objective
INFO636 Week 9
18
www.ischool.drexel.edu
Process Yield
• Total process yield is helpful – the percent
of defects removed before test
• You can also evaluate yield for every life
cycle phase, but only after hidden defects have
been later discovered
• The life cycle is a filtering process
– Each phase has some kind of step to help find
and remove defects to keep from feeding them into
the next phase
INFO636 Week 9
19
www.ischool.drexel.edu
No One Answer
• The PSP is a personal process for many
reasons – one is that there is no one best
way to find and remove defects
– Need to look for the best ways for you to
improve your work
– Then reevaluate your strategy periodically
INFO636 Week 9
20
www.ischool.drexel.edu
Benchmarking
• It can be helpful to compare your product
and process measures to other people’s,
or to industry standards
– For example, plot the A/FR ratio over time
• It should increase (p. 288)
– Or look at A/FR versus the yield
– Or Test defects per kLOC versus A/FR
INFO636 Week 9
21
www.ischool.drexel.edu
Benchmarking
• Productivity can be compared to other
measures, such as yield, to look for trends
• Yield acts like the filter part of the life cycle
process
– Each step’s yield reduces the number of
defects to be found later, so a small
improvement in yield can be significant
INFO636 Week 9
22
www.ischool.drexel.edu
Injection and Removal
• The key to understanding your software
development process is to balance the
injection of new defects with activities to
detect and remove them
• To achieve greater control, need to
examine the factors which cause defects
to occur
INFO636 Week 9
23
www.ischool.drexel.edu
Defect Causes
• Education – didn’t know how to do it
• Communication – weren’t informed about
something
• Oversight – forgot to do something
• Transcription – knew what to do, but made
a mistake
• Process – the process misled your actions
INFO636 Week 9
24
www.ischool.drexel.edu
Yield Management
• While the overall process yield doesn’t
care if you have 80% review yield, and
40% test yield, or 40% review and 80%
test, the costs and A/FR ratio of those
situations are quite different
• This is why so much emphasis is placed
on the quality of early reviews (see
recommendations on pp. 298-300)
INFO636 Week 9
25
www.ischool.drexel.edu
Defect Prevention
• Finding and fixing defects is purely
defensive
• To be more proactive, need to prevent
defects from being created
• Analyze the types of defects which occur
frequently in your experience
• Determine the causes of common defect
types (see slide 24 again)
INFO636 Week 9
26
www.ischool.drexel.edu
Defect Prevention
• Then look for ways to prevent those
defects from being made again, e.g.
– Improve review checklists
– Get additional instruction
– Conduct more detailed design
– Get independent reviews
• And change your processes accordingly
INFO636 Week 9
27
www.ischool.drexel.edu
Defect Prevention
• The critical follow-up is to re-examine your
defects later, and see if your changes
helped or not
• Incidentally, defect prevention and
continuous process improvement are the
highest levels of process maturity
– CMM and CMMI level 5 activities
INFO636 Week 9
28
www.ischool.drexel.edu