INFO 631 Prof. Glenn Booker

Week 1 – Defect Analysis and Removal INFO631 Week 1 1 www.ischool.drexel.edu

Motivation

• Much of software measurement is devoted to improving the quality of the product • To do so, it helps to understand how, when, and why mistakes are made during the software life cycle, resulting in defects • It is assumed that defects are queued for being fixed (removed) after they are detected INFO631 Week 1 2 www.ischool.drexel.edu

Defect Analysis Overview

• Defect analysis looks at: – When a defect was created (“injected”) – When a defect was found (“detection”) – What caused a defect (type of defect, and/or orthogonal defect classification) – How a defect was found (“triggers”) • Yes, there are often many terms for the same activity!

INFO631 Week 1 3 www.ischool.drexel.edu

Defect Creation (Injection)

• Defects may be created, detected, and removed during every phase of the software life cycle, including during maintenance – Most are created during requirements analysis, design, and coding – Testing and maintenance create relatively few defects INFO631 Week 1 4 www.ischool.drexel.edu

Waterfall Life Cycle

Requirements Analysis High Level Design Low Level (Detailed) Design Coding Unit Testing Component (Integration) Testing INFO631 Week 1 System Testing Maintenance 5 www.ischool.drexel.edu

Defect Detection

• Defects may be found: – During quality assurance activities between phases of the life cycle (such as during major reviews and inspections); and often – Within each phase (during preliminary reviews, peer reviews and testing activities) • Here we mostly consider the former (between phase activities) INFO631 Week 1 6 www.ischool.drexel.edu

Defect Detection

• Defects are found (discovered) by – Reviews and formal inspections • For requirements analysis, high level design, low level design, and coding – Testing • Unit testing • Integration testing • System testing • Acceptance testing INFO631 Week 1 7 www.ischool.drexel.edu

Origin of Defects & Type

• What kind of defects are made in each life cycle phase?

– Requirements - incorrect specification; missing requirements – High level design - design does not cover all requirements, or is inflexible – Low level design - mismatch between HLD and LLD; design does not cover all requirements INFO631 Week 1 8 www.ischool.drexel.edu

Origin of Defects & Type

– Coding - code errors – Integration – interface, compatibility problems – Unit testing - bad fixes – Component testing - bad fixes – System testing - bad fixes – Acceptance testing - bad fixes – Maintenance (enhancement) - bad requirements, design, coding, testing INFO631 Week 1 9 www.ischool.drexel.edu

Defect Causal Analysis

• Regardless of when defects are created or discovered, they may be analyzed to determine their cause – Causes may range from the mundane to the esoteric – Once identified, causes should be recorded to support defect prevention activities INFO631 Week 1 10 www.ischool.drexel.edu

Types of Defect Causes

• Some causes may include: – Unclear requirements – Incorrect architecture – Incomplete design – Unfamiliarity with programming language – Typos – Inadequate understanding of interfaces – Inadequate understanding of standards (e.g. TCP/IP, ODBC, SQL, etc.) INFO631 Week 1 11 www.ischool.drexel.edu

Types of Defect Causes

• and: – Poor documentation for legacy system – Design based on outdated requirements – Code written to outdated design – Poor design for future expansion (e.g. hardwired constants, buried assumptions, etc.) – Conflicting requirements – Bad fix of a previous problem – …and many more INFO631 Week 1 12 www.ischool.drexel.edu

Orthogonal Defect Classification

• ODC is a formal type of defect causal analysis • Classify defects by the type of defect, and the life cycle phase when it was created • This particular scheme is still somewhat experimental, but the basic concept is widely accepted INFO631 Week 1 13 www.ischool.drexel.edu

ODC - Defect Type

• A possible set of definitions: –

Function error

–affects capability, user, product or hardware interfaces, global data structures –

Assignment error

– errors with initialization of control blocks or data structure – –

Interface error

– errors in interacting with other components, modules, or or device drivers

Checking

– errors in validating data and values INFO631 Week 1 14 www.ischool.drexel.edu

ODC - Defect Type

– – – –

Timing/serialization

– errors in management of shared and real-time resources

Build/package/merge

– errors due to mistakes in library systems, change management, or version control

Documentation

– errors in publications and maintenance notes

Algorithm

– errors regarding efficiency or correctness that affect the task INFO631 Week 1 15 www.ischool.drexel.edu

Orthogonal Defect Classification

Type Functional Interface Checking Assignment Timing Build/package Documentation Algorithm Phase Caused Design Low Level Design (LLD) LLD or Coding Coding LLD Library tools Publications LLD INFO631 Week 1 16 www.ischool.drexel.edu

Defect Trigger

• Is a condition that allows a defect to surface (become visible); examples include: • Design conformance • Logic or data flow incomplete • Workload or stress (performance) • Boundary conditions (extremes) • Bug fix • Recovery timing • User code INFO631 Week 1 17 www.ischool.drexel.edu

Defect Removal Modeling

• Defect removal is critical to reducing (development or maintenance) cycle time and cost, and improving quality • Michael Fagan and Capers Jones are noteworthy authors • See, for example, “Fagan Inspections” INFO631 Week 1 18 www.ischool.drexel.edu

Defect Removal Model (DRM)

• Software development phase-based DRM covers – Defect injection into a phase – Defect removal during a phase – Defect carryover across phases – Effectiveness of defect removal within a phase INFO631 Week 1 19 www.ischool.drexel.edu

Defect Removal Model

• DRM is a quality management tool • Provides insights into the defect removal process and where this process might be improved • Used after a project is completed; provides

post mortem

information about that project • If defect removal process is similar for a new project, then the DRM can be used to improve the defect removal process of that new project INFO631 Week 1 20 www.ischool.drexel.edu

Defect Removal Model

• Defects are created (injected) by incorrect requirements gathering, analysis, design, coding, or by bad fixes – Any given defect can be removed in the phase in which it was injected or in a later phase • Remove defect means fix bug, redo specs and/or design as needed INFO631 Week 1 21 www.ischool.drexel.edu

Defect Removal Effectiveness

• Phase based defect detection activities – Requirements analysis & inspection – Design inspection – Code inspection, Build verification testing – Unit, string, integration, regression, system testing – Fix verification inspection & testing INFO631 Week 1 22 www.ischool.drexel.edu

Defect Removal Model

Defects Injected (new mistakes) Undetected Defects Net Defects Net Defects from previous Phase Life Cycle Phase Defect Detection (inspection) Bad Fix Defects Known Defects Unfixed Defects + Bad Fix Defects Fixed Defects

Applies to each life cycle phase

INFO631 Week 1 23 www.ischool.drexel.edu

Defect Removal Effectiveness

• For any given phase of defect removal

Removal Effectiveness = Defects present at removal operation X 100

• Defects present at removal operation – Defects found during removal operation + defects found later – May be determined

ex post facto

(after the fact), or using a statistical prediction model INFO631 Week 1 24 www.ischool.drexel.edu

Defect Removal Effectiveness

• Defect removal effectiveness = (# of defects found by inspection) / (# of defects originally present) *100 • Early detection percentage = (# of major inspection errors) / (# of major and minor errors) * 100 INFO631 Week 1 25 www.ischool.drexel.edu

Defect Matrix Assumptions

• Defects are removed in the same life cycle phase when they are found • No defects are knowingly left unfixed • No bad fixes – Or at least they are blended into the number of defects created in that life cycle phase INFO631 Week 1 26 www.ischool.drexel.edu

i x e d W h e n F o u n d / F Rqmts Top Lev

Design

Low Lev Design Coding Unit Testing Compon Testing System Testing Field Total Sample Defect Matrix — When Originated vs. When Found When Originated (injected, or created) Rqmts Top Lev. Design Low Lev Design Coding Unit Testing Compon Testing System Testing Field Total

0 49 6 12 21 20 6 8 122 681 42 28 43 41 8 16 859 681 114 43 61 24 16 941 223 261 72 40 939 1537 2 2 4 4 1 1 1 1 0 730 729 1095 332 387 111 81 3465 INFO631 Week 1 27 www.ischool.drexel.edu

Defect Removal Effectiveness

Defects

Created

this Phase Defects passed from

Previous

life cycle phases Life cycle Phase(s) Defects passed to

Next

cycle phase Defects

Found

& removed this Phase Next = (Previous + Created) – Found DRE = Found / (Previous + Created) * 100 life INFO631 Week 1 28 www.ischool.drexel.edu

High Level Design Effectiveness

• There are no requirements defects removed; 122 defects are passed to HLD • High (Top) Level Design (I0) Inspection Effectiveness – Defects found and removed at I0: 730 – Defects existing on step entry (escapes from requirements phase: 122 – Defects injected in current phase: 859 – E(I0) = 730/(122+859) x 100 = 74% INFO631 Week 1 29 www.ischool.drexel.edu

Low Level Design Effectiveness

• Low Level Design (I1) Inspection Effectiveness – Defects found and removed at I1: 729 – Defects existing on step entry (escapes from requirements phase and I0): 122+859-730 = 251 – Defects injected in current phase: 939 – E(I1) = 729/(251+939) x 100 = 61% INFO631 Week 1 30 www.ischool.drexel.edu

Code Inspection Effectiveness

• Code Inspection (I2) Effectiveness – Defects found and removed at I2: 1095 – Defects present on step entry (escapes from requirements phase, I0, and I1): 251+939-729 = 461 – Defects injected in current phase: 1537 – E(I2)= 1095/(461+1537) x 100 = 55% INFO631 Week 1 31 www.ischool.drexel.edu

Unit Testing Effectiveness

• Unit Testing (UT) Effectiveness – Defects found and removed at UT: 332 – Defects existing on step entry (escapes from previous phases): 461+1537-1095 = 903 – Defects injected in current phase (bad fixes): 2 – E(UT) = 332/(903+2) x 100 = 37% • Can follow the same pattern for the other testing phases and post-release INFO631 Week 1 32 www.ischool.drexel.edu

Summary Effectiveness Measures

• Overall Design & Coding Inspection Effectiveness – IE = (730+729+1095)/(122+859+939+1537) x 100 = 74% • Overall Effectiveness of all Testing activities – TE = (332+387+111)/(903+2+4+1)x100 = 91% • Overall Defect Removal Effectiveness of the development process (not including release) – DRE = (0+730+729+1095+332+387+111)/(122+859+ 939+1537+2+4+1) x 100 = 3384/3464*100 = 97.7% INFO631 Week 1 33 www.ischool.drexel.edu

Rayleigh Model

• The Rayleigh Model describes the number of defects which will be

discovered

, by development phase • It’s a special case of the Weibull family of distributions, which we’ll cover later INFO631 Week 1 34 www.ischool.drexel.edu

Rayleigh Model Assumptions

• Defect rate observed during development process is positively correlated with defect rate in field • Given the same defect injection rate, if more defects are discovered and removed earlier, fewer will remain in later stages, leading to fewer defects in the field INFO631 Week 1 35 www.ischool.drexel.edu

Rayleigh Model Inspections

• I0 = High Level Design Inspection • I1 = Low Level Design Inspection • I2 = Code Inspection • UT = Unit Testing • CT = Component (Integration) Testing • ST = System Testing • GA = after General Availability (release or Fielding of the system) INFO631 Week 1 36 www.ischool.drexel.edu

Rayleigh Model

Number of Defects Development Phase I0 I1 I2 UT CT ST GA

Each bar represents the number of defects

found

during that life cycle phase’s inspection activity INFO631 Week 1 37 www.ischool.drexel.edu

Desired Rayleigh Curve Trends

Effect of Early Defect Removal and Reducing Error Injection INFO631 Week 1 38 www.ischool.drexel.edu

Cost of Defect Removal by Phase

• Defect removal and rework is less costly the closer that the defects are found relative to the phase in which they are injected – Rework in the I0, I1, and I2 inspection levels can be 10 to 100 times less expensive than if it is done during formal testing – Reviews can reduce number of defects reaching testing phases by factor of 10 • These reductions cut testing costs by 50-80%, even including the review costs INFO631 Week 1 39 www.ischool.drexel.edu

Cost Effectiveness of DRM

• Relative cost of fixing a problem found in design/coding, testing, or after release are: 1:20:82 (Remus, 1983) 1:13:92 (Kan, 1989) • Cost of defect removal can be analyzed by inspection type, testing phase, defect severity, defect origin, etc.

INFO631 Week 1 40 www.ischool.drexel.edu