INFO 636 Software Engineering Process I Prof. Glenn Booker

Week 7 – Measurement INFO636 Week 7 1 www.ischool.drexel.edu

Measurement

• Measurements are critical, because

the act of measuring something focuses people’s attention on it

• We have already looked at several basic measures in the PSP – Now expand to a more general process for choosing measurements and how to present them INFO636 Week 7 2 www.ischool.drexel.edu

Measurement

• The approach presented here is a variation on the GQM approach cited in the text, called GQ(I)M – The (I) adds Indicators, which are the means used to present the measurements graphically – This approach is also used in INFO 630 INFO636 Week 7 3 www.ischool.drexel.edu

Why Care About Measurement?

• To quote Albert Einstein, "Not everything that counts can be counted and not everything that can be counted counts." • We are seeking to identify things – 1) which can be counted, and – 2) which count toward achieving our goals INFO636 Week 7 4 www.ischool.drexel.edu

Reasons for Measurement

• Measurements are required by all major process models (CMM, ISO 9000, etc.) – To

Characterize

or understand the current status of activities or products – To compare that understanding to our objectives, and

Evaluate

status is “good” or “bad” whether the current INFO636 Week 7 5 www.ischool.drexel.edu

Reasons for Measurement

– To

Predict

future performance, based on past trends – To form the basis for measuring

Improvement

• You won’t know if you improved, if you don’t know where you started!

INFO636 Week 7 6 www.ischool.drexel.edu

Where do we get Information?

• Metrics are often used to support decision making (the Evaluate step on a previous slide) • Decisions should be based on quantified information • To get that information, we calculate measures from raw data called data elements • The data elements each come from a data source Data Source Data Elements Measures INFO636 Week 7 Information 7 www.ischool.drexel.edu

How do we Choose what to Measure?

• A commonly used method for selecting measurements is called GQ(I)M for Goal, Question, Indicator, and Measurement – It is based on GQM work by Victor Basili (first reported

circa

1988-89) – GQ(I)M was published by Robert Park

et al

INFO636 Week 7 8 www.ischool.drexel.edu

How do we Choose what to Measure?

• The GQ(I)M method uses ten steps to describe measurements systematically • The steps don’t have to be followed in the order presented; their main purpose is to help ensure that measurements have been fully thought out – You can start at the top, or the bottom, or in the middle INFO636 Week 7 9 www.ischool.drexel.edu

1. Identify Business Goals

•

Business goals

are the big, vague, lofty desired accomplishments for an organization • Think of what you’d find bragged about in a company’s annual report – Reduce cycle time – Improve customer satisfaction INFO636 Week 7 10 www.ischool.drexel.edu

1. Identify Business Goals

– Develop detailed process history – Respond to changing business environment – Reduce overhead – Improve competitive position – Increase market share – Improve product quality INFO636 Week 7 11 www.ischool.drexel.edu

2. Identify Desired Knowledge

• Break down each goal into

products, resources, and activities

needed to meet those goals • Think of questions like: – What activities do I manage or execute?

– What do I want to achieve or improve?

– To do this, I will need to … INFO636 Week 7 12 www.ischool.drexel.edu

3. Identify Subgoals

• Set

subgoals

(objectives) for each of the areas you manage • What do you want to know about the results of step 2? What kind of information is important to you?

– How big/fast/expensive/complex/much time will a process/product/tool/resource take?

INFO636 Week 7 13 www.ischool.drexel.edu

4. Identify Entities and Attributes

• Formulate questions to identify

entities

(documents, work products) created by your process, and the

attributes

of them you are interested in (size, quality, duration, cost) • Don’t get too detailed at this point - just identify the type of information desired (what do you want to understand), and the subject of that information (what about it do you want to know) INFO636 Week 7 14 www.ischool.drexel.edu

4. Identify Entities and Attributes

• Entities that you want to measure can come from any of: – A process’ inputs – The activities within a process – The process’ outputs • Then decide what characteristics of each entity are of interest INFO636 Week 7 15 www.ischool.drexel.edu

5. Define Measurement Goals

• First choose the type of measurement goals – Active goals: reduce or improve something – Passive goals: identify, assess, understand • Lower maturity organizations start with passive goals, then work on active goals after some history is known INFO636 Week 7 16 www.ischool.drexel.edu

5. Define Measurement Goals

• Form structured statements of the measurement goals for each attribute – This step defines a metric in the form of a sentence INFO636 Week 7 17 www.ischool.drexel.edu

5. Define Measurement Goals

• The general format or syntax is: – [objective] • where – is active or passive, as chosen earlier – is the actual measure to be collected – indicate the scope or time frame of the measurements (a.k.a. independent variables), – [objective] is only given for active measurement goals – what value is desired?

INFO636 Week 7 18 www.ischool.drexel.edu

5. Define Measurement Goals

• Passive measurement goal examples: – “Measure the number of requirements which changed each month.” – “Identify the voluntary turnover rate per month for programmers and software engineers.” • Active measurement goal examples: – “Reduce the defect rate of developed code over time to under 20 defects/KLOC” – “Improve the percent of satisfied customers after 30 days of product use to 95% or more” INFO636 Week 7 19 www.ischool.drexel.edu

5. Define Measurement Goals

• DO NOT report traits of individual people (for fear of judgment) – Unless that’s part of their job description (e.g. sales people) • Also need to balance how often measurements are made – More frequent measurement gives finer control, but excessive measurement wastes time and slows the process being measured INFO636 Week 7 20 www.ischool.drexel.edu

6. Quantify Questions and Indicators

• Pose

questions

to address your measurement goals (quantifiable ones, if addressing active goals) – Active: “Can we resolve customer emergencies, on average, in under 24 hours?” – Passive: “How many requirements do we have at the end of the Requirements Definition phase?” INFO636 Week 7 21 www.ischool.drexel.edu

6. Quantify Questions and Indicators

• Sometimes the search for a meaningful metric starts with a question, and that leads to filling out the GQ(I)M from the middle • Identify

indicators

to show the results effectively (see later this lecture), such as: – Pie charts – Bar graphs – Scatter plots, etc.

INFO636 Week 7 22 www.ischool.drexel.edu

7. Identify Data Elements

• Identify the

data elements

needed to prepare (calculate) each indicator – e.g. “defect rate by module each month” needs a list of defects found, with the module each came from, for the last month, and the size of each module in kSLOC INFO636 Week 7 23 www.ischool.drexel.edu

7. Identify Data Elements

• Determine the

source

for each data element – from where do you get it?

– E.g. defect data will come from our change request database – Size data will be generated by the development environment INFO636 Week 7 24 www.ischool.drexel.edu

8. Define Measures

• Define exactly what you mean by each measure – Use a checklist if needed to show what is and isn’t included in its definition (such as we used for definition of LOC) • Cite the source if an unusual measure is used; or if you made it up, explain why – New measures are allowed!

INFO636 Week 7 25 www.ischool.drexel.edu

8. Define Measures

• Include any rules, assumptions, constraints, and environment which are part of the measure’s definition – E.g. Defect rate = (# of defects whose origin was in each module)/(# of kSLOC in that module) INFO636 Week 7 26 www.ischool.drexel.edu

9. Identify Measurement Implementation

• Deciding how to implement a measurement program is generally done in three steps –

Analyze

what measures are currently collected (if anything) and how they’re used –

Diagnose

how well the current measurements meet your goals – find what’s missing?

INFO636 Week 7 27 www.ischool.drexel.edu

9. Identify Measurement Implementation

–

Act

on implementing new measurements, possibly in a phased approach based on project or organizational priorities • Implementing measurements is a cultural change, so like any other change, it’s best done in small steps – Add a few measurements now, wait six months, add a few more, etc.

– Let people see benefits from the new measurements!

INFO636 Week 7 28 www.ischool.drexel.edu

10. Prepare Measurement Plan

• Take all of the aforementioned information and create a complete plan to identify and implement measurement for your organization or project • This is generally called a Metrics Plan or a Measurement Plan INFO636 Week 7 29 www.ischool.drexel.edu

Summary of GQ(I)M Steps

• Goal • Subgoal (objective; why collect this metric) • Question (answered by this metric) • Indicator (how display metric) • Measurement (the actual metric and its definition) • Data Elements (used to calculate metrics) • Source (of each data element) INFO636 Week 7 30 www.ischool.drexel.edu

Indicators

• Indicators are the

means used to present measures

, such as charts, graphs, etc.

• Consider how your data will be presented - in color or B/W, live or printed – Will your audience see pristine originals, or will it be copied a zillion times?

INFO636 Week 7 31 www.ischool.drexel.edu

Indicators

• To choose the right indicator, consider: 1. The Amount of Data to be presented for each interval (e.g. one measure at a time or five different survey responses at once) 2. The Number of Intervals to be shown, such as time units, modules of code, etc.

INFO636 Week 7 32 www.ischool.drexel.edu

Indicators

• Different indicators are better at different Amount or Number characteristics INFO636 Week 7 33 www.ischool.drexel.edu

Indicators

• For most graphs: – The X-axis of a graph (the horizontal line) is the independent variable; often a , such as time, severity, etc. • If the X axis is Time, it should show how often measurements are made (weekly, monthly, etc).

INFO636 Week 7 34 www.ischool.drexel.edu

Indicators

– The Y-axis of a graph (the vertical line) is the dependent variable; the thing you are measuring (the Measure). Y Dependent variable (measure) x x x x x x x x INFO636 Week 7 X axis Independent variable (qualifier) 35 www.ischool.drexel.edu

Pie Chart

• The lowly pie chart is good for presenting a small amount of data – % of customers satisfied and not satisfied – % of defects by severity at this moment • Amount of Data: Shows a few data points (2-10) • Number of Intervals: One - it generally shows only one moment in time INFO636 Week 7 36 www.ischool.drexel.edu

Sample Pie Chart

Sales ($M) 1.2

4.6

2.4

West Northeast South Midwest East 1.1

1.7

INFO636 Week 7 37 www.ischool.drexel.edu

Ishikawa’s Basic Tools

• Developed

circa

1989 for manufacturing production – Used widely in quality control • Focuses on project level concerns; is this batch good enough to accept?

• Not very useful for research; has little theoretical basis INFO636 Week 7 38 www.ischool.drexel.edu

1. Check sheet

• Used to gather data easily, consistently, and in a standard format • A check sheet used to help the quality of a process or product is a “checklist” • Helps to define key parts of a process • Examples include code inspection checklist, detailed test procedures INFO636 Week 7 39 www.ischool.drexel.edu

2. Pareto Diagram

• Used to identify problem areas - where to fix first; what are the biggest fires to put out – Defects tend to cluster in buggy portions of code • Use Pareto to plot a column graph of the defect rate by the module it came from – Could add a line graph for the cumulative % of defects above it (not shown) – Pareto diagram

must

list X axis categories in descending order of value INFO636 Week 7 40 www.ischool.drexel.edu

2. Sample Pareto Diagram

Pareto Diagram of Component Defect Rate 25 20 15 10 5 0 22.3

15.6

Interface Query 13.3

Core 11.1

Import

Component

8.9

Export 5.6

Reports INFO636 Week 7 41 www.ischool.drexel.edu

3. Histogram

• A bar chart is used to break down data by an ordered category (e.g. defect severity, satisfaction rating) – Can choose to put bars next to each other (clustered), or stack them. • Stack when they add to a constant (e.g. 100%), or when the total is also a useful measure (total number of defects) INFO636 Week 7 42 www.ischool.drexel.edu

3. Histogram

• Can show limited time spans, e.g. a few time intervals

No. of Problem Reports by Status

234 Simple bar graph  250 200 150 100 50 0 23 New 50 34 15 8 Closed Pending Withdrawn Analysis

Status

Testing INFO636 Week 7 43 www.ischool.drexel.edu

500 400 300 200 100 0

3. Histogram

Problem Reports in each Status over Time

 Stacked Testing Analysis Withdrawn Pending Closed New

Problem Reports in each Status over Time

Jan-02 Feb-02 Mar-02

Time

Apr-02 May-02 Clustered  300 250 200 150 100 50 0 Jan-02 Feb-02 Mar-02

Time

Apr-02 May-02 New Closed Pending Withdrawn Analysis Testing INFO636 Week 7 44 www.ischool.drexel.edu

3. Histogram

• A true histogram can group ranges of values in the X axis, and show counts or average Y values for each group – E.g. average salary (Y axis) for employees aged 18-29, 30-39, 40-49, etc. (X axis groups) – Good for hiding individual values, and looking at larger trends in the data INFO636 Week 7 45 www.ischool.drexel.edu

4. Run Charts

• Plots some measure versus time using a line graph – Often compare values to a desired or target value, especially at higher process maturity levels (CMM Level 3 and up) – Special case: The “S” curve plots (cumulative % completion of something) versus time INFO636 Week 7 46 www.ischool.drexel.edu

4. Sample Run Chart

Number of Defects Found by Severity

20 15 10 5 0 45 40 35 30 25 Jan-02 Feb-02 Mar-02

Time

Apr-02 May-02 Minor Major Total INFO636 Week 7 47 www.ischool.drexel.edu

5. Scatter Diagram

• Plot two measures against each other to see if there’s a correlation between them – E.g. Defect rate per module versus module complexity, or productivity versus experience • When we say plot ‘Blah versus Ick,’ usually Blah is the Y axis, and Ick is the X axis INFO636 Week 7 48 www.ischool.drexel.edu

5. Scatter Diagram

Current Salary vs. Educational Level 60000 50000 40000 30000 20000 10000 0 6 8 10 12 EDUCATIONAL LEVEL 14 16 18 20 22 INFO636 Week 7 49 www.ischool.drexel.edu

5. Scatter Diagram

• Once there’s enough data, can add curve fitted lines, and +/- variances • The scatter diagram is the basis for regression analysis – Curve fitting to the data, to help define a statistically significant connection between the two measures INFO636 Week 7 50 www.ischool.drexel.edu

6. Control Chart

• Statistical Process Control tool • Rare in software development since – Specifications poorly defined – Each project takes a long time – Too many uncontrolled process variances – Process-quality relationship not well defined INFO636 Week 7 51 www.ischool.drexel.edu

6. Control Chart

– Too many processes used – Rapidly changing technology 93.000

Control Chart: Percent Passed 89.000

85.000

81.000

77.000

1 2 3 4 5 6 7 8 9 10 11 12 Sigma lev el: 3 Percent Passed UCL = 92.3343

Av erage = 85.0833

LCL = 77.8324

INFO636 Week 7 52 www.ischool.drexel.edu

6. Control Chart

• Pseudo-control charts include: – The ‘u’ chart for defect rates by component, BMI, etc. versus time – The ‘p’ chart for percentages, such as inspection effectiveness or customer satisfaction rating • There are many other varieties of control charts INFO636 Week 7 53 www.ischool.drexel.edu

6. Control Chart

• Usual control limits are +/- 3 sigma from the mean, which define the upper and lower control limits (UCL and LCL) – Can add a warning limit at +/- 2 sigma • Control Chart tend to be used in very mature (CMM level 4 or 5), long term projects INFO636 Week 7 54 www.ischool.drexel.edu

Fishbone Diagram

• Technically, the fishbone diagram is Ishikawa’s seventh tool, but I doubt anyone will use it in conjunction with the PSP INFO636 Week 7 55 www.ischool.drexel.edu

Measurement Basics

• Given all that on GQ(I)M and indicators, the most essential types of measurements are still pretty basic – Process – how long does it take?, how much does it cost?, are we starting it on time?, etc.

– Tools – (rare) how well are our development tools being used?

INFO636 Week 7 56 www.ischool.drexel.edu

Measurement Basics

– Resources – how much does it cost to keep our people trained? How much does it cost to find new personnel?

– Product – how big is our product (LOC)? What is its quality? How many defects have been found? How happy is the customer with it?

INFO636 Week 7 57 www.ischool.drexel.edu

Gathering Data

• Critical from the PSP perspective is defining our processes to gather data consistently and efficiently – How do you record time spent?

– How do you collect measurements from various people and merge them? (A critical concern for INFO 637) INFO636 Week 7 58 www.ischool.drexel.edu

Gathering Data

• Our forms have been designed to help collect key aspects of the processes – Product size – Time spent in each part of the process – Defect injection and removal phases – Defect type and fix time – Bad fixes (the infamous Fix Error field) – Productivity (LOC/hour) INFO636 Week 7 59 www.ischool.drexel.edu

Gathering Data

• The result of this is a large body of data that can be collected across many projects – While the data gathering and analysis takes time, it also produces real answers about how quickly and well you perform various tasks – This is your

personal process baseline

INFO636 Week 7 60 www.ischool.drexel.edu