COSYSMO Risk/Confidence Estimation Prototype John Gaffney March 14, 2005

Setting The Stage: The State of the Management World

• Managers and technical personnel need to make decisions under uncertainty. • They should assess the extent of the uncertainty in the data and quantitative information that they rely on so that they can make better decisions. – Program managers often get a lot of (raw) data and frequently very little information .

– Assess uncertainty using a systematic process. Recognize that there is uncertainty in both program input data (e.g.,goals, historical data, estimates) and program outcomes and assessing them is key.

– Associate this process with “early/ leading indicators.” • All too often, “the” value for effort or schedule is presented, unaccompanied by any statement of the degree of uncertainty in that value. Program managers and others involved in developing estimates for proposals should be able to quantify the degree of uncertainty in the estimates that they produce. Estimating cost, schedule, and other product or process variables as single numbers fails to provide decision makers with information sufficient to make good bidding and other decisions. © Lockheed Martin Corporation, 2004/2005 2 .

The Problem of Unrealistic Pricing & Schedule

“Projects often overrun their cost estimates, sometimes by staggering amounts. This occurs even with carefully-constructed bottom-up cost estimates completed to a very detailed level by experienced project teams.” "Initial cost and schedule estimates for major projects have invariably been over-optimistic. The risk that cost and schedule constraints will not be met cannot be determined if cost and schedule estimates are given in terms of single points rather than distributions." "The purpose of a cost uncertainty analysis is to provide the project manager with a cost that has an acceptable probability of being exceeded." The notion that there is a probability of exceeding an EAC is a difficult one for some people. But, the fact is that every project has risk, and ignoring risk does not make it go away.” Dr. David Hulett © Lockheed Martin Corporation, 2004/2005 3

What is “Risk” ?

•Risk is commonly evaluated as the product of the likelihood (taken as the probability) of an occurrence of an event and the impact or consequence of the event with respect to a specific factor (cost, schedule, technical parameter, etc). •A plot of occurrence probabilities and consequences is a “risk profile” or a “Farmer curve.” The probability is often referred to as the “exceedance” probability, because it is the

probability

that the

consequence value

will be exceeded*. *Ayyub, Bilal M., Risk Analysis in Engineering and Economics, Chapman and Hall/CRC, 2003.

© Lockheed Martin Corporation, 2004/2005 4

“Risk” Vs. “Confidence,” Alternative Views or Perceptions

, page 1 of 2

• You can

consider uncertainty

figure or the “confidence” in it.

in terms of the “risk” in a • Example Definitions:

–

Effort Risk

=Probability (complementary cumulative distribution) that the effort

will exceed

the indicated value. Called the “exceedance probability” in some risk literature.

–

Effort Confidence

=Probability (cumulative distribution) that the effort

will not exceed

the indicated value. This is the upper bound of an effort estimate at the stated

confidence

level. Example: 90% confidence=10% risk. © Lockheed Martin Corporation, 2004/2005 5

“Risk” Vs. “Confidence,” Alternative Views or Perceptions

, page 2 of 2 • It depends on with whom you are talking and how you want to communicate the data and the

best way that you can communicate the uncertainty in the data (and information)

.

• It depends on what question you are trying to answer. For example: – If you are a program manager or upper management of the contracting or bidding organization, you might be asking “What is my cost (effort) exposure or risk if I bid this figure?” – If you are the customer or acquirer or you are the bidder trying to communicate to this person, you might be asking “What is the bidder’s confidence in this cost (effort) figure ?” • That is, there are

two alternative points

of view. Both are correct. It depends to whom the message quantifying uncertainty is being conveyed.

© Lockheed Martin Corporation, 2004/2005 6

COSYSMO Risk Tool Prototype Overview

This excel-based tool is a prototype of an add-on to the COSYSMO systems engineering labor estimation model. It enables the user to quantify his belief in the uncertainties in the values of various parameters of the COSYSMO model, and hence in the value of the output of the model, systems engineering person months (PM). Each uncertainty is represented by two equivalent distributions, "

risk

"and the other for "

confidence,

" as was defined before.

The tool uses a three-point estimator to approximate the distribution of the values of each uncertain quantity for which data is applied in the model/tool. The three values required from which the distribution function is approximated are: the 5% fractile, the median, and the 95% fractile. Operationally, these values are obtained from a combination of expert opinion and historical data (as available and relevant). Due to the uncertainties that often prevail, the three points elicited are frequently somewhat loosely interpreted as the smallest, most likely, and largest, allowing for some margin below the smallest value and some beyond the largest.

The tool consists of five sheets:

-TOOL Description -COSYSLABRISK -COSSIZEDR -COSTDRIV -PLOTS

© Lockheed Martin Corporation, 2004/2005 7

COSYSLABRISK

This is the "main" part of the model/tool. It computes both the PM Risk and PM Confidence distributions, based in part on distribution vales for the equivalent requirements size (see

COSSIZEDR

tool sheet description) and cost drivers product value (see

COSTDRIV

tool sheet description). It also computes a PM Overrun Risk, the range of probabilities for a range of possible overruns, relative to some target PM that the user enters. This tool sheet determines the range for PM per the following equation:

Person Months, PM=A*S E *D

; where: D=cost driver product value,S=Equivalent Requirements Size, A=Baseline Unit PM per S value, and E=Exponent. This tool sheet requires the user to enter the three-point distribution approximation values for A and E. The tool sheet obtains the three-point values for D and S from from the

COSTDRIV

and

COSTSIZEDR

tool sheets.

This tool sheet must be executed whenever you change the value of at least one parameter (ones that may be changed are in yellow cells). To execute this sheet press "cntrl+b."

© Lockheed Martin Corporation, 2004/2005 8

22-Feb-05

PROTOTYPE COSYSMO EFFORT RISK ESTIMATOR Place value in each box in dicated in yellow. DO NOT MAKE ENTRIES IN ANY OTHER CELLS.

Then, press cntrl+b to execute the tool and obtain the .

person month risk and cumulative probabilities

Risk Component Number 1 2 Name A S 3 4 E D Range Estimate Values Low Estimate 2.40

3829 Likely Estimate 3.00

High Estimate 3.60

4244 5039 0.95

0.4420

1.00

1.1310

1.10

1.9688

Person Months,PM=A*S E *D; where: D, cost driver product, an S, size driver, are obtained from other tool sheets.

Most Likely PM 14399 PM Target 25000

Summary COSYSMO Person Months Risk/Confidence Statistics Minim um PM= Risk= Confidence= Most Likely PM= Risk= Confidence= Maxim um PM= Risk= Confidence= 20% Risk/ 80% Confidence PM= 2689 99.88% 0.12% 14399 42.52% 57.48% 83762 0.00% 100.00% 23808 30% Risk/ 70% Confidence PM= 17096 13902 50% Risk/ 50% Confidence PM= 4447 95% Risk/5% Confidence PM= 39838 5% Risk/95% Confidence PM=

© Lockheed Martin Corporation, 2004/2005 9

Person Months Risk

100% 95% 90% 85% 80% 75% 70% 65% 60% 55% 50% 45% 40% 35% 30% 25% 20% 15% 10% 5% 0% 0 25000 50000

Person M onths

75000 © Lockheed Martin Corporation, 2004/2005 100000 10

100% 95% 90% 85% 80% 75% 70% 65% 60% 55% 50% 45% 40% 35% 30% 25% 20% 15% 10% 5% 0% 0

Person Months Confidence (Cumulative Probability)

25000 50000

Person Months

75000 © Lockheed Martin Corporation, 2004/2005 100000 11

100% 95% 90% 85% 80% 75% 70% 65% 60% 55% 50% 45% 40% 35% 30% 25% 20% 15% 10% 5% 0% 0

Person Months Overrun Risk For Target= 25000 PM

5000 10000 15000

Person Months Overrun

20000 25000 © Lockheed Martin Corporation, 2004/2005 12

COSSIZEDR

This tool sheet develops the distribution function for the values of the four drivers from which the value for Equivalent Requirements is computed: The drivers are: system requirements, system interfaces, system specific algorithms, and operational scenarios. Four distributions are developed, one for each driver. The three points for each of them is a weighted sum for the counts of the "Easy," "Nominal," and "Difficult" values for each of the four drivers. The distribution approximation obtained has 81 points (81=3*3*3*3). This number of points provides a relatively smooth distribution curve.

This tool sheet must be executed whenever you change the value of at least one parameter (ones that may be changed are in yellow cells). To execute this sheet press "cntrl+a."

© Lockheed Martin Corporation, 2004/2005 13

COSYSMO Equivalent Requirements (1) Size Range/Risk Estimator

cntrl+a macro1

Size Driver Weights

22-Feb-05 Driver Name # of System Requirements # of System Interf aces # of System-Specif ic Algorithms # of Operational Scenarios Easy 0.50

1.50

3.00

10.30

Nominal 1.00

3.90

5.80

24.60

Dif f icult 4.23

8.20

Note: These weights should not be modified without the agreement of the COSYSMO model owner.

16.60

53.80

(1): Number of Equivalent Requirements=Weighted Sum of Each of Four Easy,Nominal, and Dif f icult Size Driver Values.

Size Driver Range Values Data Entry

Range Values (1) Driver Name # of System Requirements # of System Interf aces # of System-Specif ic Algorithms # of Operational Scenarios Low Estimate Likely Estimate High Estimate Low Estimate Likely Estimate High Estimate Low Estimate Likely Estimate High Estimate Low Estimate Likely Estimate High Estimate Number of Driver Items At Dif f iculty Level (2) Easy 47.5

50 65 Nominal 71.25

75 97.5

Dif f icult 261.25

275 357.5

Estimator Range Value (3) Size Driver Value (4) 95.00% 100.00% 130.00% 1200 1263 1642 Data Entry Error Message Board (5) 27 30 33 12.75

15 18.75

54 60 66 17 20 0 72 80 88 25.5

30 0 90.00% 100.00% 110.00% 85.00% 100.00% 125.00% 842 935 1029

ENTRY MUST BE > 100%

560 659 824 7.5

10 15 11.25

15 22.5

12.75

17 25.5

75.00% 100.00% 150.00% 1040 1387 2080

Caution: Enter data in yellow cells only!

Size Data Entry Notes: (1): The probability distributions f or each of the f our size drivers are estimated based on the nominal (assumed to be the mode) and the Low (assumed to be the 5% f ractile), and the High (assumed to be the 95% f ractile). This is done f or the count values that you enter f or easy, nominal, and dif f icult levels of dif f iculty. (2): The size driver values f or the Low Range (5% f ractile) and the High Range (95% f ractile) are entered as percents of the nominal values. The estimator tool proportions the counts f or Easy, Nominal, and Dif f icult.

(3): The percentages f or the Low and the High range values are entered as percents of the Nominal range value.

(4): The size driver value is equal to the w eighted sum of the number of Easy, Nominal, and Dif f icult size driver counts f or each of the f our size drivers. The w eights used are provided above.

(5): Error messages are given if you enter a Low Range Value >100% of the Nominal and/or if you enter a High Range Value <100% of the Nominal.

© Lockheed Martin Corporation, 2004/2005 14

Summary COSYSMO Equivalent Requirements Size Risk/Confidence Statistics M inim um Size = Ris k = 3642 99.88% Confide nce = 0.12% M os t Lik e ly Size = Ris k = Confide nce = M axim um Size = Ris k = Confide nce = 20% Ris k / 80% Confide nce Size = 4244 48.15% 51.85% 5574 0.00% 100.00% 4716 30% Ris k / 70% Confide nce Size = 4441 4239 50% Ris k / 50% Confide nce Size = 95% Ris k /5% Confide nce Size = 5% Ris k /95% Confide nce Size = 3829 5039

© Lockheed Martin Corporation, 2004/2005 15

Equivalent Requirements Risk

100% 95% 90% 85% 80% 75% 70% 65% 60% 55% 50% 45% 40% 35% 30% 25% 20% 15% 10% 5% 0% 3500 4000 4500 5000

Equivalent Requirem ents Value

5500 6000 © Lockheed Martin Corporation, 2004/2005 16

100% 95% 90% 85% 80% 75% 70% 65% 60% 55% 50% 45% 40% 35% 30% 25% 20% 15% 10% 5% 0% 3500

Equivalent Requirements Confidence (Cumulative Probability)

4000 4500 5000

Equivalent Requirements Value

5500 6000 © Lockheed Martin Corporation, 2004/2005 17

COSTDRIV

This tool sheet develops the distribution for the product of four of the COSYSMO cost drivers: Requirements Understanding, Technology Risk, Personnel/Team Capability, and Tool Support. More or even all of the cost drivers could be covered in a future prototype or final version of COSYSMO as may be desired. The four chosen were somewhat arbitrarily selected to illustrate the process of estimating the uncertainty of cost driver values and the application of the uncertainty in determining the uncertainty of the value of Person Months by COSYSMO.

This tool sheet must be executed whenever you change the value of at least one parameter (ones that may be changed are in yellow cells). To execute this sheet press "cntrl+c." © Lockheed Martin Corporation, 2004/2005 18

PROTOTYPE COSYSMO COST DRIVER RISK ESTIMATOR 22-Feb-05 Place one "X" in each row to select the low er, the m ost likely, and Upper values covering the range of your uncertainty in the value of each of the four cost drivers.

Then, press cntrl+c to execute the tool and obtain the risk curve and cum ulative probabilities for the product of the values of these cost drivers.

DO NOT MAKE ENTRIES IN ANY CELLS EXCEPT THOSE INDICATED IN YELLOW.

Note: Values inidcated in

red

from COSYSMO team; others are hypothetical.

Driver Number 1 2 3 4 Name

Reqm'ts. Under.

Low Estimate Likely Estimate High Estimate

Technol. Risk

Low Estimate Likely Estimate High Estimate

Pers/Team Cap.

Low Estimate Likely Estimate High Estimate

Tool Support

Low Estimate Likely Estimate XL 1.9

0.5

1.59

1.43

VL 1.71

0.68

x 1.5

x 1.4

x L 1.30

1.26

1.12

1.1

Values N H 1.00

0.75

x x x 1.00

1.00

1.00

x x 1.3

0.87

0.87

x x x VH 0.65

1.75

0.68

0.75

XH 0.50

2.00

0.62

0.62

High Estimate Most Likely Driver Product Value= 1.131

x © Lockheed Martin Corporation, 2004/2005 19

COSYSMO Cost Driver Product Size Risk

100% 95% 90% 85% 80% 75% 70% 65% 60% 55% 50% 45% 40% 35% 30% 25% 20% 15% 10% 5% 0% 0.00

0.50

1.00

1.50

2.00

2.50

Product of Cost Driver Values

3.00

3.50

4.00

© Lockheed Martin Corporation, 2004/2005 20

COSYSMO Cost Driver Product Size Confidence (Cumulative Probability)

100.00% 90.00% 80.00% 70.00% 60.00% 50.00% 40.00% 30.00% 20.00% 10.00% 0.00% 0.000

0.500

1.000

1.500

2.000

2.500

Product of Cost Drivers

3.000

3.500

4.000

© Lockheed Martin Corporation, 2004/2005 21