Transcript Risk Analysis and Estimating Uncertainty

Approved for Public Release;
Distribution Unlimited. #06-0697
Risk Analysis & Estimating Uncertainty
…and what this has to do with the price of milk in McLean.
The Society of Cost Estimating & Analysis (SCEA)
Greater Washington DC Chapter
Phil Beenhouwer
May 17, 2006
MITRE
The Economic and
Decision Analysis Center
© 2006 The MITRE Corporation. All rights reserved.
For MITRE Internal Use Only.
Approved for Public Release;
Distribution Unlimited. #06-0697
Agenda
 The Problem
 Variability & Standard Deviation
 Briefing Goals
 An Uncertainty Primer
 Definitions
 Distributions and Simulation…
 Excel Prowess
 Benefits, Headlines, Other Disciplines
MITRE
The Economic and
Decision Analysis Center
© 2006 The MITRE Corporation. All rights reserved.
For MITRE Internal Use Only.
Slide 1 of 26
Approved for Public Release;
Distribution Unlimited. #06-0697
My Premise…
There are three kinds of people in the World…….,
…………….those who can count, and those who can’t.
The lack of risk/uncertainty analysis and
poor cost/schedule estimating (across all levels of costing)
are the primary reasons that Programs are over-budget.
MITRE
The Economic and
Decision Analysis Center
© 2006 The MITRE Corporation. All rights reserved.
For MITRE Internal Use Only.
Slide 2 of 26
Approved for Public Release;
Distribution Unlimited. #06-0697
"It's tough to make predictions, especially about the future."
-- Yogi Berra.
The Problem
Combined Cost
Modeling and Cost
Driver Uncertainty
Cost = a + bXc
Cost Modeling
Uncertainty
Cost
Estimate
$
Historical data point
Cost estimating relationship
Cost Driver
Uncertainty
Input variable
MITRE
The Economic and
Decision Analysis Center
© 2006 The MITRE Corporation. All rights reserved.
For MITRE Internal Use Only.
Standard percent error bounds
Cost Driver
Source: Timothy P. Anderson,
The Aerospace Corporation,
2005 DoDCAS Symposium
Slide 3 of 26
Approved for Public Release;
Distribution Unlimited. #06-0697
“Variability” and “Standard Deviation”
 When someone says it’ll cost you $100…
“Figures lie
"There are lies,
and liars
damned lies
figure…”
and statistics."
 The average American carries $9,000 in credit card debt.1
- In reality, most Americans owe nothing to credit card
-
1
companies.
Most households that carry balances owe $2,000 or less.
Only about 1 in 20 American households owes $8,000 or more
on credit cards.
CardWeb.com, a service that tracks credit card trends.
The Economic and
Decision Analysis Center
MITRE
© 2006 The MITRE Corporation. All rights reserved.
For MITRE Internal Use Only.
Slide 4 of 26
Briefing Goals
 The “threshold” is for you to
think in terms of risk,
uncertainty, and variability…
Hey Phil..., there’s a
good chance that
we’ll have to replace
the hardware every
two to three years…
Timmy’s saying that
the soccer trip to
Europe will only
cost three grand,
but…
You know,
Phil….., we may
have to place a
sensor suite
anywhere from
1.2 to 3.1 miles…
Phil…, I think we
should plan for a
worst-case of 2.5
million
transactions per
year…
 The “objective” is for
you to speak in terms
of risk, uncertainty,
and variability…, and
then to apply it…
MITRE
Approved for Public Release;
Hmmm…,
this #06-0697
Distribution Unlimited.
Contractor’s
saying that the
kitchen remodel
will cost twenty
grand, but…
The Economic and
Decision Analysis Center
© 2006 The MITRE Corporation. All rights reserved.
For MITRE Internal Use Only.
Slide 5 of 26
Approved for Public Release;
Distribution Unlimited. #06-0697
An Uncertainty Primer… (Slide 1 of 4)
MITRE
SDLC Phase
Action
 Concept and Business Case
 Resolve the lack of milk at home
 Initiation and Authorization
 Spousal approval/funding
 Project Definition
 Buy a gallon of milk
 System Design
 Find merchant along route home
 Construction
 Drive to store and purchase milk
 Acceptance
 Did you have enough money?
 Operational Readiness
 Get a clean glass!
The Economic and
Decision Analysis Center
© 2006 The MITRE Corporation. All rights reserved.
For MITRE Internal Use Only.
Slide 6 of 26
Approved for Public Release;
Distribution Unlimited. #06-0697
An Uncertainty Primer… (Slide 2 of 4)
 Gallon of Milk Data Collection
- Convenience Store: $3.49
- Warehouse Club #1: $2.59
- Warehouse Club #2: $2.69
- Grocery Store #1: $2.69
- Grocery Store #2: $3.19
- Grocery Store #3: $2.89
- Grocery Store #4: $2.79
- Specialty Store #1: $3.09
- Specialty Store #2: $2.99
- Specialty Store #3: $3.29
MITRE
The Economic and
Decision Analysis Center
© 2006 The MITRE Corporation. All rights reserved.
For MITRE Internal Use Only.
Slide 7 of 26
Approved for Public Release;
Distribution Unlimited. #06-0697
An Uncertainty Primer… (Slide 3 of 4)
 So, if you budgeted $3.00 for
milk (the mean), you can go to…
- Warehouse Club #1: $2.59
- Warehouse Club #2: $2.69
- Grocery Store #3: $2.89
- Grocery Store #1: $2.69
- Specialty Store #2: $2.99
I could go
- Grocery Store #4: $2.79 here with
 But you can’t go to…
- Convenience Store: $3.49
- Specialty Store #3: $3.29
- Grocery Store #2: $3.19
- Specialty Store #1: $3.09
21 cents
more…
80% of the stores sell milk for less than $3.21 per gallon.
MITRE
The Economic and
Decision Analysis Center
© 2006 The MITRE Corporation. All rights reserved.
For MITRE Internal Use Only.
Slide 8 of 26
Approved for Public Release;
Distribution Unlimited. #06-0697
An Uncertainty Primer… (Slide 4 of 4)
 What can you afford to buy at these stores?
But you can’t go to…
- Conv. Store: $3.49
- Is a half-gallon acceptable?
- Spec. Store #3: $3.29
- Grocery Store #2: $3.19
- Will you even leave with milk?
- Spec. Store #1: $3.09
- How will ‘Operational Readiness’ go when you get home?
- Can you use ‘legacy’ orange juice in tomorrow’s cereal instead?
- Will you need to reduce the number of cereal users?
- Will you need to cut all cereal training from the budget?
- Will there be a GAO report on your pillow in the morning?
 “Sure”, you say, “if I end-up at the Convenience Store with only
$3.21, I can find the 28 cents I need from under the driver’s seat…”.
- But what if these weren’t dollars, but billions of dollars…?
- Could you find $28M under the Program Manager’s seat?
MITRE
The Economic and
Decision Analysis Center
© 2006 The MITRE Corporation. All rights reserved.
For MITRE Internal Use Only.
Slide 9 of 26
Approved for Public Release;
Distribution Unlimited. #06-0697
“Program Risk” versus
“Estimating Uncertainty”
 Total Risk = RiskProgram + (+/- UncertaintyEstimation)1
 For the purposes of quantitative risk analysis, I have defined:
- “Program Risk” – the probability and severity of loss linked
to hazards2 (e.g., software development cannot begin if the
environment is not ready, system testing might fail, etc.)
- “Estimating Uncertainty” -- the estimated amount or
percentage by which an observed or calculated value may
differ from the true value3 (e.g., the number of users could
be 25% less, the COTS license cost could be $1,000 more,
training could take one week longer, etc.)
 Should also consider benefits and schedule, in addition to cost
1 Source: Keith Horenstein,
MITRE
The Economic and
Decision Analysis Center The MITRE Corporation
© 2006 The MITRE Corporation. All rights reserved.
For MITRE Internal Use Only.
2 Source: DoD Dictionary,
www.dtic.mil/doctrine/jel/doddict/
3 Source: www.answers.com
Slide 10 of 26
Approved for Public Release;
Distribution Unlimited. #06-0697
Definitions
 Simulation: any analytical method meant to imitate a real-life
system.1
 Monte Carlo simulation: a simulation that randomly generates
values for uncertain variables over and over to simulate a model.1
 xth percentile: the percentage at which x% of all outputs are at, or
below, the associated cost value
i.e., the 80th percentile in the gallon of milk example means that 80%
of the values are at, or below, $3.21.
- Conversely, 20% of the values exceed $3.21.
-
MITRE
The Economic and
Decision Analysis Center
© 2006 The MITRE Corporation. All rights reserved.
For MITRE Internal Use Only.
1 Source: Decisioneering’s website;
www.decisioneering.com
Slide 11 of 26
Approved for Public Release;
Distribution Unlimited. #06-0697
Description of Monte Carlo Tools
 “Crystal Ball applications transform your spreadsheets into
dynamic models that solve almost any problem involving
uncertainty, variability and risk.”1
 “Simply by running a
simulation, @RISK takes your
spreadsheet model from
representing just one possible
outcome to representing
thousands of possible
outcomes.”2
2
MITRE
Source: Palisade website
The Economic and
Decision Analysis Center www.palisade.com/
© 2006 The MITRE Corporation. All rights reserved.
For MITRE Internal Use Only.
1 Source: Decisioneering’s website;
www.decisioneering.com
Slide 12 of 26
Approved for Public Release;
Distribution Unlimited. #06-0697
Triangular Distributions…
Perfectly
symmetrical;
used far too
often
Probably the most
realistic….,
‘low’ and ‘most likely’
are the same!!
Getting better….,
skewed to the right
MITRE
The Economic and
Decision Analysis Center
© 2006 The MITRE Corporation. All rights reserved.
For MITRE Internal Use Only.
Slide 13 of 26
Approved for Public Release;
Distribution Unlimited. #06-0697
…Triangular Distributions…
80th
Percentil
e
80th
Percentil
e
80th
Percentil
e
MITRE
The Economic and
Decision Analysis Center
© 2006 The MITRE Corporation. All rights reserved.
For MITRE Internal Use Only.
Slide 14 of 26
Approved for Public Release;
Distribution Unlimited. #06-0697
Monte Carlo Simulation
Sin k
L ig h tin g
L ab o r rate
Co u n terto p s
1 4 0 .0 0
1 5 7 .5 0
1 7 5 .0 0 1 2 0 .0 0
1 9 2 .5 0 1 3 5 .0 0
7 6 5 .0 0
6 4 0 .0 0
7 2 0 .0 0
6 0 0 .0 0
6 7 5 .0 0
8 0 .0 0
8 5 0 .0 0
9 3 5 .0 0
9 0 .0 0
1 0 0 .0 0
1 ,0 2 0 .0 0 6 8 0 .0 0
4 0 .0 0
8 8 0 .0 0
4 5 .0 0
9 6 0 .0 0
5 0 .0 0
6 4 0 .0 0
7 5 0 .0 0 5 0 0 .0 0
8 2 5 .0 0 5 6 2 .5 0
9 0 0 .0 0 662050.0
.000
118705.0
.000 1 2 0 .0 0
L ab o r rate
1 9 2 .5 0 1 3 5 .0 0
1 2 0 .0 0
8 5 0 .0 0
8 0 .0 0
9 3 5 .0 0
9 0 .0 0
6 0 .0 0
8 0 0 .0 0
4 0 .0 0
8 8 0 .0 0
4 5 .0 0
775500.0
.000 5 0 0 .0 0
1 8 0 .0 0
1 ,0 2 0 .0 0
1 0 0 .0 0
1 1 0 .0 0
1 2 0 .0 0
9 6 0 .0 0
5 0 .0 0
5 5 .0 0
6 0 .0 0
6 8 7 .5 0
7 5 0 .0 0
F u rn itu re
F lo o rin g
668775.5
.000
1 6 5 .0 0
Cab in ets
Ap p lian ces
5 5 .0 0
7 2 0 .0 0
2 1 0 .0 0 1 5 0 .0 0
L ab o r h o u rs
W in d o w s
1 1 0 .0 0
7 6 5 .0 0
F u rn itu re
F lo o rin g
L ig h tin g
Co u n terto p s
116557.0
.500
Cab in ets
Ap p lian ces
8 0 0 .0 0
2 1 0 .0 0 115400.0
.000
L ab o r h o u rs
W in d o w s
6 8 0 .0 0
Sin k
8 2 5 .0 0 5 6 2 .5 0
9 0 0 .0 0 6 2 5 .0 0
Forecast: Total Cost (20)
5,000 Trials
7 6 0 .0 0
8 5 5 .0 0
9 5 0 .0 0
5 6 0 .0 0
1 ,0 4 5 .0 0
6 3 0 .0 0
7 0 0 .0 0
1 ,1 4 0 .0 0 7 6 0 .0 0
7 7 0 .0 0
8 5 5 .0 0
8 4 0 .0 0 5 6 0 .0 0
6 3 0 .0 0
7 0 0 .0 0
9 5 0 .0 0
1 ,0 4 5 .0 0
1 ,1 4 0 .0 0
7 7 0 .0 0
8 4 0 .0 0
Frequency Chart
49 Outliers
.025
124
.019
93
.012
62
.006
31
.000
0
9,689.82
MITRE
The Economic and
Decision Analysis Center
© 2006 The MITRE Corporation. All rights reserved.
For MITRE Internal Use Only.
11,452.04
13,214.25
14,976.46
16,738.68
Slide 15 of 26
Approved for Public Release;
Distribution Unlimited. #06-0697
Monte Carlo Output
Input
Gallon of Milk
Inputs
Gallon of Milk
Simulation
1
Simulation
1
Statistics / Cell
$B$16
Iteration / Cell$B$16
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
…
9997
9998
9999
10000
MITRE
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
…
$
$
$
$
2.45
2.77
3.16
3.13
3.21
3.22
2.88
3.18
2.38
3.16
2.72
3.22
3.29
2.71
2.71
2.96
2.54
3.06
3.31
$
1.57
Maximum
$
4.07
Mean
$
2.97
Standard Deviation $
0.29
Variance
$
0.09
Skewness
-0.004912689
Kurtosis
3.020921751
Number of Errors
0
Mode
$
2.97
10.0%
$
2.59
20.0%
$
2.72
30.0%
$
2.82
40.0%
$
2.90
50.0%
$
2.97
60.0%
$
3.04
70.0%
$
3.12
80.0%
$
3.22
90.0%
$
3.35
Minimum
The Economic and
Decision Analysis Center
© 2006 The MITRE Corporation. All rights reserved.
For MITRE Internal Use Only.
Slide 16 of 26
Approved for Public Release;
Distribution Unlimited. #06-0697
…Other Common Distributions…
Uniform distribution:
Normal distribution: bellrepresents a range with
curved shape represents
no ‘most likely’ value
exam. scores, height of the
people in this room, etc.
Discrete distribution:
distinct points represents
the roll of a die
Another discrete
distribution: represents
the unique costs for four
pieces of hardware
MITRE
The Economic and
Decision Analysis Center
© 2006 The MITRE Corporation. All rights reserved.
For MITRE Internal Use Only.
Slide 17 of 26
Approved for Public Release;
Distribution Unlimited. #06-0697
With a little Excel prowess…
Years
1
2
3
4
5
Low
28%
33%
38%
43%
52%
ML
35%
42%
48%
54%
64%
3
High
42%
50%
58%
65%
77%
35%
42%
48%
54%
64%
16%
With every iteration, the
‘number of years’ changes,
and this formula computes
the annual cost based on
the ‘number of years’.
MITRE
The Economic and
Decision Analysis Center
© 2006 The MITRE Corporation. All rights reserved.
For MITRE Internal Use Only.
Slide 18 of 26
Approved for Public Release;
Distribution Unlimited. #06-0697
With a little Excel prowess…
Number of Weeks
Labor Category 1
# of
Staff
Program Manager
Senior Level App. Developer
Mid-Level App. Developer
Senior Level Engineer
Mid-Level Engineer
Senior Software Engineer
Mid-Level Software Engineer
TOTAL
1
1
1
1
2
1
2
9
Development, Integration,
& Testing Duration
21
Hours
Per
Week
Labor
Rate per
Hr
44
44
44
44
44
44
44
$168
$93
$77
$177
$155
$168
$111
FY07 FY08 FY09
52.0
52.0
52.0
52.0
52.0
52.0
52.0
39.0
39.0
39.0
39.0
39.0
39.0
39.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Total
Labor Cost
$383,887
$212,567
$175,402
$404,330
$708,714
$383,887
$508,820
$2,777,606
DIT Distribution [10,16,37]
X <= 10.00
0.0%
8
X <= 26.35
80.0%
7
Probability
Values x 10^-2
6
5
4
3
2
1
0
5
10
15
20
25
30
35
40
Number of Months
MITRE
The Economic and
Decision Analysis Center
© 2006 The MITRE Corporation. All rights reserved.
For MITRE Internal Use Only.
Slide 19 of 26
Benefits to Quantifying Risk/Uncertainty
& Using Monte Carlo
Approved for Public Release;
Distribution Unlimited. #06-0697
 Identify and apply risk/uncertainty within a model where it really
exists (I.e., risk/uncertainty does not really exist “+/- 10%”
around software integration!)
 Sensitivity analysis
 Risk-adjusted estimates are included in the individual items of
the model instead of
aspresent
a bottom-line
“tax”
Just
all your numbers
the 80th percentile!
Makes it harder foratdecision-makers
to remove the “risk”
line-item
-
MITRE
The Economic and
Decision Analysis Center
© 2006 The MITRE Corporation. All rights reserved.
For MITRE Internal Use Only.
Slide 20 of 26
Approved for Public Release;
Distribution Unlimited. #06-0697
Headlines
 GAO Testimony: “CAPITOL VISITOR CENTER -- Results of
Risk-based Analysis of Schedule and Cost”, GAO-06-440T,
February 15, 2006
 GAO Report: “INFORMATION TECHNOLOGY -- Agencies Need
to Improve the Accuracy and Reliability of Investment
Information”, GAO-06-250, January 2006
 GAO Report: “DEFENSE MANAGEMENT -- Additional Actions
Needed to Enhance DOD’s Risk-Based Approach for Making
Resource Decisions”, GAO-06-13, November 2005
MITRE
The Economic and
Decision Analysis Center
© 2006 The MITRE Corporation. All rights reserved.
For MITRE Internal Use Only.
Slide 21 of 26
Approved for Public Release;
Distribution Unlimited. #06-0697
Relationship to Other Disciplines
 Budgeting / Investment Planning
-
Provides risk-adjusted requirements of the E300 and other planning activities
 Acquisition
-
Cost, Schedule, and Performance are part of defining an acquisition strategy
 Contracting
-
Provides a cost basis for negotiation
 Program Management
-
Provides insight into Program risks; helps prioritize mitigations
It’s just good Program Management !!
 Earned Value/Baseline Management
-
Provides an input to the management reserve level
More objective inputs to the EVMS than the typical Integrator provides
Quantification of risks and uncertainties will result in less re-baselines
 Engineering
MITRE
Offers an approach to incorporate uncertain aspects of the engineering design
The Economic and
Decision Analysis Center
© 2006 The MITRE Corporation. All rights reserved.
For MITRE Internal Use Only.
Slide 22 of 26
Approved for Public Release;
Distribution Unlimited. #06-0697
MS Project Monte Carlo Analysis
MITRE
The Economic and
Decision Analysis Center
© 2006 The MITRE Corporation. All rights reserved.
For MITRE Internal Use Only.
Slide 23 of 26
Approved for Public Release;
Distribution Unlimited. #06-0697
Conclusion
 “Think” and “speak” in terms of risk and uncertainty (and then
apply it…)
Collect uncertainty inputs when you gather data
- Use Monte Carlo applications (e.g., Crystal Ball, @Risk, etc.)
$ These are relatively inexpensive compared to other
applications in a coster’s toolkit
$ There is even a $15 application that we are currently
investigating (Excel Business Solutions’ XL Modeling:
www.excel-modeling.com/index_007.htm)
 Include “Program Risk” and “Estimating Uncertainty” in cost,
benefit, and schedule analyses
-
MITRE
The Economic and
Decision Analysis Center
© 2006 The MITRE Corporation. All rights reserved.
For MITRE Internal Use Only.
Slide 24 of 26
And if all else fails…..,
re-define the word “outlier”…
MITRE
The Economic and
Decision Analysis Center
© 2006 The MITRE Corporation. All rights reserved.
For MITRE Internal Use Only.
Approved for Public Release;
Distribution Unlimited. #06-0697
Slide 25 of 26
Approved for Public Release;
Distribution Unlimited. #06-0697
Questions / Comments
MITRE
The Economic and
Decision Analysis Center
© 2006 The MITRE Corporation. All rights reserved.
For MITRE Internal Use Only.
Slide 26 of 26