Transcript Discrete Statistical Scenario-Based (DSSB) Cost

Two Complementary EVM Cost-Risk
Models
1. Use of EVM Trends to Forecast Cost Risks
2. Integrated Cost-Risk Model (ICRM)
Utilizing ACEIT
For
18 MAR SoCal ICEAA Workshop
David R. Graham
Consultant, Salient Federal Solutions
Carlsbad, CA
[email protected]
703-489-6048
Use of EVM Trends to Forecast Cost
Risks
Some Charts from Original Presentation at
2011 ISPA/SCEA Conference, Albuquerque, NM
Dr. Roy Smoker*
MCR LLC
[email protected]
(Other Charts Added by David R. Graham)
*Roy E. Smoker (2011): Use of Earned Value Management Trends to Forecast Cost
Risks, Journal of Cost Analysis and Parametrics, 4:1, 31-51
(C)2011 MCR, LLC
Main Points of Paper
1.
EVM data is taken from the PMB’s S-curve at it’s most linear section
– The early part of the PMB’s S-curve represents start up so is not linear
– The ending part of the PMB’s S-curve represents contract ending so is also not
linear
– Linear regression equations work best when data is linear
2.
Regression equation developed to forecast BAC
– Unique from most regression equations used in EVM performance projections
– Basis: BAC grows due to learning more about the nature of the work as the effort
proceeds and additional work is put on contract through ECOs
3.
Ending month of contract can also be forecasted using regression equations
– Basis: At end of contract BCWP must equal BAC
– Regression equations for BCWP and BAC are set equal to solve for months
• BCWP = BAC  $86.35M*Months = $4,970.56 + $31.76M * Months
• Then just solve for months = 91.06
– NOTE: Calculation made at month 42 - Coefficients change depending on month selected due
to amount of available EVM data increasing over time
– Preferred over usual BAC/Avg BCWP or Lipke’s Earned Schedule
Main Points of Paper (cont)
4.
Regression equation developed to forecast percent complete (PC)
–
–
PC = 0 + 0.013772546*Month (“0” is the intercept, that is, at 0 time there is 0
percent complete)
At month 42, “0.013772546” is the rate of PC/month so, at that rate the
completion month is: Completion Month = 100% Complete/1.3772546 %= 72.6
months
• The PC was measured as the raw value of each monthly BCWP divided by the value of
the BAC for month 42
–
5.
NOTE: Percent complete (PC) using this method understates completion
months due to not taking into account rate at which BAC is growing
Variance at Completion (VAC) is a quantification of value of risk that must
be burned down
– Regression equation developed to forecast EAC
– Subtract forecasted EAC from forecasted BAC = forecasted VAC
– True significance of forecasted VAC:
• Some risk issues that are part of VAC are known and should be described on Format 5
• Unexpected risks have not yet been discovered but are part of this forecasted VAC
18 Months of Data
Equations:
(1) BCWS18 = $89.12M * Months
(0.7525) T-stat = 118.44 R2 = 0.9988
(2) BCWP18 = $86.35M * Months
(0.6925) T-stat = 124.68 R2 = 0.9989
(3) BAC18
= $4,970.56M + $31.76M * Months
( 86.92)
( 2.56)
R2 = 0.9056
T-stats
57.19
12.39
(4) EAC18
= $4,393.13M + $52.62M * Months
( 106.65) ( 3.15)
R2 = 0.9459
T-stats
41.19
16.73
Regressions run in Excel.
EOM
# Months
Cum
Cum
Cum
BCWS
BCWP
ACWP
LRE
BAC
EAC
Sep-95
Oct-95
Nov-95
Dec-95
Jan-96
Feb-96
Mar-96
Apr-96
May-96
Jun-96
Jul-96
Aug-96
Sep-96
Oct-96
25
26
27
28
29
30
31
32
33
34
35
36
37
38
$
$
$
$
$
$
$
$
$
$
$
$
$
$
2,034
2,150
2,247
2,358
2,477
2,586
2,705
2,817
2,921
3,038
3,152
3,245
3,370
3,479
Nov-96
39
$
3,579 $
3,470 $ 3,656 $ 6,118.34 $ 6,396.00
Dec-96
40
$
3,667 $
3,554 $ 3,571 $ 6,145.90 $ 6,424.00
Jan-97
41
$
3,765 $
3,647 $ 3,869 $ 6,292.16 $ 6,570.00
Feb-97
42
$
3,866 $
3,738 $ 3,978 $ 6,269.68 $ 6,548.00
$
$
$
$
$
$
$
$
$
$
$
$
$
$
1,988
2,095
2,191
2,294
2,400
2,501
2,617
2,729
2,828
2,939
3,047
3,138
3,258
3,373
$
$
$
$
$
$
$
$
$
$
$
$
$
$
2,017
2,142
2,243
2,353
2,462
2,565
2,694
2,817
2,932
3,051
3,169
3,274
3,407
3,535
$
$
$
$
$
$
$
$
$
$
$
$
$
$
5,750.00
5,776.00
5,785.00
5,822.94
5,870.95
5,906.54
5,961.16
6,000.22
6,088.19
6,160.98
6,156.54
6,132.15
6,211.78
6,173.79
All Cost are $M
Note: Even with the significance in the parameters in these equations there is a
good degree of variability as indicated by their standard errors in parenthesis.
(C)2011 MCR, LLC
5
$
$
$
$
$
$
$
$
$
$
$
$
$
$
5,750.00
5,776.00
5,785.00
5,823.00
5,871.00
5,917.00
5,972.00
6,017.00
6,146.00
6,350.00
6,344.00
6,352.00
6,401.00
6,363.00
Full 43 Months of EVM Data (months 25 through 67)
(from the Excel based EVM Trend Tool)
Essentially Linear Data
Research: Can we predict the
performance of a long term
contract from only 18 months of
data covering months 25 thru 42
using linear assumptions?
Answer: Perhaps apply the
approach to other, completed
programs to validate the results
presented in Dr. Smoker’s paper.
EVM Trend Data
(Months 25 thru 67)
$9,000
$8,000
$7,000
$6,000
TY $M
Question: What do you get when
you remove the initial start up
months and the contract closeout
months from the usual S-curve?
Answer: A data set that exhibits
the graphic forms shown here.
Note, there is a hint of an S-curve
without the usual tails.
BCWS
$5,000
$4,000
BCWP
$3,000
ACWP
$2,000
BAC
$1,000
EAC
$0
20
40
60
80
Months
We know:
Monthly BAC but not the final BAC
Monthly EAC but not the final EAC
Monthly VAC but not the final VAC
We don’t know the month of the final VAC
(C)2011 MCR, LLC
7
BAC & EAC as a Function of Time
• However, both equations
based on 18 monthly
observations are significant and
can be used to predict the
growth of both BAC and EAC
thru time
EAC vs BAC by Month
TY $M
• The uncertainty in the BAC,
based on program office
decisions to remove work and
add work to the contract, also
creates uncertainty in the EAC as
shown here
$6,700
$6,600
$6,500
$6,400
$6,300
$6,200
$6,100
$6,000
$5,900
$5,800
$5,700
$5,600
EAC = 52.618x + 4393.1
R² = 0.9459
BAC
BAC= 31.762x + 4970.6
R² = 0.9056
0
10
20
30
40
EAC
50
Months
With EAC growing faster than BAC, the
question is when will this contract reach
completion so that VAC stops growing?
(C)2011 MCR, LLC
8
VAC as a Measure of Risk
• Risk is measured in EVM terms as any deviation from
the original baseline.
– That is, risk is anything that results in a variance
• Therefore, VAC is the basic measure of risk
encountered by the end of the contract effort
– Whether the risk is rooted in opportunity with a positive
variance
– Or, is rooted in issues related to planning of scope,
estimating, scheduling, or technical criteria that are
identified during testing and generally associated with a
negative variance
(C)2011 MCR, LLC
9
Risk Burn down (1 of 2)
• The final VAC may be estimated as the difference
between the linear forecast of BAC and EAC
• Risk burn down may be measured as the amount of
VAC that has been worked off
• Therefore, it is possible to show the %risk burn down
as a function of the amount of cumulative VAC that has
been incurred relative to the final VAC
(C)2011 MCR, LLC
10
Risk Burn down (2 of 2)
•
Here the green line represents the
%Risk that has been burned down
and measured as:
1 - Cum VAC/Final VAC
•
It is interesting to note that early in
this program, risk is being burned
down faster than the remaining
work is being accomplished.
Finally, Risk is burned down to zero
as remaining work is reduced to
zero and percent complete
approaches 100%
•
(C)2011 MCR, LLC
11
Summary – Contract Scope
• We have learned
– An S-curve with its tails removed exhibits significant
linearity with variability
– The scope of a contract grows across time
– New work pushes out the expected completion date
– There is a future date where
• BCWP will equal BAC
• From this equivalency the expected completion date can be
calculated
– Each monthly %complete drops as BAC grows
(C)2011 MCR, LLC
12
Summary – Trend Analysis
• We have learned
– Normal monthly EACs fall short of final EAC
• Due to same contract scope growth that affects BAC
– Trend analysis
• Helps identify the completion date
– Can then estimate the final EAC
– Can then estimate the final BAC
– Final VAC
• Can be estimated as: (Final BAC – Final EAC)
• VAC appears useful in measuring the value of a program’s risks
(planning, estimating, scheduling, technical)
• May be used to measure how risks get burned down across the
period of performance from ATP to Estimate Completion Date
(C)2011 MCR, LLC
13
Integrated Cost-Risk Model (ICRM)
Utilizing ACEIT
David R. Graham
Consultant, Salient Federal Solutions
Carlsbad, CA
[email protected]
703-489-6048
NOTE: Special thanks to Darren Elliott of Tecolote, Inc., for actual ICRM programming
Outline
•
•
•
•
•
•
•
•
What ICRM Brings
ICRM Model Overview Narrative
ICRM Model Structure Illustration
Note on Simulation Assumptions Used in the ICRM Model
5X5 Risk Matrix Rating Scales
Dominant Likelihood Algorithm
DAU EVM 201 LAR Risk Register
ICRM Mechanics General Overview
•
•
•
•
ICRM WBS Element and Risk Prioritization Tornado Charts & PDF/CDF Graphs
ACEIT Workscreen Examples
ICRM Customization
Summary
What ICRM Brings
1. EVM data and risk register results into a probabilistic context
using the DAU Light Assault Reconnaissance (LAR) Vehicle case
study as the database for illustrating the ICRM
2. True confidence levels of contractor WBS-level EACs
– Through applying a range of WBS-level EVM PFs to create a distribution of possible
‘adjusted’ BCWR values at the WBS element level (cum ACWP + adjusted BCWR =
EAC)
– Allowing identification of where contractor WBS-level EACs fall in the distributions
3. WBS-level Risk Register-driven cost-risk distributions
– Identifies risk register risks to affected WBS elements
– Applies risk likelihoods and cost consequence ranges to WBS element BCWR values
4. Integration of both WBS PF-based and WBS Risk Register-based
distributions by statistically summing them through monte
carlo simulations in ACEIT producing an overall EAC cost-risk
distribution
What ICRM Brings (cont)
5. Enables prioritization
– By WBS elements most cost-impacted by risks, and
– By risks causing the most significant cost impacts
6. These results provide the basis for an ongoing
meaningful dialogue that is not happening today
between the EVM analysts, technical risk management
teams, cost estimators, schedule analysts, project officers
and, ultimately, the program managers based on cost
impacts caused by risks
ICRM Model Overview Narrative
•
•
Enter in EVM data (e.g., BCWS, BCWP, ACWP cum-to-date)
Non-Probabilistic EAC calculation
–
–
•
Derive BCWR (BAC-BCWP); adjust by performance factor; develop EAC (i.e., ACWP cum-to-date +
adjusted BCWR)
NOTE: Can use BAC assuming growth derived from EVM Trends Approach
Performance Factors range data
– Use CPI; SPI; CPI*SPI; (can use other PFs) as separate cases or two at a time in a Min & Max
uniform distribution
– Utilize ACEIT’s capability to identify minimum/maximum results to construct a PF-based
range distribution relaxing the analyst’s workload
•
Risk Register data
– Identify risks and their impacts to specific WBS elements
• One risk to one WBS; one risk to many WBSs; many risks to one WBS
– Use midpoint of risk likelihoods (e.g., if range=5%-20%, use 12.5%)
– Identify risk cost consequence ranges (i.e., low, most likely & high values) and apply
resulting percent impacts on adjusted BCWRs
•
Incorporate all risks in Latin Hypercube ACEIT simulations and calculate EAC as a total
value that includes all risk effects
ICRM Model Structure Illustration
Statistical PF Range & Risk Register-Impacted EAC
Risk Register Risk Inputs (ID WBSs impacted; Cost Consequences; Likelihoods)
Performance Factor Range Impact Calculation (Range * affected WBS items)
EAC Calculations Based on Single Performance Factor (PF)
Note on ICRM Simulation Assumptions
• Decision Rule on Whether the Risk Actually
Happens
– Random number generator produces a number
between “0” & “1” that compares against the risk
register’s likelihood
• The “Dominant Likelihood” algorithm determines the
‘winner’ of the comparison and either includes the full
risk’s cost consequence or none of the consequence
5x5 Risk Matrix Rating Scales
•
Level Likelihood of Occurrence
–
–
–
–
–
1 Not Likely (5% - 20%)
2 Low Likelihood (21% - 40%)
3 Likely (41% - 60%)
4 Highly likely (61% - 80%)
5 Near certainty (81% - 99%)
(Note: Percentages from DoD range guidance)
• Cost Consequence Rating (see notes,
Alternatives 1,2 &3)
• 5 Critical (23% - 28%)
• 4 Serious (15%- 20%)
• 3 Moderate (10% - 15%)
• 2 Minor (5% - 10%)
• 1 Negligible (1% - 5%)
• OPP (opportunities) Potential cost
savings (added to matrix)
Consequence
Definitions From Program’s Risk Management IPT
5
4
3
2
1
OPP
0
0
0
2
0
1
2
1
0
1
1
2
0
0
0
1
4
1
0
2
0
1
0
0
0
0
0
0
0
1 2 3 4 5
Likelihood
Total Risks =
19
High =
5
Medium =
9
Low =
5
Opportunities =
0
NOTES ON COST
CONSEQUENCE
APPROACHES
Alternative 1: Percent
of last approved cost
estimate
Alternative 2: Percent
of affected WBS
element’s BCWR
(i.e., S/C, P/L, etc.)
Alternative 3: Percent
additional resources
taken as a function of
burn rate per
schedule slip on WBS
element(s) affected
(Note: Percentages from DoD range guidance)
NOTE: Number of risks in above example are from DAU EVM 201 LAR Risk Assessment
Dominant Likelihood Algorithm
General Process Overview
5
2). Estimate Consequence
WBS, Months delay, Phase
of Delay
WBS % Increase in NRE, REC
or both
Convert both to $
NOTE: Random # Draw = > 0 and < 1
M
H
H
H
H
Consequence
1
2
3 4
1). Identify Discrete Risks
Risk #1
Risk #2
Risk #3
…
L
M
H
H
H
L
M
M
H
H
L
L
M
M
H
L
L
L
L
M
1
2
3 4
Likelihood
5

Max. # Risks
Likelihood
> Random
# Draw
Yes
No
N=1
Add full
consequence
Don’t add
consequence
}
3). Estimate Likelihood
Remote, Unlikely, Likely,
Very Likely, Near
Certainty
Run appropriate
# of iterations
Develop Cumulative Distribution
100%
90%
Percent Likelihood of Coming in on Budget (%)
80%
5x5 Risk Distribution
Best Case
Most-Likely Scenario
Worst Case Scenario
70%
60%
50%
40%
30%
20%
10%
0%
0%
10%
20%
30%
40%
50%
Percent Cost Reserve Over Baseline Contract Costs (Without Launch Vehicles)
60%
DAU EVM 201 LAR Risk Assessment
High:
Medium
6. Fuel System
7. Controls/Instrument
8. Subsystem Test
9. Armament
10. Integration/Assembly
LIKELIHOOD
1. Engine
2. Cooling
3. Exhaust
4. Suspension/Steering
5. Power Pac/Sub Sys Design
5
.
.
.
.
1, 3
4
.
9
4,
5
4,5
2
.
3
15
8
6, 7
.
.
10
.
.
.
3
4
CONSEQUENCE
5
2
.
14
11, 12
13, 16
1
17, 18
.
19
1
2
Low
11. Frame
12. Aux Automotive
13. Body Cab
14. Communications
15. Sys Engineering & PM
16. Sys Test & Evaluation
17. Training
18. Data
25
19. Peculiar
Supt Equipment
As of Dec 2003
Mechanics
General Overview
• A basic assumption in using this ACEIT-based ICRM model is that
program EVM analysts are not expected to be proficient ACEIT users but
can work with their cost estimator counterparts who are proficient
• The ACEIT v7.3a ICRM capability is enabled with the new “Probability of
Occurrence” column
– This enables the ‘likelihood’ part of the risk register to be a direct
input into ACEIT and function in its Latin Hypercube monte carlo
simulations IAW the Dominant Likelihood algorithm
• This particular ICRM model incorporates this new ACEIT capability plus
custom ACEIT Dynamic Equation Columns (DECs)
– These DECs enable EVM metrics and index information to be used in
calculating an EAC range that is probabilistically derived
– These DECs also provide new variables and functions used in ICRM
calculations
Mechanics
General Overview (cont)
• The ICRM ACEIT model’s Workscreens enable analysis
functions
– These analysis functions exist in the interplay between
the Equation/Throughput columns, DECs and within
the INPUT VARIABLES sections of each of the
Workscreens
– These analysis functions enable the essential
calculations for the probabilistic EAC to be produced
• Calculations incorporate both EVM best and worst case EVM Indexderived EACs and risk register likelihoods and cost consequences
Cost Impact Prioritization by WBS
Quantitative Ranking – 70% Tail
ICRM Customization
• Future versions of ICRM can be easily customized
– Add or subtract Workscreens to make it more
efficient
– Different distributions can be specified for risk
register impacts
– Different EVM performance factors can be applied
– EVM analysis can be applied to risk register cost
consequences
– Cost estimator analysis can be applied to risk register
cost consequences