Transcript Company Capabilities - Department of Defence

Joint Analysis of Cost and Schedule (JACS)
Australian Department of Defence
2nd Cost Estimation Conference
29 - 30 October 2012
Alfred Smith, CCEA
Jennifer Kirchhoffer, CCEA
 Los Angeles  Washington, D.C.  Boston  Chantilly  Huntsville  Dayton  Santa Barbara
 Albuquerque  Colorado Springs  Goddard Space Flight Center  Johnson Space Center  Ogden  Patuxent River  Washington Navy Yard
 Ft. Meade  Ft. Monmouth  Dahlgren  Quantico  Cleveland  Montgomery  Silver Spring  San Diego  Tampa  Tacoma
 Aberdeen  Oklahoma City  Eglin AFB  San Antonio  New Orleans  Denver  Vandenberg AFB
28 Aug 2012 PRT-148
Agenda
 What is JACS?
 Overview of the JACS modeling process
 Key reports from a well constructed JACS model
 Concluding remarks
28 Aug 2012 PRT-148
Approved for Public Release
2 of 35
What is Joint Analysis of Cost and
Schedule (JACS)?
 Cost, schedule and risk assessments traditionally have
been performed by separate teams of professionals
 In recent years, it has become more common for the cost
analyst to report a “risk adjusted” result as a budget
recommendation rather than a point estimate
 However, it appears that cost uncertainty models routinely:

try to force a 70 or 80% cost result into the point estimate schedule

ignore risk management team statements like “High probability this event
will occur and if it does, the consequence will be severe”
Joint Analysis of Cost and Schedule is a disciplined,
systematic and repeatable process to integrate three
critical pieces of information: Cost, Schedule, Risk
28 Aug 2012 PRT-148
Approved for Public Release
3 of 35
Current Approach to Model
Cost Estimating Uncertainty
Cost = a + bxc
Historical data points
Cost
Sources of Uncertainty:
• Cost estimating method
• Cost method inputs
Input, e.g., weight
28 Aug 2012 PRT-148
Approved for Public Release
Focus is on estimating total
cost uncertainty with limited
influence from duration
uncertainty or potential
events that may influence
cost/schedule.
4 of 35
Evolving Trends in
Uncertainty and Risk Analysis

Capture schedule uncertainty on Time-Dependent Costs
Time-Dependent (TD)
[Level of Effort - LOE]
 Inclusion of Discrete Risks (5x5’s)
Risk 1
2
n
1
Translate 5x5 into probability of
occurrence times uncertain
consequence which can impact
cost and/or duration of one or
more tasks
Risk 2
.
.
.
Risk n
28 Aug 2012 PRT-148
Approved for Public Release
5 of 35
Integrated Risk & Uncertainty Landscape –
the JACS Paradigm
TI = Time-Independent Cost, e.g., Materials
Uncert
TI $
Uncert
Uncert
TI $
Uncert
Uncert
TD $ = Segment Duration X Burn Rate
Uncert
Project
Start
TI $ Uncertainty
Uncert
TI $
Duration
Uncertainty
Uncert
Project
End
TI $
Uncert
Probability of
Occurrence
Uncert
Task Duration
TI $
Uncert
Uncert
TI $
Risk Register
TD = Time-Dependent Cost, e.g. ‘marching army’ cost
28 Aug 2012 PRT-148
Approved for Public Release
Uncert
TD $ = Segment Duration X Burn Rate
Burn Rate
Uncertainty
6 of 35
Agenda
 What is JACS?
 Overview of the JACS modeling process
 Key reports from a well constructed JACS model
 Concluding remarks
28 Aug 2012 PRT-148
Approved for Public Release
7 of 35
The JACS Process:
Develop the Analysis Schedule
Risk
Sched
Collect
Sched
Data
Create
Analysis
Schedule
Validate
Before
Continuing
Cost
28 Aug 2012 PRT-148
Approved for Public Release
8 of 35
The Need for an Analysis Schedule
 Joint analysis of cost and schedule begins with a model of the
schedule logic

Serves as the backbone for the analysis

Cost, risks and uncertainty are mapped into the logic to assess impacts
 Project/program integrated master schedules (IMS) are unsuitable for
this role

They are generally too big, complex and too detailed

Logic common in an IMS can be a problem for a JACS analysis (e.g.,
constraints)
 A JACS appropriate schedule must be created from available data
(including the program IMS)

Typically referred to as an “analysis schedule”
28 Aug 2012 PRT-148
Approved for Public Release
9 of 35
Attributes of an Analysis Schedule
√
Captures the major work-flows of the project IMS
√
Provides insight into major cross-dependencies within or across
management responsibility boundaries
√
Creates a solid framework to capture cost / schedule uncertainties
and discrete risk events
√
Structured around management/ budget responsibility
√
Allows mapping of budgeted work effort to schedule scope
√
√
Identifies key tasks that support major deliverables/ tracking items
√
√
Aligns with cost/budget data
Detailed IMS step by step work items and task flows are combined
while maintaining critical path logic
Has traceability and transparency to the more detailed IMS
28 Aug 2012 PRT-148
Approved for Public Release
10 of 35
The JACS Process:
Map Costs to Schedule Tasks
Risk
Sched
Collect
Sched
Data
Create
Analysis
Schedule
Update
Analysis
Schedule
Collect
Cost
Data
Identify
as
TD or TI
Map to
Sched
Activities
Cost
28 Aug 2012 PRT-148
Approved for Public Release
11 of 35
Mapping of Cost to Schedule

Easier to map cost model results to a schedule model rather than replicating schedule
network logic in a cost model


Schedule models are generally populated with throughputs, that is they don’t allow equations to
estimate the cost of one task based on the cost of another or some technical characteristics
Mapping cost estimate to a schedule model is simplified by:



Unifying cost (often product based) and schedule (often task based) work breakdown structures
Specifying Time Dependent and Time Independent costs and their uncertainty separately
Defining how the TI or TD cost is phased over the task duration
Total Cost
TD Cost
TI Cost
TD Phasing
TI Phasing
28 Aug 2012 PRT-148
Approved for Public Release
12 of 35
The JACS Process:
Mapping the Risk Register to Schedule Tasks
Risk
Collect
Risk
Data
Assign
Likelihood,
Estimate
Impact
Map to
Sched
Activities
Collect
Sched
Data
Create
Analysis
Schedule
Update
Analysis
Schedule
Collect
Cost
Data
Identify
as
TD or TI
Map to
Sched
Activities
Sched
Cost
28 Aug 2012 PRT-148
Approved for Public Release
13 of 35
Discrete Risk
 Defined as: If risk event A occurs, there is a cost consequence or opportunity.
The probability of A occurring is x%

Often modeled as a separate task inserted into the schedule network
 If there are only a few such risk events, treat as discrete what-if cases
(event cost or schedule impact is either “in” or “out” of the estimate)

Point estimate includes the full impact (when there are few)
 If there are many such risk events, model using the Yes/No distribution
(also known as the Bernoulli distribution)


When there are many, there is no standard on how to treat the Point Estimate:

Include none and assess separate from the Point Estimate?

Include all (worst case scenario)?

Include the sum of the expected values?  this is a common approach
The risk register should account for:



Uncertainty of the cost consequence or opportunity
Correlation across duration and cost uncertainties
Probabilistic branching  rarely attempted
28 Aug 2012 PRT-148
Approved for Public Release
14 of 35
5x5 Matrix Definitions*
Risk Management conventions*

Consequence






Likelihood of Occurrence






1 Minimal or no impact
2 Additional resources < 5%
3 Additional resources = 5-7%
4 Additional resources = 7-10%
5 Additional resources > 10%
Consequence

0
0
1
2
1
0
0
0
1
0
1
2
1
4
0
1
3
4
2
4
0
1
2
2
4
1 2 3 4 5
1 Remote (10%)
2 Unlikely (30%)
3 Likely (50%)
4 Highly likely (70%)
5 Near certainty (90%)
Likelihood
Opportunities

5
4
3
2
1
Should have a separate matrix to
address potential opportunities to save
(not addressed in our example)
Total Risks =
36
High =
13
Medium =
13
Low =
10
*Note: Taken from Risk Management Guide for U.S. DoD Acquisition
Every Agency will set its own standards for these values
28 Aug 2012 PRT-148
Approved for Public Release
15 of 35
The JACS Process:
Assigning Uncertainties
Risk
Collect
Risk
Data
Assign
Likelihood,
Estimate
Impact
Map to
Sched
Activities
Assess Event
Cost and
Duration
Uncertainty
Collect
Sched
Data
Create
Analysis
Schedule
Update
Analysis
Schedule
Assess
Duration
Uncertainty
Collect
Cost
Data
Identify
as
TD or TI
Map to
Sched
Activities
Assess
Cost
Uncertainty
Sched
Cost
28 Aug 2012 PRT-148
Approved for Public Release
16 of 35
The Only Certainty is Uncertainty
 Most common method to address uncertainty is to assign
distributions to uncertain elements and run a Monte Carlo simulation

Objective Uncertainty Distributions

Derived from historical data
 Something you can defend mathematically and historically

Subjective Uncertainty Distributions

Based more on expert opinion than statistical analysis
 Often necessary due to lack of information to characterize it objectively
 Every duration, cost and consequence in the model is generally an
estimate and therefore uncertain

Time Independent Cost Uncertainty

Time Dependent Cost (Burn Rate or Resource Utilization) Uncertainty

Duration Uncertainty

Discrete Risk Uncertainty –Probability of Occurrence
 Uncertainty should be applied in a consistent manner across the
entire model
28 Aug 2012 PRT-148
Approved for Public Release
17 of 35
Typical Uncertainty Distributions
DISTRIBUTION
TYPICAL APPLICATION
KNOWLEDGE OF
MOST LIKELY
NUMBER OF
PARAMETERS
REQUIRED
RECOMMENDED PARAMETERS
50% (median) and high value
Lognormal
Default when no better info.
Mean or median
Probability skewed right.
known better than the
Replicate another model result.
mode (most likely)
Power OLS CER uncertainty.
2
(some tools have a 3rd parameter : “Location” .
By default, it is zero. Used to “slide” the
lognormal left or right (even into negative region).
Triangular
Expert opinion. Finite min/max.
Chance reduces towards
endpoints. Skew possible.
Labor rates, labor rate
adjustments, factor methods
Good idea
3
Low, mode, and high
BetaPert
Like triangular, but treats mode
as 4 times more important than
min or max.
Very good idea
3
Low, mode, and high
Beta
Like triangular, but min/max
region known better than mode.
Not sure
4
Min, low, high, and max
Normal
Equal chance low/high.
Unbounded in either direction
Linear OLS CER uncertainty.
Good idea, but
unbounded in either
direction
2
Mean/Median/Mode
and high value
Uniform
Equal chance over uncertainty
range. Finite min/max.
No idea
2
Low and High
(some tools require min and max)
Note: Low/high are defined with an associated percentile (by default 15/85). Min/Max are the absolute
lower/upper bound (also known as the 0/100). Some policies require truncation at zero.
28 Aug 2012 PRT-148
Approved for Public Release
18 of 35
The Double Counting Time Dependent (TD)
Cost Uncertainty Dilemma
 Consider how TD cost is calculated

Typically calculated using: Duration * Cost/Day (duration in days)

Cost/Day can be derived from similar, completed project totals

Cost/Day factor may already capture cost and duration uncertainty

Uncertainty on Duration and Cost/Day factor may be double counting
 However, basis for Cost/Day must be carefully understood

Cost/Day factor may change as the duration changes

Shorter duration achieved by using more resources ($/day larger)
 Shorter duration achieved by using more expensive resources ($/day larger)
 Longer duration a consequence of scarce resources ($/day smaller)

In these contexts, uncertainty on Duration and Cost/Day is appropriate

Correlation between the two should be considered as well
28 Aug 2012 PRT-148
Approved for Public Release
19 of 35
The JACS Process:
Apply Correlation, Validate then Run
Collect
Risk
Data
Assign
Likelihood,
Estimate
Impact
Map to
Sched
Activities
Assess Event
Cost and
Duration
Uncertainty
Collect
Sched
Data
Create
Analysis
Schedule
Update
Analysis
Schedule
Assess
Duration
Uncertainty
Collect
Cost
Data
Identify
as
TD or TI
Map to
Sched
Activities
Assess
Cost
Uncertainty
Sched
Cost
28 Aug 2012 PRT-148
Approved for Public Release
Apply Correlation
Risk
Validate
File
Run
Analysis
20 of 35
Apply Correlation

JACS models developed in any tool will have limited “functional” correlation between
uncertain elements (correlation due to model mathematics)

Consider applying correlation to ensure elements that should, do move together

Tools should allow application of correlation across any uncertain elements


Just because you can apply correlation, does not mean you should!

Correlating Dur with TD may be double counting if TD is modeled as a function of Duration
Ideally, measure the correlation present in the model first, then apply as needed
28 Aug 2012 PRT-148
Approved for Public Release
21 of 35
Perform a Comprehensive Health Check
to Ensure Model Validity

Prior to running an integrated simulation, the user should review the file to ensure that
there are no potential issues within the file

There are many commercial tools that will perform a schedule health assessment


These tools look for violations of schedule model best practices (e.g., task without predecessor)
A JACS Health check is more comprehensive and should also uncover issues such as:



Critical issues: e.g., cost not phased, invalid uncertainty, duration with no cost, invalid correlation
Warnings: e.g., uncertainty on zero cost, risk event with zero probability, baseline outside uncertainty
Information: e.g., extraordinary float, risk event turned off, duration without uncertainty, no correlation
28 Aug 2012 PRT-148
Approved for Public Release
22 of 35
The JACS Process
Collect
Risk
Data
Assign
Likelihood,
Estimate
Impact
Map to
Sched
Activities
Assess Event
Cost and
Duration
Uncertainty
Collect
Sched
Data
Create
Analysis
Schedule
Update
Analysis
Schedule
Assess
Duration
Uncertainty
Collect
Cost
Data
Identify
as
TD or TI
Map to
Sched
Activities
Assess
Cost
Uncertainty
Sched
Cost
28 Aug 2012 PRT-148
Approved for Public Release
Apply Correlation
Risk
Validate
File
Run
Analysis
23 of 35
Agenda
 What is JACS?
 Overview of the JACS modeling process
 Key reports from a well constructed JACS model
 Concluding remarks
28 Aug 2012 PRT-148
Approved for Public Release
24 of 35
Typical Output from a Cost Only
Uncertainty Analysis
 Cost uncertainty is generally performed on total costs
 Rarely linked to schedule uncertainties

Unable to relate a specific cost result to a specific schedule result
 No insight into uncertainty by year, or the impact of schedule slips
28 Aug 2012 PRT-148
Approved for Public Release
25 of 35
A JACS Model Relates Uncertain Cost
with Uncertain Duration
Each dot is
the cost and
schedule
result from
one trial
New
View identifies
Shows Cost
Lower
joint
70%
Costleft
Confidence Level
Aligned
with
Schedule
probability
of meeting
BOTH
(CCL)
Indicates
Reserves
70% cost
and schedule
Capture
Schedule
Growth
(58.4%)
28 Aug 2012 PRT-148
Approved for Public Release
26 of 35
Annual Funding Charts
 The JACS Model provides total and annual uncertainty results
 Identifies both WHAT the uncertainty is and WHEN it is
Point Estimate vs Annual Uncertainty
28 Aug 2012 PRT-148
Approved for Public Release
27 of 35
Additional Views Are Possible
Point Estimate similar to Mean
Need to use Reserves to
move point estimate
towards the Mean
28 Aug 2012 PRT-148
Approved for Public Release
28 of 35
Additional Views Are Possible

There are many tools
available and they all
produce some version
of most of these charts
28 Aug 2012 PRT-148
Approved for Public Release
29 of 35
Most Misunderstood Chart:
Tornado

The term “Tornado” is used in a variety of contexts across schedule, cost and
general uncertainty analysis; tool documentation and the literature

Various definitions include:

Find Cost Drivers: Create a low and high “what-if” case for every “cost driver” based on
their 10/90% values (evaluated one at a time)…find driver that has most impact


Find Uncertainty Drivers: Measure the correlation between each defined input
distribution and the output of interest. The highest correlation identifies find distribution
that has the most impact on the total uncertainty.


Problem: How you bin the trials has massive effect on results; may require huge number of trials
to obtain stable results
Brute Force: Use one of the above methods to find the top 10, then run the simulation
10 times turning one at a time and record the impact on the output of interest.


Problem: Element that is highly correlated with output may have nothing to do with that output
Hybrid: Sort the trial results by cost driver (one at a time) to find associated bounds on
the output mean or selected percentile


Problem: ignores correlation effects; does not address drivers influenced by multiple distributions
Problem: Very tedious, time consuming and may be misleading if its loss triggers unexpected
conditions in the simulation
Conclusion: Beware of the Tornado chart!
28 Aug 2012 PRT-148
Approved for Public Release
30 of 35
Tornado Chart Best Practices

Total Cost Drivers: (left chart)



Measure impact on inflated results, not constant year cost results
Run simulation to find combined uncertainty results for 10/90%
Total Uncertainty Drivers: (right chart)
7k Trials Required

Adjust the algorithm to account for applied correlations
 Identify how many trials used (demonstrate sufficient)

Notice the answers are quite different
28 Aug 2012 PRT-148
Approved for Public Release
31 of 35
Prototype Integrated Time-Based View of
Costs, Schedule, Budget, and Risks
 Solid line is the point estimate (BCWS)
 Dots are cost/schedule uncertainty results at various milestones
 X- Axis identifies when key risk register events occur

Size of symbol indicates impact, color indicates probability (or type)
28 Aug 2012 PRT-148
Approved for Public Release
32 of 35
Agenda
 What is JACS?
 Overview of the JACS modeling process
 Key reports from a well constructed JACS model
 Concluding remarks
28 Aug 2012 PRT-148
Approved for Public Release
33 of 35
What Data Do We Need?

Schedule + Risks + Costs = JCL Model

Can use any of the below, as long as you have one data source for each
category

Schedule
 Detailed IMS
 Simple schedule with just a few moving parts

Costs – preferably time phased
 Budget data
 Lower level cost data (LCC databases) / EVM data
 Parametric costs

“Risks”
 Risk management system
 What –if’s
 Basic uncertainty
Keep it simple and use what you have
28 Aug 2012 PRT-148
Approved for Public Release
34 of 35
Why JACS is Becoming Popular
 Delivers an integrated view to Project Managers:

Schedule probability of success

Cost probability of success

Impact of discrete program risks

Results of any number of what-if scenarios

Both total and annual funding reserve requirements
 For NASA: Regulatory requirement (7120.5 E)

Identify a cost and schedule range by milestone KDP (~milestone) B

Baseline program to a specific joint probability level by KDP
(~milestone) C
28 Aug 2012 PRT-148
Approved for Public Release
35 of 35
Questions?
28 Aug 2012 PRT-148
Approved for Public Release
36 of 35