Transcript No Slide Title

OSD Cost Assessment
Joint Strike Fighter: Case Study of Analyzing
the Costs of a Major Weapon System
David Marzo
Operations Research Analyst, Weapon Systems Cost Analysis Division
U.S. Department of Defense
OSD Cost Assessment and Program Evaluation
[email protected]
(703) 697-2452
June 2011
DISTRIBUTION STATEMENT A. Approved for
1 public release; distribution is unlimited
Agenda
OSD Cost Assessment
• OSD CA Overview
• Recurring Challenges & Considerations
• Joint Strike Fighter Example to illustrate
CAPE estimating proclivities
DISTRIBUTION STATEMENT A. Approved for
2 public release; distribution is unlimited
OSD CAPE Overview
OSD Cost Assessment
• Analytical arm within the Office of the Secretary
of Defense
– Cost Assessment (CA): Mandatory independent cost estimates
and internal discretionary reviews
– Program Evaluation (PE): Broader portfolio review & oversight
• CA also known as the CAIG or Cost Analysis
Improvement CAIG
– Established in 1972
– Bolstered through a series of regulations & acts, most recently
the 2009 Weapon Systems Acquisition Reform Act
DISTRIBUTION STATEMENT A. Approved for
3 public release; distribution is unlimited
OSD CA Today
OSD Cost Assessment
• Staff size: 30 Full-Time Equivalents (FTEs)
• 25 analysts available for CAPE projects -- 16 analysts
qualified to lead CAPE projects
• About 55-60 CAPE estimates per year*
• Average time commitment per cost estimate
•6 calendar months
•8 to 10 FTE-months of effort
• Perform full lifecycle cost estimates (Development, procurement,
military construction, operating & support)
* Includes special studies and Nunn-McCurdy support
DISTRIBUTION STATEMENT A. Approved for
4 public release; distribution is unlimited
Agenda
OSD Cost Assessment
• OSD CA Overview
• Recurring Challenges & Considerations
• Joint Strike Fighter Example to illustrate
CAPE estimating proclivities
DISTRIBUTION STATEMENT A. Approved for
5 public release; distribution is unlimited
Cost Estimating Challenges
OSD Cost Assessment
• Program definition uncertainty
• Fallacy of Detail
• “Black Box” Models
• Optimism
• Actuals: The “Gold Standard”
DISTRIBUTION STATEMENT A. Approved for
6 public release; distribution is unlimited
Program Uncertainty Over Time
OSD Cost Assessment
Cost, Technical, and
Programmatic Uncertainty
Pre
Systems Acquisition
Acquisition
Pre
Systems
Concept
Refinement
Concept
Refinement
Estimating
Techniques
Technical
Reviews
A
Parametric/
Analogy
Systems Acquisition
Engineering &
Manufacturing Development
Production
& Deployment
B
ASR
SRR
Preferred
System
Concept
System
Specification/
CDD
Parametric/
Analogy
C
IBR
SFR
System
Functional
Baseline
PDR
Allocated
Baseline
TRR
CDR
FRR
Product
Baseline
Parametric/
Analogy
Engineering Build-Up (PO & Contractor)
Sustainment
OTRR
IOC
Operations
& Support
FOC
Disposal
PCR
SVR/PRR
ECPR
Product
Baseline
Extrapolation of Actuals
Analogy
Parametric
Engineering Build-Up (PO & ktr)
Extrapolation of Actuals
Analogy
Parametric
Engineering Build-Up (PO&
Ktr)
Program definition uncertainty lessens over time
FOR OFFICIAL
ONLY
DISTRIBUTION STATEMENT A. Approved for
release; USE
distribution
is unlimited
7 public
Fallacy of Detail
OSD Cost Assessment
• Level of aggregation is probably one of the most important
decisions made in an estimate
– Greater complexity/detail is often seen by the authors as an index of estimate
quality (after all, it reflects great effort)
– Subject to the erosion of accuracy for the sake of precision
• Observation: Cost Analysis—as a discipline—inevitably relies on
the proposition of correlation
– Highly correlated elements MUST be consolidated
• Failure to consolidate correlated elements will provide erroneous results
• This is the fundamental flaw in cost risk analysis as generally practiced
– Consolidating correlated cost elements simplifies the estimating effort, and
focuses analytic attention on the fundamental drivers of cost
• Note that the availability of lower level data does not compel its use
Better to have an estimate that is “Approximately Right”
rather than “Precisely Wrong”
DISTRIBUTION STATEMENT A. Approved for
8 public release; distribution is unlimited
“Black Box” Models and Evidence
OSD Cost Assessment
• First requirement of an estimate is that it be trusted
– Estimator is asking senior managers to bet their careers . . .
– Must explain—in quite specific terms—the chain that runs from
data/evidence, through models/methods to final estimate
• Proprietary “closed” models are inherently problematic
– Many commercial forms
– One is left with showing inputs, black box, and estimates
– The best one can say of credibility is the “trust” in the trademark
• As an estimate consumer, one inevitably wonders
– How would I know if X model is appropriate for this application?
– Have the inputs been gamed, particularly through “calibration”?
Other than “trust me”, what do I have to believe to
have confidence in this black box estimate?
DISTRIBUTION STATEMENT A. Approved for
9 public release; distribution is unlimited
Optimism
Delusions of Success
OSD Cost Assessment
By Dan Lavallo and Daniel Kahnemon, Harvard Business Review, July 2003
Key Points from the Article
• When planning major initiatives
–
–
Routinely exaggerate benefits and discount costs when projecting risky project
Sets up project for failure; psychologists “planning fallacy” -- spin success scenarios and overlook
potential for mistakes
• Optimism traced to cognitive biases and organizational pressures
–
–
–
–
People highly optimistic most of the time – exaggerate talents and degree of control; attribute
negative consequences to external factors
Competition ($, time) for new projects – incentive to accentuate the positive
Anchoring magnifies optimism; initial plan accentuates positive, skews subsequent analysis
Most pronounced for initiatives companies have never attempted before
• Temper with “outside view”
–
–
Supplements traditional forecasting w/ statistical analysis of analogous efforts – reality check on
initiative inside view
Don’t remove optimism, but temper effects -- balance between optimism and realism - goals
(motivate) and forecasts (decide whether to make commitment in the first place)
DISTRIBUTION STATEMENT A. Approved for
10 public release; distribution is unlimited
Actuals: The Gold Standard
OSD Cost Assessment
• Actuals have one sterling advantage: they reflect all
“unknown/unknown” problems of achieving a capability
• Actuals-based estimates are easy to explain
• Two distinct cases of actuals estimating:
– Early Actuals on a specific program to extrapolate
future lots
– Actuals from other programs, often adjusted for
content/scope, to forecast costs of new program
Even if you don’t use actuals for a particular element
of cost, presenting relevant ones helps to calibrate
the discussion
DISTRIBUTION STATEMENT A. Approved for
11 public release; distribution is unlimited
Is Cost A Self-Fulfilling Prophecy?
OSD Cost Assessment
• Common Complaints
– “CAPE Estimates are inherently negative and reinforce past
mistakes”
– “I don’t need all of this extra money—if you give it to me, the
program will just spend it unnecessarily.”
– “Cost is a self-fulfilling prophecy”
• Thoughts
– Programs can address known “mistakes” from past programs
and not repeat them – but each program has its unique
unknowns—the inevitable challenges of complex weapon
systems
– Trick is to find the dividing line between too much and not
enough –the “thesis analogy”
– Good management has an important role
DISTRIBUTION STATEMENT A. Approved for
12 public release; distribution is unlimited
Agenda
OSD Cost Assessment
• OSD CA Overview
• Recurring Challenges & Considerations
• Joint Strike Fighter Example to illustrate
CAPE estimating proclivities
DISTRIBUTION STATEMENT A. Approved for
13 public release; distribution is unlimited
Joint Strike Fighter
OSD Cost Assessment
• What is the program status?
• How is CAPE involved?
• How do we approach estimating it? What are
some of the unique estimating challenges?
• How does the Department react to differing cost
estimates?
• What are the impacts of international
procurements?
DISTRIBUTION STATEMENT A. Approved for
14 public release; distribution is unlimited
JSF – Program Status
OSD Cost Assessment
• Program underwent a “Nunn-McCurdy” cost
breach in summer 2010
– Program originally based in 2002
– Projected unit cost >50% resulted in critical breach
– Department of Defense must re-assess requirements, cost,
status
– Summer 2010: Department states no other alternative meets
requirements within acceptable cost—proceed with JSF
DISTRIBUTION STATEMENT A. Approved for
15 public release; distribution is unlimited
JSF Program Status
OSD Cost Assessment
• Fall 2010 : Comprehensive Technical Baseline
review resulted in a restructured development
program
• Block 3 Qualification Projected Complete in
2016
• CAPE position: reasonable development
baseline established; focus on execution;
challenges remain
DISTRIBUTION STATEMENT A. Approved for
16 public release; distribution is unlimited
CAPE Involvement
OSD Cost Assessment
• Departmental Internal Requests
– 2008 Joint Estimating Team (“JET 1”)
– 2009 Joint Estimating Team (“JET II”)
• Statutory Requirements for Cost Estimates
– 2010 Nunn McCurdy Cost Breach Review: Development and
procurement
– 2011 Milestone B Review (Ongoing): Development,
procurement, operating and support
DISTRIBUTION STATEMENT A. Approved for
17 public release; distribution is unlimited
JSF Estimating Approach
OSD Cost Assessment
• Development
– Function of schedule and personnel staffing levels; “time is
money”
– Remaining schedule is a function of test point burndown
– Project actual costs forward; shaped by legacy history
• Procurement
– Pre Nunn McCurdy: Based on analogy to F/A-18, F-22; adjusted
for weight and system complexity
– Since Nunn McCurdy
• Based on actuals from JSF lots; bounded by legacy history
• Separately model each section of aircraft; and separate
major contractors
DISTRIBUTION STATEMENT A. Approved for
18 public release; distribution is unlimited
Theoretical Example
OSD Cost Assessment
Example
1200
1000
800
600
Example
400
200
0
0
5
10
15
20
25
DISTRIBUTION STATEMENT A. Approved for
19 public release; distribution is unlimited
Challenge of Modeling Commonality
OSD Cost Assessment
Past
Future
?
CTOL
3. Fundamental Difference?
STOVL
2. Temporary Discontinuity?
1. 100% Commonality?
DISTRIBUTION STATEMENT A. Approved for
20 public release; distribution is unlimited
Data Example #1
OSD Cost Assessment
Aircraft Section
SDD
CTOL
STOVL
CV
Production
HOURS
0
10
20
30
40
DISTRIBUTION STATEMENT A. Approved for
21 public release; distribution is unlimited
Data Example #2
OSD Cost Assessment
Example 2
SDD
CTOL
STOVL
CV
Production
0
5
10
15
20
25
30
35
DISTRIBUTION STATEMENT A. Approved for
22 public release; distribution is unlimited
History as a Bound
OSD Cost Assessment
JSF Estimates vs. Legacy History
Compare CAPE Estimates to
analogous historical experience –
Is our estimate within a logical
range given legacy experience?
1
10
100
DISTRIBUTION STATEMENT A. Approved for
23 public release; distribution is unlimited
FOR OFFICIAL USE ONLY // REL TO USA, GBR MOD, ITA MOD, NLD MOD, TUR MND, CAN DND, AUS DOD, DNK MOD, and NOR MOD
The Challenge of Concurrent Development and Production
OSD Cost Assessment
Actuals
Cumulative Aircraft Quantity
Flight Test Hours
Projections
-2
-1
0
1
2
3
4
5
Years from Start of DT Flight Test
6
7
8
24
Operating and Support Cost
• Operating and support (O&S) cost
OSD Cost Assessment
– It will cost more to operate and maintain JSF than legacy
fighter platforms; however, JSF is more capable
– Significant drivers of O&S cost (e.g., Depot Level Reparables,
engine/fuel consumption) are driven by inherent aircraft
characteristics already designed in (e.g., stealth, complex
mission systems)
– Other significant drivers of O&S cost are driven by ops
requirements (e.g., mission personnel & flight hours)
The Department is continually examining the major drivers of
sustainment costs and aims to capitalize on various
opportunities to reduce costs
DISTRIBUTION STATEMENT A. Approved for
25 public release; distribution is unlimited
Annual “Steady State” Cost Example
OSD Cost Assessment
Element of Cost
% of Total
Cost Drivers
Mission Personnel
~20%
Crew ratios & staffing, # of
Aircraft
Unit Level Consumption
~20%(Fuel)
~25% (DLR)
~5%(Other)
# of Aircraft,
Engine Fuel Consumption,
System Reliability, Cost to
repair
Depot Maintenance
5%
Depot Induction
Contractor & Sustaining Support
~15% (Supt Heads,
Other)
~10% (Mods)
Support Staff
& Aircraft
Total
100%
Acronyms:
PAA: Primary Aircraft Authorized
TAI: Total Aircraft Inventory
DLR: Depot Level Reparables
DISTRIBUTION STATEMENT A. Approved for
26 public release; distribution is unlimited
Other Challenges of Estimation
OSD Cost Assessment
• Concurrency of phases—same contractor staff
charging simultaneously to development,
production, and sustainment—need to account
for correctly
• International buy profiles – international
purchases move processes further down the
curve; when US defers purchases, how do we
model shifts in international profiles?
• Evolution of production processes – what is
built in house now may be outsourced later
DISTRIBUTION STATEMENT A. Approved for
27 public release; distribution is unlimited
Department Treatment of Cost Estimates
OSD Cost Assessment
• Department leadership has a choice against which
estimates to baseline and budget against
– “Fund to program estimate” or “Fund to CAPE estimate”
– Each program is different
• For JSF, this has been an evolutionary process—
– For development, incremental movement over past 3 years
towards funding to CAPE and Tech Baseline Review estimates
– For procurement, Department has funded to CAPE estimates.
There is still ambiguity, however, with respect to the effects of
commonality and its impact on costs
• Assessing risk and funding to the 50% estimates vs
80% estimates
DISTRIBUTION STATEMENT A. Approved for
28 public release; distribution is unlimited
Impact of International Participation
OSD Cost Assessment
• We estimate that the US gets about a 5% unit
cost benefit from international participation
• International participation does introduce
complexities
– Establishment of production capabilities in other countries
– Shifting quantity profiles
DISTRIBUTION STATEMENT A. Approved for
29 public release; distribution is unlimited
Closing Comments
OSD Cost Assessment
• CAPE has responsibility for performing cost estimates
in response to both US statute as well as internal
Departmental requests
• Our approach emphasizes:
– Using a combination of historical analogous data & actual costs when
available
– Relatively simple but sound logic
• JSF is a good example of our involvement and cost
estimating methodology; we recognize that estimating
JSF is a dynamic process informed by ongoing dialogue
with the contractor and program office
DISTRIBUTION STATEMENT A. Approved for
30 public release; distribution is unlimited