Estimating the Cost of a Defense Contract with Performance Indices Do the Old Methods Still Work?

David S. Christensen, Ph.D.

Southern Utah University (435)865-8058 [email protected]

Air Force Institute of Technology 20 May 2002

Overview

• Background – Earned Value Management Systems – Variance Analysis • Evaluating the EAC • Are the rules still valid?

Earned Value Management Systems (EVMS)

Links:

•

http://www.acq.osd.mil/pm

•

http://www.suu.edu/faculty/christensend/ev-bib.html

Cost Management Reports (DOD 5000.2-R)

Contract Cost Data Report Contract Fund Status Report Cost Schedule Status Report Cost Performance Report

Cost/Schedule Status Report

• Monthly • Cost thresholds(FY00$)

– Prod < $315 MIL – R&D < $73 MIL

• FFP contracts normally exempt

Cost Performance Report

• Monthly • Cost thresholds

– Prod > $315M – R&D > 73M

• FFP normally exempt • EVM criteria required

Objectives of EV Criteria

To provide insight into contract progress, implementing EVMS provides the program manager with data which: – Relate time-phased budgets to specific contract tasks/work; – Indicate work progress; – Properly relate cost, schedule, and technical accomplishment; – Are valid, timely, and auditable; – Supply managers with information at a practical level of summarization; and – Are derived from the same internal EMVS used by the contractor to manage the contract

Cost Performance Report

• Does it provide data or information?

• Is it reliable?

• Is it relevant?

Relevance

Decision Usefulness

Reliability Predictive Value EAC Feedback Value VARIANCE ANALYSIS Timely MONTHLY Verifiable A C CESS Valid Neutral SAME DATABASE OBJECTIVE DATA Comparability/Consistency Benefits > Costs THRESHOLDS, SUMMARY REPORTS DIDs, CWBS

THE BASIC DATA ELEMENTS

• Actual Cost of Work Performed (ACWP) $ • Budgeted Cost of Work Scheduled (BCWS) – aka “Planned Value” ACWP BCWS BCWP

•

Budgeted Cost of Work Performed (BCWP)

–

aka “Earned Value

” NOW TIME EAC BAC

THE BASIC DATA ELEMENTS

Estimate At Completion (EAC) Budget At Completion (BAC) $ EAC NOW ACWP BCWS BCWP TIME BAC

THREE VARIANCES

Variance at Completion = BAC - EAC Cost Variance = BCWP - ACWP Schedule Variance = BCWP - BCWS $ EAC VAC ACWP BCWS BCWP SV CV NOW TIME BAC

Performance Measurement Baseline (PMB)

$ EAC NOW ACWP

PMB

BCWS BCWP TIME BAC

1. IDENTIFY THE SCOPE OF W ORK

2. EXTEND TO W ORK PACKAGE LEVEL

DESIGN MATERIAL ORDER ASSEMBLY TEST 3. ARRANGE W ORK PACKAGES IN ORDER

JAN FEB MAR APR MAY JUN DESIGN MATERIAL ORDER ASSEMBLY TEST 4. SCHEDULE WORK PACKAGES

DESIGN MATERIAL ORDER ASSEMBLY LABOR 10 MATERIAL 2 OTHER 2 10 3 0 20 10 0 TEST 1 0 5 TOTAL 14 13 30 6 5. BUDGET THE W ORK PACKAGES

JAN FEB MAR APR MAY JUN DESIGN 2 9 3 MATERIAL ORDER ASSEMBLY 4 7 2 9 15 TEST 6. SPREAD BUDGET OVER TIME 5 3 1 3

JAN FEB MAR APR MAY JUN DESIGN 2 9 3 MATERIAL ORDER ASSEMBLY 4 7 2 9 15 TEST BCWS 2 13 19 17 7. CALCULATE MONTHLY BCWS 5 3 8 1 3 4

PERFORMANCE MEASUREMENT BASELINE (PMB )

60 50 40 30 20 10 0 JAN FEB 8. CREATE PMB MAR APR MAY JUN

Numerical Example

The Plan Build 5 sections of railroad track in 5 weeks for $5000

BCWS per week = ?

BAC = ?

Performance After two weeks, one section is completed for $2600

Cumulative ACWP = ?

Cumulative BCWS = ?

Cumulative BCWP = ?

EAC = ?

$

Numerical Example

EAC = ?

BAC = $ 5000 ACWP = $2600 BCWS = $2000 BCWP = $1000 2 Weeks 5 Weeks

Variance Analysis

An example from an aircraft project

AIA LASER SCANNER AIRCRAFT

Dollars in thousands WBS L Description 2 Air Vehicle 3 Air Frame 3 Avionics 3 Auto Flt Cont 3 Cent Integ 3 Aux Elec Eq 3 Wpns Deliv 2 NonStd Spt Eq 2 Sys T&E 3 Dev Tests 3 Mockups 3 T&E Support 3 Sys T&E Integ 2 Sys/Prog Mngt 2 Data 1 Acft Sys (Direct) Indirect G&A UB PMB MR Total 1-31 October, 1991 Cumulative BCWS 69171 62377 2896 260 1166 1688 784 396 36175 34210 854 635 476 15907 139 121788 57619 19735 BCWP 65533 58944 2780 248 1127 1689 745 414 33154 31165 897 616 476 15901 134 115136 54086 18614 199142 199142 187836 187836 ACWP 72611 66855 2425 242 877 1561 651 326 37160 35233 904 586 437 15274 119 125490 58000 20184 203674 203674 SV -3638 -3433 -116 -12 -39 1 CV At Completion BAC -7078 174269 LRE 177187 -7911 355 6 250 128 151900 6230 3941 3679 5014 155300 5900 3928 3639 4971 -39 18 -3021 -3045 43 -19 0 -6 -5 30 39 627 15 -6652 -10354 -3533 -1121 94 88 -4006 -4068 -7 -3914 -1570 -11306 -15838 -11306 -15838 3505 3474 92417 77800 1084 12700 833 34000 324 304484 127116 51297 2200 485097 32534 517631 3449 3474 96417 81800 1101 12700 816 33600 314 310992 131086 50502 2200 494780 21891 516671 -9683 10643 960 VAC -2918 -3400 330 13 40 43 56 0 -4000 -4000 -17 0 17 400 10 -6508 -3970 795

Level 4 detail supporting Level 3, Air Frame AIA LASER SCANNER AIRCRAFT

1-31 October, 1991 Dollars in thousands WBS L Description 4 Basic Struct 4 Flt Veh Power 4 Envi Cont Sys 4 Crew Station 4 A/V Design Spt 4 Eng Equip 4 Aux Control 4 Landing Gear 4 Integ/Assembly 4 A/V Tech Spt 3 Air Frame Cumulative BCWS 15500 10350 3057 9700 11600 2264 1830 591 2385 5100 62377 BCWP 14000 9900 2985 9056 11500 2240 1820 503 1940 5000 58944 ACWP SV 16812 -1500 11500 2660 -450 -72 12400 11500 2150 2000 -644 -100 -24 -10 514 2650 -88 -445 4669 -100 66855 -3433 CV -2812 -1600 325 At Completion BAC 51500 20000 6325 LRE 53000 21500 6450 -3344 0 90 -180 -11 -710 331 -7911 18700 15500 5850 6384 2950 15700 8991 151900 18800 15600 5800 6584 2986 15880 8700 155300 VAC -1500 -1500 -125 -100 -100 50 -200 -36 -180 291 -3400

Level 5 detail supporting Level 4, Crew Station AIA LASER SCANNER AIRCRAFT

1-31 October, 1991 Dollars in thousands WBS L Description 5 Cont & Display 5 Crew Accomoda 5 Crw Escape/Safety 5 Life Support 5 Crw Stat Integ 4 Crew Station Cumulative BCWS 3000 1400 3900 600 800 9700 BCWP 2900 1300 3500 586 770 9056 ACWP 3400 1800 5400 800 1000 12400 SV -100 -100 -400 -14 -30 -644 CV At Completion BAC LRE -500 -500 -1900 -214 -230 -3344 4800 3000 8000 1000 1900 18700 4400 3300 8000 1100 2000 18800 VAC 400 -300 0 -100 -100 -100

CREW ESCAPE AND SAFETY - Cost Account

Cost Element Labor Material A Material B Other BCWP 200,000 hours @ $7.80/hr 5,000 lbs @ $200/lb 16,000 lbs @ $50.00/lb .14 x Material A Analysis of Cost Variance (CV) by Element of Expense ($000) ACWP 234,000 hours @ $10.00/hr 8,000 lbs @ 187.50/lb 27,000 lbs @ $50.00/lb .14 x Material A Labor Material A Material B Other Total BCWP 1560.0

1000.0

800.0

140.0

3500.0

ACWP 2340.0

1500.0

1350.0

210.0

5400.0

CV -780.0

-500.0

-550.0

-70.0

-1900.0

Analysis of Cost Variance (CV) by Price and Usage ($000) Labor Material A Material B Other Total Price -514.8

100.0

0.0

14.0

-400.8

Usage -265.2

-600.0

-550.0

-84.0

-1499.2

CV -780.0

-500.0

-550.0

-70.0

-1900.0

Evaluating the Estimate At Completion Using Earned Value Performance Indices

Estimate At Completion(EAC)

• The EAC appears on the

Cost/Schedule Status Report

the

Cost Performance Report

• Accuracy controlled by EVMS Criteria • Factors to consider – Actual Costs to date – Performance to date – Cost and schedule variances – Reliability and relevance of data Overhead and Inflation rates Future performance on work Changes to requirements Organizational culture

EAC Formula

EAC = Costs to date + Estimated Cost of Remaining Work $ Predicted final cost Plan Cost to date Performance to date now Time

EAC Formula

EAC = ACWPcum + [(BAC - BCWPcum)/Performance Factor] $ EAC BAC ACWP BCWP now Time

Performance Factor

Usually a performance index: • CPI • SPI • Combination of CPI and SPI • w1(CPI) +w2(SPI), where w1 +w2 =1 • CPI x SPI

Cost Performance Index

Example Data BCWSc = $10 million BCWPc = $8 million ACWPc = $12 million BAC = $20 million LRE = $25 million CPI = BCWP/ACWP

= 8 / 12 = 0.67

Interpretation Cumulative, Current, Average

Schedule Performance Index

Example Data BCWSc = $10 million BCWPc = $8 million ACWPc = $12 million BAC = $20 million LRE = $25 million SPI = BCWP / BCWS

= 8 / 10 = 0.8

Interpretation Cumulative, Current, Average Value is 1 at end of contract

$

BCWS = BCWP = BAC when all work is completed

EAC BAC ACWP BCWS now BCWP Time

Combinations of CPI and SPI w1(CPI) + w2(SPI)

• .8(CPI) + .2(SPI) is an AF favorite • Always between CPI and SPI Example Data BCWSc = $10 million BCWPc = $8 million ACWPc = $12 million BAC = $20 million LRE = $25 million

.8 (.67) + .2 (.8) = 0.69

Combinations of CPI and SPI w1(CPI) + w2(SPI)

Sometimes based on Percent Complete: Weight .8

SPI CPI .2

0% 100%

Combinations of CPI and SPI CPI x SPI

• An OSD favorite Example Data BCWSc = $10 million BCWPc = $8 million ACWPc = $12 million BAC = $20 million LRE = $25 million • When CPI <1 and SPI < 1, SCI <<1

0.67 x 0.80 = 0.53

Twelve index-based EAC formulas

Index

CPI SPI w1(CPI) + w2(SPI) CPI x SPI

Monthly

x x x x

Cumulative

x x x x

Average

x x x x

Which one is best?

A-12 CPR Data (April 1990, $MIL) BCWS BCWP ACWP

2080 1491 1950

SV

-589

CV BAC LRE VAC

-459 4046 4400 -354

Index

CPI x SPI SPI .8CPI + .2SPI

CPI

Value

0.5481

0.7168

0.7551

0.7646

EAC

6,612 5,514 5,334 5,292

Which EAC is best?

Estimates at Completion A-12 Program

7 6.5

6 5.5

5 4.5

4 3.5

M ar -89 M ay 89 Ju l-89 Se p 89 N ov 89 Jan -90 M ar -90 SPIxCPI CPI3 CPI6 CPIcum Government Contractor

Evaluating the EAC

1. Compare the CV to the VAC CV -459 VAC -354 Implication LRE too small DOD Experience: Once a contract is more than 15-20% complete, the final overrun will be worse than the present overrun

(Christensen/Wilson 1992)

Evaluating the EAC

2.

Compare the CPI with the TCPI LRE CPI 0.7646

TCPI 1.043

Implication LRE is too small TCPI LRE = (BAC - BCWPcum) / (LRE - ACWPcum) = (4046 - 1491) / (4400 - 1950) DOD Experience:

Once a contract is over 20% compete, the cum CPI does not change by more than 10 percent, and in most cases it only worsens

(Christensen/Heise 1993)

Evaluating the EAC

3.

Compare to a range of independent EACs LRE 4400 IEAC floor 5292 IEAC ceiling Implication 6612 LRE is too low DOD Experience

CPI-based EAC is floor to final cost SCI-based EAC is often the most accurate estimate

(Christensen 1996)

Evaluating the EAC

DOD Experience: No single EAC formula is always best.

(Christensen, Antolini, McKinney 1992)

Navy (Covach, et al., 1981 14 Development, 13 Production)

State of completion Best index-based formula

Early (0-40%) Middle (20-80%) Late (60-100%) CPI3, CPI6, SCIc CPI3, CPI6, CPIc, SCI CPI3, CPI6, CPI12

Army (Howard and Bright, 1981, 11 Develoment)

State of completion Best index-based formula

Early (0-40%) Regression, Composite, SPIc, SCI Middle (31-80%) Late (81-100%) CPI3, CPI6, CPI12, SCI CPIc, SCI

Evaluating the EAC

DOD Experience: No single EAC formula is always best.

(Christensen, Antolini, McKinney 1992)

Air Force (Riedel and Chance, 1989 16 Development 40 Production)

Phase

Devel Prod Devel Prod Devel Prod

System

Aircraft Aircraft Avionics Avionics Engine Engine

25%

SCIc SCIc SCIc 20/80 CPImon PC

50%

CPI3 CPI3 CPI3 SCIc SCIc CPIc

75%

CPI3 SCIc SCIc 20/80 CPI3 SCIc

100%

20/80 CPIc CPIc SCIc CPI3 PC

Overall

SCIc SCIc CPI3 20/80 CPI3 CPIc

Organizational Culture and the EAC

• Accuracy controlled by EVMS Criteria • Factors to consider – Actual Costs to date – Performance to date – Cost and schedule variances – Reliability and relevance of data Overhead and Inflation rates Future performance on work Changes to requirements

Organizational culture

Organizational Culture and the most likely EAC

• Program managers do not support EACs most likely to be experienced on their programs.

• Instead they support EACs most likely to be tolerated by OUSD and Congress.

Estimates at Completion A-12 Program

7 6.5

6 5.5

5 4.5

4 3.5

M ar -89 M ay 89 Ju l-89 Se p 89 N ov 89 Jan -90 M ar -90 SPIxCPI CPI3 CPI6 CPIcum Government Contractor

Why were the contractor and government EACs the smallest?

EAC Comparisons (64 contracts) 0.05

0 -0.05

-0.1

-0.15

-0.2

-0.25

10 20 30 40 50 60 70 Percent Complete 80 90 100

How did the government and contractor estimates compare to this range?

(Christensen, 1996)

EACsci EACcpi

EAC Comparisons (64 contracts) 0.05

0 -0.05

-0.1

-0.15

-0.2

-0.25

10 20 30 40 50 60 70 80 90 100 Percent Complete

(Christensen, 1996)

EACsci EACgov EACcon EACcpi

Overrun Optimism (64 Contracts) 22 20 18 16 14 12 10 8 6 4 2 0 -2 10 Current Overrun Est Final Overrun(G) Est Final Overrun(K) 20 30 40 50 60 70 80 Percent Complete (Christensen, 1994) 90 100

EAC Evaluation Methods: Do They Work On Post-A12 Contracts?

David S. Christensen, Carl Templin, and David Rees Southern Utah University (435)865-8058 [email protected]

2002 National Conference of the Society of Cost Estimating and Analysis Scottsdale, Arizona (June 11-14, 2002)

References

• Christensen, David S. and Carl Templin. “EAC Evaluation Methods: Do They Still Work?” Acquisition Review Quarterly (forthcoming) • Christensen, David S., and David A. Rees. “Is The CPI-based EAC a Lower Bound to the Final Cost of Post-A12 Contracts?” Journal of Cost Analysis and Management (forthcoming) • Earned value literature bibliography: http://www.suu.edu/faculty/christensend/ev-bib.html

Background

• A-12 cancellation – a catalyst for change – The A-12 EAC was too low – Realistic EACs were suppressed – DOD-wide epidemic (Beach 1992) • DoD policy changes to achieve cost realism – Have not worked previously (Drezner et al 1993) – Will they work this time?

Background

• Current reform initiatives are improving cost performance (Coopers and Lybrand 1997) • Methods to evaluate EACs which assume deteriorating cost performance may no longer be valid

EAC Evaluation Rules

• 1.

2.

• 3.

The final cost variance (in dollars or as a percentage) will be worse than the cost variance at the 20 percent completion point The cumulative cost performance index (CPI) will not change by more than 0.10 from its value at the 20 percent completion point, and in most cases only worsens The EAC computed using the cumulative CPI is a reasonable lower bound to final cost Based on observations made by OUSD; validated by researchers at AFIT on contracts completed in 1970s and 1980s; cited by Beach A12 Example – CV 20 – CPI 20 – = -$459M Estimated Final Cost Variance = -$354M = 0.765 To Complete Performance Index = 1.043

CPI-based EAC = $5,292M Contractor EAC = $4,400M

Hypotheses

• The final cost variance will be worse than the cost variance at the 20% completion point CV$ final CV% final – CV$ 20 < 0 – CV% 20 < 0 CV$ = Earned Value – Actual Cost CV% = (CV$ / Earned Value) x 100

Hypotheses

• The cumulative CPI will not change by more than 0.10 from its value at the 20% completion point, and in most cases only worsens |CPI final CPI final – CPI 20 | < 0.10

– CPI 20 < 0 CPI = Cumulative Earned Value / Cumulative Actual Cost

Hypotheses

• The EAC computed using the cumulative CPI is a reasonable lower bound to final cost CPI-based EAC < Final Cost • To control for differences in contract size, DAC < 0 EAC = (BAC – Earned Value)/ Cum CPI + Actual Cost Deviation at completion (DAC) = (EAC – Final Cost) / Final Cost

More Definitions

• CV 20 • CV final = Cumulative CV at 20 percent completion point = Cumulative CV at end of contract • CPI 20 • CPI final = CPI at end of contract = CPI at end of contract • Percent Complete = Cumulative Earned Value / BAC

Hypothesis testing

• Averages of the differences were computed and hypotheses were tested at alpha of .05 using paired-t test • Sensitivity of results were tested across – Contract phase (development versus production) – Contract type (fixed price versus cost reimbursable – Managing service (Army, Air Force, Navy)

The data

• DAES database – Based mostly on CPR data – Its reliability is controlled via EVMS Criteria • Every completed contract was included with actual cost, earned value, and BAC at 20% complete and at completion, resulting in – 147 pre-A12 contracts (finished before A12 cancellation) – 52 post A12 contracts (started after A12 cancellation) – 42 “transition period” contracts (started before and finished after A12 cancellation) • 31 Dec 91, about one year after the A-12 cancellation, was chosen as the cut-off date

Results

• Cost variances did not worsen significantly on Post-A12 contracts from the 20% completion point • The cumulative CPI is a reasonable floor to final cost in the early and middle stages of contract life on Post-A12 contacts • Cost performance during the Transition period contracts was worse than cost performance during Pre-A12 and Post-A12 periods • Results generally insensitive to contract size, contract phase, and military service

The Final Cost Variance Will Be Worse Than The Cost Variance At The 20 Percent Completion Point

0.00

-50.00

-100.00

-150.00

Cost Variance ($MIL)

CV$20 CV$final Pre-A12 Transition Post-A12 n CV$20 CV$final Difference Significance Pre-A12 Transition 147 -0.80

-12.2

-11.4

0.000

41 -3.40

-120.8

-117.4

0.000

Post-A12 52 -3.00

-15.8

-12.8

0.008

The mean CV$ worsened significantly for each period of contracts (but CV$ is not adjusted for contract size and inflation) 0.00

-5.00

-10.00

-15.00

Cost Varince (Percentage)

CV%20 CV%final Pre-A12 Transition Post-A12 n CV%20 CV%final Difference Significance Pre-A12 Transition 147 -1.70

-6.5

-4.8

0.000

41 -2.00

-13.9

-11.9

0.000

Post-A12 52 -2.60

-4.4

-1.7

0.286

The mean CV% worsened significantly for Pre-A12 and Transition period contracts but not for Post-A12 contracts

The Final Cumulative CPI Will Not Change By More Than 0.10 From Its Value At The 20% Completion Point, And In Most Cases It Only Worsens Mean Absolute Deviation (MAD) of the Cumulative CPI

0.150

0.100

0.050

0.000

Pre-A12 Transition Post-A12 n MAD Significance Pre-A12 Transition 147 41 0.085

0.000

0.117

0.487

Post-A12 52 0.067

0.000

The "10% rule" is valid on Pre-A12 and Post-A12 contracts, but not on Transition period contracts

Cumulative Cost Performance Index

CPI20 CPIfinal 1.050

1.000

0.950

0.900

0.850

0.800

Pre-A12 Transition Post-A12 n CPI20 CPIfinal Difference Significance Pre-A12 Transition 147 0.996

0.951

-0.045

0.000

41 0.990

0.896

-0.094

0.000

Post-A12 52 0.977

0.964

-0.013

0.286

The mean cumCPI worsened for Pre-A12 and Transition period contracts, but not for Post-A12 contracts

The Cumulative CPI-based EAC Is A Floor To Final Cost

0.0% -5.0% -10.0% -15.0% -20.0% -25.0% -30.0%

Deviation at Completion (147 pre-A12 contracts)

Early Middle

Contract Stage

Late Pre-A12 Significance Early -10.3% 0.000

Middle -0.2% 0.475

Late -0.2% 0.4535

Deviation at Completion (41 Transition Contracts)

0.0% -5.0% -10.0% -15.0% -20.0% -25.0% -30.0% Transition Significance Early Middle

Contract Stage

Late Early -24.3% 0.000

Middle -11.2% 0.000

Late -5.7% 0.005

Deviation at Completion (52 Post-A12 Contracts)

0.0% -5.0% -10.0% -15.0% -20.0% -25.0% -30.0% Post-A12 Significance Early Middle

Contract Stage

Late Early -10.7% 0.001

Middle -4.4% 0.020

Late -1.9% 0.215

Conclusion

• Cost performance on post-A12 contracts is improving • The A-12 cancellation changed the acquisition culture • The EAC evaluations should continue to be used • Acquisition reform initiatives can adversely affect cost performance of on-going contracts