Transcript Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs RADM Rand Fisher, Chair Dr.

Optimizing U.S. Air Force and
Department of Defense Review of
Air Force Acquisition Programs
RADM Rand Fisher, Chair
Dr. Dan Stewart, Vice Chair
AIR FORCE STUDIES BOARD
1
Bottom Line
• Unless they do it better than they are now, Air Force
and DOD attempts to address poor acquisition
program performance with additional reviews will fail
• The Air Force and DOD need to
– Engage in timely planning for reviews
– Align reviews with program decision points and milestones
– Before creating new reviews, determine whether existing
reviews could accomplish objectives
– Staff review teams with subject matter experts
– Document all review outputs
AIR FORCE STUDIES BOARD
2
Background
• DOD spends over $300 billion per year to develop,
produce, field, and sustain weapons systems
– Air Force over $100 billion per year
• Large cost overruns and schedule delays lead to
loss of confidence in acquisition system and people
“DOD is not receiving expected returns on its large investment in
weapon systems. Our analysis does not show any improvements in
acquisition outcomes as programs continue to experience increased
costs and delays in delivering capabilities to the warfighter. In fact,
when compared to the performance of the fiscal year 2000 portfolio of
major defense acquisition programs, cost and schedule performance
for current programs is actually worse. Without improved acquisition
outcomes in the future, achieving DOD’s transformational objectives
in a constrained fiscal environment is highly unlikely.” (GAO, 2008)
AIR FORCE STUDIES BOARD
3
Background
DOD response:
More reviews
AIR FORCE STUDIES BOARD
4
Background
• Significant increase in the number and frequency of
management reviews at the program, service and
OSD levels since 1996
– Separate milestone reviews for evolutionary acquisition
increments
– Air Force reviews such as Sufficiency Reviews and IPAs
– OSD-level reviews such as CSBs and PSRs
– Specialty reviews (logistics, manufacturing, technical
readiness
– Previously discretionary reviews made mandatory
AIR FORCE STUDIES BOARD
5
Background
AIR FORCE STUDIES BOARD
6
Background
• Reviews and prereviews required at multiple levels—
both vertical and horizontal
– Only PM experiences full breadth and depth of review
process—program office must support all
– Overall magnitude of review efforts significantly increases
program office workload and diverts attention from day-today management of program
• No evidence of earlier work focused on impact of
review process on resources spent by the program
office or effect of diverting PM’s attention from day-today management of his or her programs
AIR FORCE STUDIES BOARD
7
Background
• SAF/AQ requested Air Force Studies Board
(NRC) investigate
How can Air Force and DOD review of
Air Force acquisition programs be made
more effective and
its cost and burden on the program
manager lessened?
AIR FORCE STUDIES BOARD
8
Statement of Task
• Examine program reviews and assessments that Air Force
space and non-space system acquisition programs undergo
• Assess resources required to accomplish reviews
• Assess contribution reviews make to successful acquisition
• Identify overlaps
• Evaluate options to increase cost-effectiveness and lessen
workforce impact of reviews
• Recommend changes that Air Force and DOD should make
AIR FORCE STUDIES BOARD
9
Study Committee
•
•
•
•
•
•
•
•
•
RAND H. FISHER, Chair, The Aerospace Corporation
J. DANIEL STEWART, Vice Chair, University of Tennessee
JOHN A. BETTI, Department of Defense (retired)
CHRISTOPHER L. BLAKE, Lockheed Martin Aeronautics
CLAUDE M. BOLTON, JR., Defense Acquisition University
ALLAN V. BURMAN, Jefferson Solutions
JOHN T. DILLARD, U.S. Naval Postgraduate School
CHARLES E. FRANKLIN, Raytheon (retired)
CHARLES L. JOHNSON II, Boeing
AIR FORCE STUDIES BOARD
10
Study Committee
•
•
•
•
•
•
•
LESLIE KENNE, LK Associates
ANDREW P. SAGE, George Mason University
MARK SCHAEFFER, ManTech SRS Technologies
GEORGE R. SCHNEITER, Consultant
ROBERT J. SKALAMERA, RJS Consulting
RICHARD SZAFRANSKI, Toffler Associates
RANDALL S. WEIDENHEIMER, Northrop Grumman
Mission Systems
• REBECCA A.WINSTON, Winston Strategic Management
Consultants
AIR FORCE STUDIES BOARD
11
Approach
Program success orientation + PM perspective =
key question:
Can changes in number, content, or sequence
of reviews help PM execute program more
successfully?
•
•
•
•
Literature review
Presentations to committee and interviews
Comparative matrix
Survey of PMs and PEOs
AIR FORCE STUDIES BOARD
12
Findings
• Reviews are essential elements of program success
– Facilitate program execution, technical and programmatic
support, problem discovery and resolution
– Inform decisions
– Share awareness
– Engender program advocacy
• Reviews are not “free”—there are significant costs
– Money
– Time spent preparing, presenting, follow-up
– Attention diverted from executing program
AIR FORCE STUDIES BOARD
13
Findings
• Many reviews do not contribute to program in
proportion to their costs
– In every case, those interviewed or surveyed cited
significant costs to carry out reviews
• Most also noted adverse impact on other
responsibilities
– Several survey respondents cited reviews that had no
positive impact on program cost, schedule, performance
• Some even cited reviews with negative impact
AIR FORCE STUDIES BOARD
14
Findings
• No one in Air Force or DOD responsible for monitoring
number, workload, costs, effectiveness, or impact of
reviews
– Many survey respondents described DOD staff as a stovepiped bureaucracy
• Domain “czars” have purview over breadth of programs
but not horizontally integrated for knowledge sharing or
synergy
• PMs have to prepare separate information brief for each
• OSD staff does not integrate information across domains
for optimal decision making by MDA
AIR FORCE STUDIES BOARD
15
Findings
• Sequencing, timing, and frequency of reviews often
not tied to program schedule to most effectively
support program execution
– Survey respondents suggested least beneficial reviews
could be more effective if conducted less frequently and at
more appropriate time in program’s life cycle (generally
earlier)
– Speakers gave examples of requirements reviews being
conducted after contracts had been awarded
AIR FORCE STUDIES BOARD
16
Findings
• Reviews often not attended by right personnel
(review principals, key stakeholders, and subject
matter experts) or, in some cases, attended by too
many personnel
– Many reviews conducted without “right” people present
– Survey respondents noted that more effort should be
given to ensuring that right subject matter experts and
appropriate senior officials attend program reviews and
that number of attendees be limited to those who can add
value to meeting
AIR FORCE STUDIES BOARD
17
Findings
• For some reviews, number of preparatory reviews is
excessive and reviews do not contribute value to
program’s management
– Many PMs stated that proliferation of meetings and
premeetings was taking time away from management of
programs
– Elimination of IIPT reviews leading to more individual
premeetings with Joint Staff, program management
offices, and OSD
– Sharing of responsibilities between NII/AT&L offices cited
as another factor
– “The problem isn’t the review. . . it’s the numerous
premeetings.”
AIR FORCE STUDIES BOARD
18
Findings
• PMs spending time on multiple reviews with similar
objectives
– Committee matrix named 31 formal reviews—10 identified
as duplicating or partially duplicating other reviews
– Survey respondents believed that selected reviews could
be combined
AIR FORCE STUDIES BOARD
19
Findings
• Purpose, scope, information needs, key issues, and
expected outcomes of many reviews not specified
– Often ill-defined, based on presumed agendas or issues
of the day
– Often no metrics for assessing effectiveness of review
• Many PMs found that PSRs and IPAs added value
–
–
–
–
–
Comprehensive in nature
Well defined processes, outcomes, and metrics
Socialized with PMs and staff
Conducted by subject matter experts
Well documented
AIR FORCE STUDIES BOARD
20
Findings
• Reviews focus on single system instead of system of
systems of which system is a part. Reviews that
attempt to address larger system-of-systems
perspective often unable to cope with complex
interfaces among programs
– Seventy percent of ACAT I PMs responding to survey
characterized amount of external interface of their programs
with other efforts as extensive
– Survey written-in responses and PM discussions with full
committee noted that some reviews did not take into account
connections with and dependencies on other programs for
mission accomplishment
AIR FORCE STUDIES BOARD
21
Conclusions
• Reducing number of reviews or combining them can increase
time available to PMs to more effectively manage their
programs
• Reviews could be more effective if sequenced and timed to
provide information needed for program execution
• Required attendance at program review meetings is not clearly
communicated nor effectively controlled
• Streamlining or combining reviews and associated prebriefs in
both vertical and horizontal directions could increase efficiency
AIR FORCE STUDIES BOARD
22
Conclusions
• Important that program review planning is accomplished in
thoughtful, purposeful manner with standard approach to
address need for communication of expectations and outcomes
• Review format and design need to reflect greater complexity
and interrelationships in current programs to ensure that
system of systems works across organizational constructs
AIR FORCE STUDIES BOARD
23
Key Recommendation
• Engage in timely planning for reviews
– Governance process directed by SAE
– Owner of review process including reviews, policies, control
of review proliferation, pre- and post review mechanisms
– Deliberate planning and direction to PM and OPR well in
advance
– At minimum, review direction to include objectives with
metrics, materials to be supplied, criteria for success
– Review team report with findings, recommendations, lessons
learned
– PM closeout report with action plan; open, closed, inprocess items; issues or risks
– SAE tracking of review process metrics
AIR FORCE STUDIES BOARD
24
Key Recommendation
• Align reviews with program decision points and
milestones
– Minimize number of reviews preceding decision points and
milestone
– Ensure review content is pertinent
– Alignment may allow reviews to be consolidated
– Could reduce costs and burden on PM and staff
AIR FORCE STUDIES BOARD
25
Key Recommendation
• Before creating new reviews, determine whether
existing reviews could accomplish objectives
– Determine whether broadening stakeholders for a given
review could accomplish objectives rather than adding
new review
– Apply same criteria to all pre-reviews
– Stakeholders should work together to consolidate
prereview process
– Establish guidance for managing prereview so it has
minimal impact schedule, cost, and program
management staff
– Encourage OSD to do the same
AIR FORCE STUDIES BOARD
26
Key Recommendation
• Staff review teams with subject matter experts
– Maintain roster of experts in standard technical areas,
taking into account back-ups, to guarantee expert is
available to attend review
– Prepare process guidance document for selection,
formation, and use of subject matter expert teams
– Consider objectives of review when staffing review team
– Ensure continuity of effort—availability of subject matter
experts not only during review but also for periods before
and after review
AIR FORCE STUDIES BOARD
27
Key Recommendation
• Document all review outputs
– It’s a best practice to capture lessons learned, identify root
causes of problems and risks, and document findings,
observations, and recommendations made during review
• Review team report and PM closeout report
– Create database for storing and sharing lessons learned;
needs to be searchable and updated regularly
• SAE is suggested owner
AIR FORCE STUDIES BOARD
28
Concluding Thoughts
• Together, these recommendations form a gold
standard for conduct of reviews
• If implemented and rigorously managed by SAE,
review effectiveness and efficiency can be
increased
• Recommendations exemplify continual learning
process that builds from one review to next
Bottom line is to help PMs successfully execute their
programs
AIR FORCE STUDIES BOARD
29
Backup Slides
AIR FORCE STUDIES BOARD
31
Statement of Task
The NRC will:
1. Review the prescribed program reviews and assessments that U.S. Air
Force space and non-space system acquisition programs in all
Department of Defense (DOD) acquisition categories (ACATs) are
required to undergo, consistent with the various phases of the
acquisition lifecycle, to verify appropriate planning has occurred prior to
concept decision, Milestone/Key Decision Point (KDP) A,
Milestone/KDP B, and Milestone/KDP C.
2. Assess each review and the resources required to accomplish it,
including funding, manpower (people and knowhow), work effort, and
time.
3. Assess the role and contribution that each review and the combined
reviews make to successful acquisition.
AIR FORCE STUDIES BOARD
32
Statement of Task (cont.)
4. Identify cases where different reviews have shared, common, or
overlapping goals, objectives, content, or requirements.
5. Identify and evaluate options for streamlining, tailoring, integrating, or
consolidating reviews of programs to increase the cost-effectiveness
and to lessen workforce impact of the reviews as a whole, including
examination and discussion of review processes used by other
agencies (such as, the National Aeronautics and Space Administration
and the Department of Energy), the other military departments (the U.S.
Army and the U.S. Navy), and industry.
6. Recommend changes that the Air Force and DOD should make to the
reviews of Air Force programs, including review goals, objectives,
content, and requirements.
AIR FORCE STUDIES BOARD
33