Transcript Technical Reviews - Space Systems Engineering

Technical Reviews Module
Space Systems Engineering, version 1.0
Space Systems Engineering: Technical Reviews Module
Module Purpose: Technical Reviews
 To understand the purpose and value of conducting
technical reviews.
 To discuss the timing of technical reviews over the
course of a project’s life cycle.
 To consider the entrance criteria and success criteria
for the standard project technical reviews.
 To understand when a technical review is complete.
Space Systems Engineering: Technical Reviews Module
2
Definition of Technical Reviews
 Technical reviews are scheduled within a project or program to
communicate an approach, demonstrate an ability to meet
requirements, or establish status.
 Technical reviews often serve as control gates for management
to make a go/no-go decision on a project.
• Control gates involve formal examination of a project’s status in
order to obtain approval to proceed.
 Technical reviews may be held for subsystems too; depending
on their size and complexity. Subsystem reviews precede their
corresponding system review.
 When do technical reviews occur?
• Throughout the entire life cycle of a project, as shown on the
following life cycle chart.
Space Systems Engineering: Technical Reviews Module
3
Reviews in a NASA Project’s Life Cycle
Space Systems Engineering: Technical Reviews Module
4
The Value of Technical Reviews
 Technical reviews are key development milestones used to
measure progress, assess project maturity and to infuse lessons
from the past. They…
• Provide confirmation of completed effort and readiness to commit
additional resources for the next phase.
• Encourage and establish project discipline with an integrated project
team perspective.
• Identify risks and review mitigation options.
• Describe plans and priorities for the next phase.
 Technical reviews give everyone (design team, non-advocate
discipline experts, customer and consumer) the opportunity to
agree on the current project baseline (requirements, interfaces,
allocations, margins, schedules, risks, budgets, etc.).
• Functional, resource and performance allocations and assumptions
are described and confirmed.
• System development constraints and interfaces are described and
agreed to by both sides.
• Risks and development problems are identified and mitigation options
discussed.
Space Systems Engineering: Technical Reviews Module
5
More Values of Technical Reviews
 A significant, and perhaps unexpected, value to the development
team is the preparation for a review. Usually everyone is busy in
their own domain - preparing for a review forces the team to
describe what they have done and understand what others are
doing.
While there is great value in the preparation, there is also unique value
in the execution. One Air Force officer once took Dwight D.
Eisenhower’s famous quote:
“Plans are nothing. Planning is everything.”
too seriously and on the day before it was scheduled cancelled the PDR
of a $100 million space mission. After a minor revolt, the PDR was
reinstated and the team benefited from the preparation and the review.
 Periodic project reviews are held to demonstrate that the appropriate
products, accomplishments and plans have been completed before
proceeding to the next phase.
Appropriate products, accomplishments and plans are based on the
lessons of hundreds of past projects (best practices). NASA, for
example, describes standard entrance and success criteria for all
standard milestone reviews.
Space Systems Engineering: Technical Reviews Module
6
NASA’s Minimum Set of Technical Reviews
 Mission Concept Review (MCR),
 System Requirements Review (SRR) and/or Mission Definition
Review (MDR),
 Preliminary Design Review (PDR),
 Critical Design Review (CDR),
 System Integration Review (SIR),
 Test Readiness Review (TRR),
 Operational Readiness Review (ORR),
 Flight Readiness Review (FRR),
 Post-Launch Assessment Review (PLAR),
 Critical Event Readiness Review (CERR), and
 Decommissioning Review (DR).
While this may seem like a large number of technical reviews, each
has its own focus and proven value.
In addition, since mission lifetimes are several years, typically
there is 6 months or more between major reviews.
Space Systems Engineering: Technical Reviews Module
7
The Primary Questions of Each Review
 Mission Concept Review - Does the proposed concept meet the
mission need and objectives?
 System Requirements Review and/or Mission Definition Review Do the functional and performance requirements and the selected
concept satisfy the mission?
 Preliminary Design Review - Does the preliminary design meet all the
system requirements within acceptable cost, schedule, and risk?
 Critical Design Review - Is the system design mature enough to
proceed with full-scale fabrication, assembly, integration and test?
 System Integration Review - Are the system, facilities, personnel,
plans and procedures ready for system integration?
 Test Readiness Review - Is the project ready to commence with
verification testing?
 Operational Readiness Review - Are all systems hardware, software,
personnel, and procedures in place to support operations?
 Flight Readiness Review - Is the system ready for launch? Are the
ground facilities and personnel ready to support launch?
 Post-Launch Assessment Review - After launch and deployment, are
the spacecraft systems ready to proceed with routine operations?
 Critical Event Readiness Review - Is the project ready to execute the
mission’s critical activities during flight operation?
 Decommissioning Review - Is the system ready to be terminated?
Space Systems Engineering: Technical Reviews Module
8
Pre-Phase A Reviews
Mission Concept Review (MCR)
 A validation that the mission has clearly established needs, objectives,
and top-level functional/performance requirements, and that at least
one way of conducting the proposed mission is realistic and attainable
within existing or projected technology and ROM cost.
 An internal Center review(s) of all Pre-Phase A activities and products
should be conducted prior to forwarding the Pre-Phase A report to
Headquarters. Technical, management, resources, and scientific
personnel should conduct the review.
Peer Reviews
 Used informally during Pre-Phase A and Phase A.
 The peer group is composed of individuals selected from outside the
project according to their expertise in the applicable disciplines.
 Throughout Pre-Phase A and Phase A, peer reviews should informally
check the evolving mission concept against objectives, requirements,
and constraints.
 Peer reviews help you take advantage of other engineering experience
from colleagues who have worked on different missions. They can point
out issues they confronted that may be similar for your mission
concept.
Space Systems Engineering: Technical Reviews Module
9
Phase A Reviews
System Requirements Review (SRR)
 The primary focus of the SRR is to verify the realism of the functional
and performance requirements, ensure their congruence with the
mission and system configuration, and ensure the mission objectives
can be satisfied.
 The SRR encompasses all major participants (NASA and contractors),
and is chaired by the Project Manager.
 A product from the SRR is the project system specification that is
formally baselined and placed under configuration management
control.
Mission Definition Review (MDR)
 A validation that the mission objectives can be satisfied, the partitioning
of the functionality to each of the systems is adequate, the top-level
performance requirements for each system have been defined, and the
technology required to develop the systems and implement the mission
is attainable.
 The MDR is keyed to the end of Phase A and evaluates the mission
definition, system design, operational concepts, schedule, and cost
estimates.
Space Systems Engineering: Technical Reviews Module
10
Example Entrance Criteria —
System Requirements Review
1. Successful completion of the Mission Concept Review (MCR) and responses made to all
MCR Requests for Actions (RFAs).
2. A preliminary SRR and/or MDR agenda, success criteria, and charge to the board have
been agreed to by the technical team, project manager, and review chair prior to the SRR
and/or MDR.
3. The following technical products for hardware and software system elements are available
to the cognizant participants prior to the review:
a. System Architecture.
b. System requirements document.
c. System software functionality description.
d. Updated concept of operations.
e. Updated mission requirements, if applicable.
f. Baselined SEMP.
g. Preliminary system requirements allocation to the next lower level system.
h. Updated cost estimate.
i. Technology Development Maturity Assessment Plan.
j. Preferred system solution definition including major trades and options.
k. Updated risk assessment and mitigations.
l. Updated cost and schedule data.
m. Logistics documentation (preliminary maintenance plan, etc.).
n. Preliminary human rating plan, if applicable.
o. Software Development Plan (SDP).
p. System safety and mission assurance plan.
q. Configuration management plan.
r. Project management plan.
s. Initial document tree.
t. Verification and validation approach.
u. Preliminary hazard analysis (PHA).
Space Systems Engineering: Technical Reviews Module
11
Example Success Criteria —
System Requirements Review
1. The resulting overall concept is reasonable, feasible, complete, responsive
to the mission requirements, and is consistent with system requirements
and available resources (cost, schedule, mass power, etc.).
2. The project utilizes a sound process for the allocation and control of
requirements throughout all levels, and a plan has been defined to
complete the definition activity within schedule constraints.
3. Requirements definition is complete with respect to top level mission and
science requirements, and interfaces with external entities and between
major internal elements have been defined.
4. Requirements allocation and flow down of key driving requirements have
been defined down to subsystems.
5. System and subsystem design approaches and operational concepts exist
and are consistent with the requirements set.
6. The requirements, design approaches, and conceptual design will fulfill the
mission needs within the estimated costs.
7. Preliminary approaches have been determined for how requirements will
be verified and validated down to the subsystem level
8. Major risks have been identified, and viable mitigation strategies have
been defined.
Space Systems Engineering: Technical Reviews Module
12
Constellation Program Office Schedule of SRRs
You are2007
Here! April
March
Issue ID, Form Entry
5/23
Provide Issue list w/
3/26
Rev. Issue Table, POC, & Sched.
PBS
May
5/10 Resolution Plan &
Impacts
3/29
Provide Status on Issue Resolution
(every Mon. @ Cx SE&I 8:30 telecon)
4/6
4/9
4/16
4/23
CLV SRR
Develop PBS
Presentation
4/30
5/14
(send updates to Cx SE&I COB every other day in May)
Issue ID, Form Entry
Provide Issue list w/
3/26
5/10 Resolution Plan &
Rev. Issue Table, POC, & Sched.
Impacts
4/6
Provide Status on Issue Resolution
(every Mon. @ Cx SE&I 8:30 telecon)
4/9
4/16
4/23
CEV SRR
Develop PBS
Presentation
4/30
5/14
(send updates to Cx SE&I COB every other day in May)
Issue ID, Form Entry
Rev. Issue Table, POC, & Sched.
Provide Status on Issue Resolution
(every Mon. @ Cx SE&I 8:30 telecon)
MO SRR
Provide Issue list w/
4/9
5/10 Resolution Plan &
Impacts
4/13
4/16
4/23
Develop PBS
Presentation
4/30
5/14
(send updates to Cx SE&I COB every other day in May)
Provide Issue list w/
5/10 Resolution Plan &
Issue ID, Form Entry
Rev. Issue Table, POC, & Sched.
GO SRR
Impacts
5/4
5/5
Develop PBS
5/14 Presentation
Provide Status on Issue Resolution
(send updates to Cx SE&I COB every other day in May)
Provide Issue list w/
5/10 Resolution Plan &
Issue ID, Form Entry
Rev. Issue Table, POC, & Sched.
EVA SRR
Provide Status on Issue Resolution
(send updates to Cx SE&I COB every other day in May)
Space Systems Engineering: Technical Reviews Module
Impacts
5/8
5/9
Develop PBS
5/14 Presentation
13
Constellation Program Office Status - Closure of SRR
RIDs
Board AI
 Total AI Count: 48 Total
 9 Closed
 39 Open
• 30 are “Past Due”
• 9 are in work, “Not Due Yet”
 Total RID Count: 6,283
 6225 Closed
 58 Open
• Majority closed prior to PBS
• All RIDs should be closed prior to CxP SDR
TBD/TBR
 Total Count: 2,532
• Received plans for 2,064 TBRs/TBDs in
72 of 95 documents (76% of docs.)
• Need burn-down plans for 196
TBRs/TBDs in 16 documents (17% of
docs.)
• 7 documents (274 TBRs/TBDs) will not
be updated/baselined until post CxP
PBS. These TBRs/TBDs will not be
closed prior to CxP PBS
Space Systems Engineering: Technical Reviews Module
Document
Document Title
CxP 70008
CxP 70023
CxP 70024
CxP 70036
CxP 70050 Vol 2
CxP 70061
CxP 70067 Vol 1
CxP 70080
CxP 70086
CxP 70135
Total
Master Integration and Verification Plan (MIVP)
CxP Prg Design Spec Natural Environments (DSNE)
CxP HSIR
Cx Environ Qual & Accept Testing Reqs (CEQATR)
Electrical Power System Spec Vol 2
C3I Strategic Plan
CxP Program Human-rating Plan Vol 1
Electromagnetic Environmental Effects (E3) Req'ts
Software Verification and Validiation Plan
Structural Design & Verification Requirements
Being
Worked
4
9
1
5
1
1
19
6
5
7
58
14
Preliminary Design Review (PDR)
 The PDR is not a single review but a number of reviews starting
with the specific component PDRs, followed by the system-level
review.
 The PDR establishes the “design-to” baseline and ensures that
it meets the program, project , system, subsystem, or specific
component baseline requirements.
 The PDR process should:
• Establish the ability of the selected design approach to meet the
technical requirements (i.e., Verifiability/ Traceability);
• Establish the compatibility of the interface relationships of the
specific end item with other interfacing items;
• Establish producibility of the selected design;
• Establish the operability of the selected design;
• Assess compliance with reliability and system safety requirements;
• Establish the feasibility of the approach;
• Address status, schedule and cost relationships.
Space Systems Engineering: Technical Reviews Module
15
When is a Review Complete?
Reviews are considered complete when the following is accomplished:
a) Agreement exists for the disposition of all Review Item Discrepancies
(RID) and Request for Actions (RFA).
b) The review board report and minutes are complete and distributed.
c) Agreement exists on a plan to address the issues and concerns in the
review board’s report.
d) Agreement exists on a plan for addressing the actions identified out of
the review.
e) Liens against the review results are closed, or an adequate and timely
plan exists for their closure.
f)
Differences of opinion between the project under review and the review
board(s) have been resolved, or a timely plan exists to resolve the
issues.
g) A report is given by the review board chairperson to the appropriate
management and governing program management committees
charged with oversight of the project.
h) Appropriate procedures and controls are instituted to ensure that all
actions from reviews are followed and verified through implementation
to closure.
Space Systems Engineering: Technical Reviews Module
16
Example Agenda for a Project System Design
Review by a Standing Review Board (SRB)
 Purpose of Review & Charge to SRB
 Project Overview & Status
 System Engineering & Status
• Requirements & V&V plans
• Trade studies
• Technical margins
 WBS-level 2 Design State & Status for each area
• System Design
• Key Requirements
• Trade Studies
• Technology Readiness
• Acquisition Strategy & Long Lead
• Logistics & Facilities
• Challenges & Risks
 Integrated System (e.g., power) State & Status for each area
 Integration & Test
 Safety & Mission Assurance (S&MA)
 Human Rating
 Risk
 Schedule
 Cost
Space Systems Engineering: Technical Reviews Module
SRB Chair
Project Manager
Project SE
WBS Managers
Discipline Leads
Integration Manager
S&MA Manager
Project HR Rep
Risk Manager
Project Planner
Cost Manager
17
Pause and Learn Opportunity
Student role-play:
You are the chief systems engineer for a New Frontiers-class
mission to Europa. Your PDR is scheduled in 3 months.
 What do you do?
 What do you ask the development team to do?
 What benefits would you expect from having a PDR?
 How might it waste time?
Or
Read and discuss the Crosslink article: The Role of
Independent Assessments for Mission Readiness
(Crosslink_Independent Review.pdf)
Space Systems Engineering: Technical Reviews Module
Module Summary: Technical Reviews
 Technical reviews are key development milestones used to
measure progress, assess project maturity and to infuse lessons
from the past. They…
• Provide confirmation of completed effort and readiness to commit
additional resources for the next phase.
• Encourage and establish project discipline with an integrated
project team perspective.
• Identify risks and review mitigation options.
• Describe plans and priorities for the next phase.
 There are 11 reviews in the minimum set of technical reviews for
a NASA robotic mission. These reviews cover the entire mission
life — assessing the concepts and designs early; readiness for
test, flight and operations in mid-life and plans for disposal at the
mission’s end.
 These reviews are held to demonstrate that the appropriate
products, accomplishments and plans have been completed
before proceeding to the next phase.
• Appropriate products, accomplishments and plans are based on the
lessons of hundreds of past projects (best practices).
Space Systems Engineering: Technical Reviews Module
19
Backup Slides
for Technical Review Module
Space Systems Engineering: Technical Reviews Module
Major Project Reviews Precede
Each Key Decision Point
FORMULATION
A
Pre-A
Project
Phases
Key
Decision
Points
Concept
Studies
IMPLEMENTATION
B
Concept &
Technology
D
Preliminary Final
Design &
Design &
Technology Fabrication
Completion
Development
C
B
A
C
E
F
System
Operations & Closeout
Assembly,
Sustainment
Test, &
Launch
D
E
F
Mission Concept Review
Systems Requirements Review
Major
Reviews
Mission/System Definition Review
Preliminary Design Review
Critical Design Review
Independent Cost
Estimates
Systems Integration Review
Operational Readiness Review
Flight Readiness Review
Post Launch Assessment Review
Decommissioning
Review
Space Systems Engineering: Technical Reviews Module
21
Purpose of Technical Reviews
A technical review is an evaluation of the project, or element
thereof, by a knowledgeable group for the purposes of:
a) Assessing the status of and progress toward accomplishing the
planned activities.
b) Validating the technical tradeoffs explored and design solutions
proposed.
c) Identifying technical weaknesses or marginal design and
potential problems (risks) and recommending improvements
and corrective actions.
d) Making judgments on the activities’ readiness for the follow-on
events, including additional future evaluation milestones to
improve the likelihood of a successful outcome.
e) Making assessments and recommendations to the project team,
Center, and Agency management.
f) Providing a historical record that can be referenced of decisions
that were made during these formal reviews.
g) Assessing the technical risk status and current risk profile.
Space Systems Engineering: Technical Reviews Module
22
Need for Management Reviews
The progress between life-cycle phases is marked by key decision
points (KDPs). At each KDP, management examines the
maturity of the technical aspects of the project. For example,
management examines whether the resources (staffing and
funding) are sufficient for the planned technical effort, whether
the technical maturity has evolved, what the technical and nontechnical internal issues and risks are, or whether the
stakeholder expectations have changed. If the technical and
management aspects of the project are satisfactory, including
the implementation of corrective actions, then the project can be
approved to proceed to the next phase.
Space Systems Engineering: Technical Reviews Module
23
SE&I/ T&V Change Requests submitted to
Configuration Management Dates 4/3/07
4/24
3/1
CEV SRR Board
3/20
70135
3/20
70025 FAD
N/A
3/27
MS SRR Board
4/12
70065,
70070-ANX05 Bk 1
4/10
70087
3/23
70012
2/15
70073 Vol 3
5/23
Program Baseline Sync
4/25
GO SRR Board
4/11
70022 Vol 3
6 weeks
5/1
Prior to PBS
EVA SRR Board
4/13
70013
4/16
70136
4/23
70070-ANX05 Bk3
5/1
70019
4/24
Cx Nomen. Plan
70133
TBD
4/30
70022 Vol 1
4/16
70085
5/7
70022 Vol 8
5/8
70130
5/9
70000
5/25
70008
4/20
70138
4/2
70044
Elapsed Time
2/4
2/2
2/11
2/18
3/23
3/29
70009 70151
4/2
4/9
Anx 1
70022 Vol 5 70023
2/25
3/4
3/11
3/18
3/25
4/18
4/11
4/9
70016
Afford.
Plan,
70064
70070-ANX05 Bk 2
4/18
70086
4/19
70137
70000 Constellation Architecure Requirements Document, Revision A
70007 DRM & Operations Concept Document
70008 Master Integration & Verification Plan
70009 System Integrated Analysis Plan
70009 Anx 1 System Integrated Analysis Plan, Annex 1 for IDAC-3
Constellation Reference Architecture Document for IDAC3
Systems Engineering Management Plan
Requirements Engineering Management Plan
C3I Interoperability Standards Book
CSI Spectrum and Channel Plan
CSI Master Link Book
CSI Information Representation Spec
CSI Data Exchange Protocol Specification
Common Command & Control Requirements
Constellation Program Design Specification for Natural
70023 Environments (DSNE)
70012
70013
70016
70022 Vol 1
70022 Vol 2
70022 Vol 3
70022 Vol 4
70022 Vol 5
70022 Vol 8
70024 Constellation Human Systems Integration Requirements
70025 Functional Analysis Document
Constellation Program Environmental Qualifications and
70036 Acceptance Testing Requirements (CEQATR).
Constellation Program Natural Environment Definition for
70044 Design (NEDD)
Electrical Power System Specification, Volume 1: Electrical
70050 Vol 1 Power Quality Performance for 28 VDC
Electrical Power System Specification, Volume 2: User
70050 Vol 2 Electrical Power Quality Performance for 28 VDC
70061 C3I Strategic Plan
4/8
4/15
4/24
4/24
70036 70080 4/24
70141
Submit CR
to CM
Title
CxP Doc #
4/1
4/22
4/24
70145
4/29
5/6
5/13
5/20
5/27
6/3
4/25
70061
4/24
4/24 70144
70132
5/4
5/9
4/30
5/7
70024
70007
70009 5/3
70022 Vol 4 TBD
4/26
70078
70022 Vol 2
Comments
Org
Doc Owner
PRIMO
C. Adamek
5/9/2007 Submit date reflected in the IMS inputs for 3/30.
5/25/2007 Date changed in 3/8 verison of IMS - changed color to yellow
4/30/2007 Added. Per IMS Update of 3/14/07
PRIMO
T&V
ATA
B. Teague
R. Cox
A. Zuniga
3/23/2007 Submit date reflected in the IMS inputs for 3/30.
ATA
A. Zuniga
ATA
COS-T
PRIMO
CSI
CSI
CSI
CSI
CSI
CSI
D. McKissock
J. Afarin
M. DiGiuseppe
K. Muery
C. Sham
J. Brase
J. Differding
M. Stagnaro
M. Severence
5/9/2007 Per update from PRIMO 2/26/07
3/23/2007
4/13/2007
4/18/2007
4/30/2007
4/26/2007
4/25/2007
5/7/2007
4/2/2007
5/7/2007
Updated per e-mail on 3/27. Expected to be reflected in the IMS
inputs for 4/6.
Confirmed by James A. on 2/26 and Dave C. on 3/8.
Updated per IMS updates for 3/23
Updated per IMS updates for 3/23
Updated with CSI input rec'd 3/19
Updated with CSI input rec'd 3/19
Updated per IMS updates for 3/23
Updated with CSI input rec'd 3/19
Updated with CSI input rec'd 3/19
4/9/2007 Updated per 3/19 IMS inputs
E&C
IMS input on 3/9 shows CR drop 5/4 (but this is prior to all updates in).
5/4/2007 Will confirm before updating here.
HFSIG
FAD through RWG only, no board baseline required. B/L date in
N/A schedule of 6/27 is currently out of sync with PBS need dates
PRIMO
6/17
5/15
5/16
70143 70050 Vol 1,
70050 Vol 2
Change Flag
K. Hwang
K. Ess
J. Williams-Byrd
4/24/2007 Update from Ed Strong, 2/21
T&V
E. Strong
4/2/2007 Updated per 3/19 IMS inputs which confirmed date
Schedule update received 3/9. Doc will be submitted but not
5/16/2007 baselined in time for PBS
Schedule update received 3/9. Doc will be submitted but not
5/16/2007 baselined in time for PBS
E&C
L. Smith
Power
R. Scheidegger
Power
R. Scheidegger
CSI
P. Paulsen
SOA
K. Watson
Updated per e-mail on 3/27. This will need to be updated in the IMS
4/25/2007 inputs for 3/30. (was 3/30)
6/10

70064 Supportability Plan
Blue text indicates changes
from last update
Space Systems Engineering: Technical Reviews Module
4/9/2007 Updated per IMS inputs from 3/30 (was 3/23/07)
Slipped
On plan and will be baselined prior to PBS
Plan confirmed but cannot be baselined prior to PBS (CR
submitted within 2 weeks of PBS) or requires IMS update.
No plan confirmed
Submitted to CM
24