Transcript Standard NOSSA Template - International System Safety Society

Ordnance Safety & Security Activity
Safety in Systems Engineering
Technical Review (SETR) Tutorial
29th International System Safety Conference
Las Vegas, NV
August 9, 2011
Mr. Arch McKinlay
Ms. Peggy Rogers
Mr. Stuart Whitford
Ms. Karen Gill
Ms. Kristin Thompson
Agenda
Ordnance Safety & Security Activity







Introduction
SETR Policy Requirements
What is SETR
Recommended SETRs
Tailoring
Safety in SETR Process
Acquisition Framework Deep Dive






Material Solution Analysis
Technology Development
Engineering and Manufacturing Development
Production and Deployment
Operations and Support
Phase Overview
Present SETRs
Artifacts
Safety Criteria Statements
Safety Driving Factors
Summary/Conclusion
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
2
Introduction
Ordnance Safety & Security Activity

Assistant Secretary of the Navy for Research Development and Acquisition ASN(RDA)
Memo dtd 13 June 2008:


ASN (RDA) Chief Systems Engineer (CHSENG) was to update the Naval Systems
Engineering Technical Review (SETR) Handbook, revision 1






“I want to institutionalize the Systems Engineering Technical Review process within the Department of the Navy
(DON) and ensure appropriate system engineering aspects are included in the Gate review.”
Appendices developed for Common Functional Areas (CFA) – one of which is Safety
Safety Appendix contains Enterprise-level Safety Criteria Checklists (i.e. common to all SYSCOMS)
The safety in SETR goal is to develop a set of Naval Enterprise level safety criteria
statements for each of the SETR events (e.g. PDR, CDR, TRR, etc.).
These criteria statements, or questions, form the basis of safety in SETR for all Navy and
Marine Corps acquisition programs.
Each Systems Command (SYSCOM) may develop additional SYSCOM-specific criteria for
the SETRs.
The safety in SETR effort also focused on better integrating safety engineering into the
overall systems engineering process by developing safety criteria for non-safety focused
documents such as the Systems Engineering Plan and Test and Evaluation Master Plan.
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
3
Ordnance Safety & Security Activity
Naval Systems Engineering Policy
and Guidance

Establishes systems engineering
policy for all Naval SYSCOMs and
affiliated PEOs and Direct
Reporting Program Managers

Establishes a common Systems
Engineering Technical Review
(SETR) process within DON as
promulgated by the Naval SETR
Handbook

Handbook provides guidance to
implement Naval SYSCOM
Systems Engineering Policy

Identifies planning, execution, and
follow-on activities for the SETR
process.
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
4
What is SETR?
Ordnance Safety & Security Activity
 System Engineering Technical Review (e.g. PDR, CDR, TRR, etc)
 Technical reviews are integral to Naval and System Engineering
processes
 Technical assessment of key health and progress of Program
 Provides Program Manager’s (PMs) with independent assessments of
program readiness to enter the next technical phase
 Assists program office management teams in documenting technical
requirements, synthesizing certifiable designs, assessing performance
and system safety risk, and producing and deploying systems to achieve
required capability
 When requested by the PM, chaired by a senior government employee
appointed by the SYSCOM Chief Engineer (CHENG), conducts the SETR
assessments in collaboration with program management
 SETR Lead is an independent Technical Authority from outside the PMO
but usually from inside the SYSCOM
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
5
Recommended SETRs
Ordnance Safety & Security Activity










Initial Technical Review - Supports a program's initial Program Objective Memorandum submission.
Alternative Systems Review – Demonstrates the preferred concept is cost effective, affordable,
operationally effective and suitable, and provides a timely solution to a need at an acceptable level of risk.
System Requirements Review – A system-level review to ensure that the system requirements have
been completely and properly identified and that a mutual understanding between the government and
contractor exists.
System Functional Review – A formal review of the conceptual design of the system to establish its
capability to satisfy requirements. It establishes a functional baseline.
Preliminary Design Review – A formal review that confirms the preliminary design logically follows the
SFR findings and meets the requirements. It normally results in approval to begin detailed design.
Critical Design Review – A formal review conducted to evaluate the completeness of the design and its
interfaces.
Test Readiness Review – A formal review of contractors’ readiness to begin testing on both hardware and
software configuration items.
System Verification Review –Verifies that the actual item (which represents the production configuration)
complies with the performance specification.
Production Readiness Review - Determines if the design is ready for production, production engineering
problems have been resolved, and the producer has accomplished adequate planning for the production
phase.
In-service Review – A formal technical review that is to characterize in-Service technical and operational
health of the deployed system by providing an assessment of risk, readiness, technical status, and trends
in a measurable form that will substantiate in-Service support and budget priorities.
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
6
Acquisition Framework with SETRs
Ordnance Safety & Security Activity
First SETR
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
7
Tailoring SETRs
Ordnance Safety & Security Activity



SETRs should be tailored to reflect technical breadth and depth of the Program
being reviewed
Tailoring needs to be agreed upon by ALL stakeholders
Tailored SETR schedule should be documented in the Systems Engineering
Plan
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
8
Tailoring SETRs – Driving Factors
Ordnance Safety & Security Activity

Systems Engineering driving factors to consider when tailoring





Acquisition Strategy (How quickly does the client need it? Is the system using
existing COTS and facilities?, etc.).
Number of systems being built, where delivered, number of incremental
developments and overlaps.
Size of the development team, their locations (virtual).
Overall complexity of the system and software/hardware/technology.
External Interfaces (How many, complexity, amount/size of data transmitted, how
often. This includes interfaces to organizations and users in addition to all systems
and databases).



All requirements that are related to throughput capacity, processing speed, database accesses/retrieval,
size of files/data being processed, etc.
Key Risks, must be well defined with Risk Mitigation that is realistic.
Understanding who are the stakeholders, and what high-level considerations do they
care about (System performance, schedule, cost, and, safety…and possibly other
matters).
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
9
Ordnance Safety & Security Activity


Tailoring SETRs Merging or Separating
May be appropriate based on system complexity and/or incremental
builds/system development, and will be documented in the tailored SETR
schedule in the Systems Engineering Plan (SEP)
When tailoring the occurrence of SETR events, the level of the reviews should
be addressed and characterized.


There may be multiple incremental reviews for multiple builds, but these reviews
may be at a lower level of detail, not requiring top-level attention.
When SETR events are tailored, engineering judgment should be used to eliminate
criteria reflecting lesser artifact maturity or to reword criteria to reflect correct
artifact maturity for the given SETR.
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
10
Ordnance Safety & Security Activity




Process to Develop Safety Criteria
Statements
The ASN(RDA)/CHSENG lead organized a Safety Horizontal Integration Team
(HIT) to coordinate the development of the Safety SETR Appendix to the Naval
SETR Handbook.
The HIT formed a Safety Working Group (SWG) that included subject matter
experts from different safety disciplines across the Navy SYSCOMS, Office of
the Chief of Naval Operations, and the Navy and Marine Corps Public Health
Center.
The SWG followed a HIT developed process to systematically identify
acquisition-related products and elements and link them to safety-related policy
requirements.
The Safety in SETR workflow was a five step process ending with completion
on Safety SETR Criteria Statements for the Handbook.
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
11
Ordnance Safety & Security Activity
Safety in SETR - Process
Workflow
NAVAIR Safetycentric products /
Artifacts
NAVSEA Safetycentric Products /
Artifacts
Modify
1
Identify Naval
Enterprise
Safety Products /
Artifacts
Identify the
Mandatory
Products / Artifacts
SPAWAR Safetycentric Products /
Artifacts
MARCOR Safetycentric Products /
Artifacts
3
2
Identify
required
elements of
each Safety
Product /
Artifact
Agree to each
Safety
Product /
Artifact
description
Validate: DoD 5000.02, Defense Acquisition
Guidebook, etc.
Other Safetycentric Products /
Artifacts
Legend
Process / action
Product
Existing
documents
Naval
Enterprise
Safety SETR
Appendix
Parking Lot
Safety Products
/ Artifacts /
Criteria not in a
Master List
5
4
Identify SETR
Criteria to evaluate
each product at
corresponding
review
Map and identify
Maturity of Product
against SETR
events (Draft,
Final, Update)
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
12
Criteria Statements You’ll See Today
Ordnance Safety & Security Activity
 DoD and Navy centric references
 References are tied to the criteria statement elements
vice the related artifact
 FOR EXAMPLE:
DoDI 5000.02 requires use of MIL-STD-882D for all
developmental and sustaining engineering activities.
MIL-STD-882D requires that hazards be identified
through a systematic hazard analysis process and use of
historical hazard and mishap data, including from other
systems.
 The typical system safety document for this is the PHL.
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
13
Agenda
Ordnance Safety & Security Activity







Introduction
SETR Policy Requirements
What is SETR
Recommended SETRs
Tailoring
Safety in SETR Process
Acquisition Framework Deep Dive






Material Solution Analysis
Technology Development
Engineering and Manufacturing Development
Production and Deployment
Operations and Support
Phase Overview
Present SETRs
Artifacts
Safety Criteria Statements
Safety Driving Factors
Summary/Conclusion
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
14
Material Solution Analysis
Ordnance Safety & Security Activity
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
15
Ordnance Safety & Security Activity
Material Solution Analysis - Activities
 Purpose: Complete the AoA to assess potential
materiel solutions to capability need, identify key
technologies, and estimate life cycle costs.
 Enter: Approved ICD and study guidance for
conducting an Analysis of Alternatives (AoA).
 Activities: Conduct AoA, develop Technology
Development Strategy (TDS) & draft CDD
 Guided by: ICD and AoA Plan
 Exit: AoA completed, materiel solution options for
the capability need identified in ICD have been
recommended by lead Component conducting AoA,
and phase-specific entrance criteria for the initial
review milestone have been satisfied
 SETR Events: Initial Technical Review and
Alternative Systems Review
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
16
Ordnance Safety & Security Activity
Artifacts for Initial Technical
Review (ITR)
 Initial Technical Review (ITR) - Supports a program's initial Program Objective Memorandum
submission.
– Programmatic ESOH Evaluation (PESHE)
– Preliminary Hazard List (PHL)
– Initial Capabilities Document (ICD)
– AoA Guidance
– Concept of Operations (CONOPS)
– Cost Estimates
– Request for Proposal (RFP)
– Test and Evaluation Strategy (TES)
– Technology Development Strategy (TDS)
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
17
Ordnance Safety & Security Activity
ITR – Criteria Statements
Criteria Statement
1
2
3
4
5
6
7
8
9
Has the program identified Environment, Safety, and Occupational Health (ESOH)
roles and responsibilities and how the program will integrate system safety-ESOH
considerations into the systems engineering process, the ESOH risk management
process, and a method for hazard tracking? (Ships only) (DoDI 5000.02,
NAVSEAINST 5000.8)
Have appropriate potential hazards been derived from historical data lessons
learned from
-Similar legacy systems
-Fielded versions of the same system
-Science and technology programs,
-Independent Research and Development Programs
-Research and Development? (MIL-STD-882, NAVSEAINST 5000.8)
Related Artifact
Programmatic ESOH Evaluation
(PESHE)
Preliminary Hazard List (PHL)
Does the Analysis of Alternatives (AoA) Plan include safety/ESOH considerations?
Has the Concept of Operations (CONOPS) been reviewed for potential operational
safety/ESOH constraints?
Do the cost estimates contain appropriate ESOH/safety-related cost data?
(NAVSEAINST 5000.8)
Has safety/ESOH reviewed the Initial Capabilities Document (ICD) for high level
ESOH-related capability statements?
Does the Request for Proposal (RFP) for alternative solution studies contain ESOH
requirements that the government wants the contractor to address?
(NAVSEAINST 5000.8)
AoA Guidance
Does the Test and Evaluation Strategy (TES) include safety/ESOH planning?
Does the Technology Development Strategy (TDS) include safety/ESOH hazard
analysis planning as part of technology development?
TES
CONOPS
Cost estimates
ICD
RFP
TDS
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
18
Highlighted text will be talked about in detail
Ordnance Safety & Security Activity
ITR – Criteria Statements
Criteria Statement
1
2
3
4
5
6
7
8
9
Has the program identified Environment, Safety, and Occupational Health (ESOH)
roles and responsibilities and how the program will integrate system safety-ESOH
considerations into the systems engineering process, the ESOH risk management
process, and a method for hazard tracking? (Ships only) (DoDI 5000.02,
NAVSEAINST 5000.8)
Have appropriate potential hazards been derived from historical data lessons
learned from
-Similar legacy systems
-Fielded versions of the same system
-Science and technology programs,
-Independent Research and Development Programs
-Research and Development? (MIL-STD-882, NAVSEAINST 5000.8)
Related Artifact
Programmatic ESOH Evaluation
(PESHE)
Preliminary Hazard List (PHL)
Does the Analysis of Alternatives (AoA) Plan include safety/ESOH considerations?
Has the Concept of Operations (CONOPS) been reviewed for potential operational
safety/ESOH constraints?
Do the cost estimates contain appropriate ESOH/safety-related cost data?
(NAVSEAINST 5000.8)
Has safety/ESOH reviewed the Initial Capabilities Document (ICD) for high level
ESOH-related capability statements?
Does the Request for Proposal (RFP) for alternative solution studies contain ESOH
requirements that the government wants the contractor to address?
(NAVSEAINST 5000.8)
AoA Guidance
Does the Test and Evaluation Strategy (TES) include safety/ESOH planning?
Does the Technology Development Strategy (TDS) include safety/ESOH hazard
analysis planning as part of technology development?
TES
CONOPS
Cost estimates
ICD
RFP
TDS
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
19
Ordnance Safety & Security Activity
Artifacts for Alternative System
Review (ASR)
 Alternative System Review (ASR) - Demonstrates the preferred concept is cost effective,
affordable, operationally effective and suitable, and can be developed to provide a timely solution
to a need at an acceptable level of risk.
– Programmatic ESOH Evaluation (PESHE)
– Request for Proposal
– Preliminary Hazard List (PHL)
– Test and Evaluation Strategy
– Critical Safety Items/Applications
– Test and Evaluation Management Plan
– Acquisition Strategy
– Total Ownership Cost
– Capabilities Development Document
– Trade Studies
– Interface Requirement Specification
– Cost Analysis Requirements Document
– Lifecycle Sustainment Plan
– Service Cost Position
– System Performance Specification
– Systems Engineering Management Plan
– Systems Engineering Plan
– Statement of Work
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
20
Ordnance Safety & Security Activity
1
2
3
4
5
6
7
8
9
10
11
ASR – Criteria Statements
Have all preliminary hazards been identified for each alternative solution? (NAVSEAINST
5000.8)
PHL
Does the program have an approved draft Programmatic ESOH Evaluation document that
identifies ESOH responsibilities and how the program will integrate system safety-ESOH
considerations into the systems engineering process, the ESOH risk management process,
the hazard tracking system, and preliminary ESOH hazards and their associated risks?
(Ships only) (DoDI 5000.02)
PESHE
Critical Safety
Has the program identified all Critical Safety Items and safety related Critical Application
Items/Critical
Items? (DFARS 209.270)
Application Items
Does the Acquisition Strategy include a summary of the Programmatic ESOH Evaluation?
(Ships only)
Acquisition Strategy
Has safety/ESOH provided ESOH capability statements for the Capabilities Development
Document (CDD)?
CDD
Does the Draft Cost Analysis Requirements Document (CARD) contain appropriate ESOHrelated cost data?
CARD
Has safety reviewed the Interface Requirement Specification (IRS) to determine if any
safety/ESOH risk exists or needs to be mitigated through the requirements process?
(NAVSEAINST 5000.8)
IRS
Has safety/ESOH provided safety and environmental requirements input to the Life Cycle
Sustainment Plan (LCSP)?
LCSP
Does the RFP, to include prototypes, specify ESOH-related requirements and Contract
Data Requirements List (CDRL)? (NAVSEAINST 5000.8)
RFP
Are the costs of government and contractor system safety/ESOH efforts included in
Personnel and Organization and acquisition costs?
Service Cost Position
Does the System Performance Specification (SPS) contain safety and ESOH
requirements?
SPS
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
21
Ordnance Safety & Security Activity
ASR – Criteria Statement (cont’d)
Does the System Engineering Management Plan (SEMP) explain how safety/ESOH is
12 integrated into the systems engineering process?
13
14
15
16
Does the Systems Engineering Plan (SEP) contain an overview of how safety/ESOH is
addressed and integrated into systems engineering to include specifically Critical
Safety Items?
Does the Statement of Work (SOW) contain safety/ESOH-related requirements and
CDRLs? (NAVSEAINST 5000.8)
Does the TES address government and contractor safety testing?
Does the TES address environmental planning for test events?
SEMP
SEP
SOW
TES
TES
Does the draft Test and Evaluation Master Plan (TEMP) address environmental
17 planning for test events?
18 Does the Total Ownership Cost (TOC) include safety/ESOH related costs?
TEMP
TOC
Do the Trade Studies include recommended mitigation measures/design changes?
19 (NAVSEAINST 5000.8)
Trade Studies
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
22
Highlighted text will be talked about in detail
Ordnance Safety & Security Activity
1
2
3
4
5
6
7
8
9
10
11
ASR – Criteria Statements
Have all preliminary hazards been identified for each alternative solution? (NAVSEAINST
5000.8)
PHL
Does the program have an approved draft Programmatic ESOH Evaluation document that
identifies ESOH responsibilities and how the program will integrate system safety-ESOH
considerations into the systems engineering process, the ESOH risk management process,
the hazard tracking system, and preliminary ESOH hazards and their associated risks?
(Ships only) (DoDI 5000.02)
PESHE
Critical Safety
Has the program identified all Critical Safety Items and safety related Critical Application
Items/Critical
Items? (DFARS 209.270)
Application Items
Does the Acquisition Strategy include a summary of the Programmatic ESOH Evaluation?
(Ships only)
Acquisition Strategy
Has safety/ESOH provided ESOH capability statements for the Capabilities Development
Document (CDD)?
CDD
Does the Draft Cost Analysis Requirements Document (CARD) contain appropriate ESOHrelated cost data?
CARD
Has safety reviewed the Interface Requirement Specification (IRS) to determine if any
safety/ESOH risk exists or needs to be mitigated through the requirements process?
(NAVSEAINST 5000.8)
IRS
Has safety/ESOH provided safety and environmental requirements input to the Life Cycle
Sustainment Plan (LCSP)?
LCSP
Does the RFP, to include prototypes, specify ESOH-related requirements and Contract
Data Requirements List (CDRL)? (NAVSEAINST 5000.8)
RFP
Are the costs of government and contractor system safety/ESOH efforts included in
Personnel and Organization and acquisition costs?
Service Cost Position
Does the System Performance Specification (SPS) contain safety and ESOH
requirements?
SPS
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
23
Ordnance Safety & Security Activity
Questions
 What are some of the obstacles you face when participating in pre-MS A technical
reviews?
 Who is the Safety POC if a Principal for Safety has not been designated?
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
24
BREAK – 10 minutes
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
25
Agenda
Ordnance Safety & Security Activity







Introduction
SETR Policy Requirements
What is SETR
Recommended SETRs
Tailoring
Safety in SETR Process
Acquisition Framework Deep Dive






Material Solution Analysis
Technology Development
Engineering and Manufacturing Development
Production and Deployment
Operations and Support
Phase Overview
Present SETRs
Artifacts
Safety Criteria Statements
Safety Driving Factors
Summary/Conclusion
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
26
Technology Development
Ordnance Safety & Security Activity
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
27
Ordnance Safety & Security Activity
Technology Development Activities
 Purpose: Reduce Technology Risk, determine and
mature appropriate set of technologies to be integrated
into a full system, demonstrate Critical Technology
Elements on Prototypes, and complete the preliminary
design.
 Enter: MDA approved materiel solution and Technology
Development Strategy (TDS); funding for Technology
Development phase activities
 Activities: Competitive prototyping; Develop Reliability
& Maintainability strategy; conduct Preliminary Design
Review (PDR)
 Guided by: Initial Capability Document (ICD) & TDS
and supported by SE planning
 Exit: Affordable increment of military-useful capability
identified; technology demonstrated in relevant
environment; manufacturing risks identified; system or
increment ready for production within short time frame
 SETR Events: Systems Requirements Review, System
Functional Review, Preliminary Design Review
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
28
Ordnance Safety & Security Activity
Artifacts for System Requirements
Review (SRR)
 System Requirements Review (SRR) - A system-level review to ensure that the system
requirements have been completely and properly identified and that a mutual understanding
between the government and potential contractor(s) exists.
– System Safety Management Plan
– System Safety Program Plan
– Software Safety Program Plan
– Hazard Tracking System/Risk Acceptance
– ESOH Risk Assessment Matrix
– Preliminary Hazard Analysis
– Threat Hazard Assessment
– PESHE
– Safety Requirements/Criteria Assessment
– Cost estimates
– Risk Management Plan
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
29
Ordnance Safety & Security Activity
SRR – Criteria Statements
1
Has the government's system safety engineering approach been clearly and fully
documented? (MIL-STD-882, NAVSEAINST 5000.8)
System Safety
Management Plan (SSMP)
2
Has the developer's system safety engineering approach been clearly and fully documented? System Safety Program
(MIL-STD-882, NAVSEAINST 5000.8)
Plan (SSPP)
3
Has the program developed a plan to manage software safety? (MIL-STD-882)
Software Safety Program
Plan (SwSPP)
4
Has a hazard tracking system been developed in accordance with MIL-STD-882? (MIL-STD882, NAVSEAINST 5000.8)
Hazard Tracking System
5
Is the ESOH risk assessment matrix used by the program derived from MIL-STD-882? (MILSTD-882, NAVSEAINST 5000.8)
ESOH Risk Assessment
Matrix
7
Have appropriate potential hazards been derived from the historical data, lessons learned
from similar legacy systems or earlier fielded versions of the same system and have
alternative candidate mitigations been identified and documented? (NAVSEAINST 5000.8)
Has the program identified potential accident and combat threat scenario hazards for the
system and documented in the hazard tracking system? (MIL-STD-882, NAVSEAINST
5000.8)
8
Has the program identified ESOH responsibilities and how the program will integrate system
safety-ESOH considerations into the systems engineering process? (DoDI 5000.02)
PESHE
9
Has the program identified initial safety requirements (prescribed or derived) from applicable
standards, specifications, regulations, design handbooks, safety design checklists, and other
sources? (MIL-STD-882)
Safety
Requirements/Criteria
Assessment (SR/CA)
6
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
Preliminary Hazard
Analysis (PHA)
Threat Hazard Assessment
(THA)
30
Ordnance Safety & Security Activity
10
11
12
13
14
SRR – Criteria Statements (cont’d)
Has the following been updated:
-Hazard Tracking System
-Hazards
-ESOH Risk
-Reports on high and serious ESOH Risk
-Reports on ESOH risk acceptance
(DoDI 5000.02,MIL-STD-882, and NAVSEAINST 5000.8)
Have the cost estimates been updated to reflect any ESOH system requirements related
cost data?
Has the safety/ESOH Program provided input to the Risk Management Plan to ensure
that the ESOH risks are identified and mitigated? (NAVSEAINST 5000.8)
Has an initial assessment of the severity and probability of mishap risk been documented
for each identified hazard in the system safety hazard tracking system? (MIL-STD-882,
NAVSEAINST 5000.8)
Have safety critical functions been identified and entered into the hazard tracking system?
(NAVSEAINST 5000.8)
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
Hazard Tracking
System/Risk Acceptance
Cost Estimates
Risk Management Plan
Hazard Tracking System
PHA
31
Highlighted text will be talked about in detail
Ordnance Safety & Security Activity
SRR – Criteria Statements
1
Has the government's system safety engineering approach been clearly and fully
documented? (MIL-STD-882, NAVSEAINST 5000.8)
System Safety
Management Plan (SSMP)
2
Has the developer's system safety engineering approach been clearly and fully documented? System Safety Program
(MIL-STD-882, NAVSEAINST 5000.8)
Plan (SSPP)
3
Has the program developed a plan to manage software safety? (MIL-STD-882)
Software Safety Program
Plan (SwSPP)
4
Has a hazard tracking system been developed in accordance with MIL-STD-882? (MIL-STD882, NAVSEAINST 5000.8)
Hazard Tracking System
5
Is the ESOH risk assessment matrix used by the program derived from MIL-STD-882? (MILSTD-882, NAVSEAINST 5000.8)
ESOH Risk Assessment
Matrix
7
Have appropriate potential hazards been derived from the historical data, lessons learned
from similar legacy systems or earlier fielded versions of the same system and have
alternative candidate mitigations been identified and documented? (NAVSEAINST 5000.8)
Has the program identified potential accident and combat threat scenario hazards for the
system and documented in the hazard tracking system? (MIL-STD-882, NAVSEAINST
5000.8)
8
Has the program identified ESOH responsibilities and how the program will integrate system
safety-ESOH considerations into the systems engineering process? (DoDI 5000.02)
PESHE
9
Has the program identified initial safety requirements (prescribed or derived) from applicable
standards, specifications, regulations, design handbooks, safety design checklists, and other
sources? (MIL-STD-882)
Safety
Requirements/Criteria
Assessment (SR/CA)
6
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
Preliminary Hazard
Analysis (PHA)
Threat Hazard Assessment
(THA)
32
Highlighted text will be talked about in detail
Ordnance Safety & Security Activity
10
11
12
13
14
SRR – Criteria Statements (cont’d)
Has the following been updated:
-Hazard Tracking System
-Hazards
-ESOH Risk
-Reports on high and serious ESOH Risk
-Reports on ESOH risk acceptance
(DoDI 5000.02,MIL-STD-882, and NAVSEAINST 5000.8)
Have the cost estimates been updated to reflect any ESOH system requirements related
cost data?
Has the safety/ESOH Program provided input to the Risk Management Plan to ensure
that the ESOH risks are identified and mitigated? (NAVSEAINST 5000.8)
Has an initial assessment of the severity and probability of mishap risk been documented
for each identified hazard in the system safety hazard tracking system? (MIL-STD-882,
NAVSEAINST 5000.8)
Have safety critical functions been identified and entered into the hazard tracking system?
(NAVSEAINST 5000.8)
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
Hazard Tracking
System/Risk Acceptance
Cost Estimates
Risk Management Plan
Hazard Tracking System
PHA
33
Ordnance Safety & Security Activity
Artifacts for System Functional
Review (SFR)
 System Requirements Review (SFR) – A formal review of the conceptual design of the system to
establish its capability to satisfy requirements. It establishes a functional baseline.
– System Safety Management Plan
– System Safety Program Plan
– Cost Analysis Requirements Document
– Software Safety Program Plan
– Program Risk (Input)
– System Safety Lead Designation Letter
– Integrated Master Schedule
– ESOH Risk Assessment Matrix
– Service Cost Position
– HAZMAT Management Plan
– Interface Requirement Specification
– Hazard Tracking System/Risk Acceptance
– Software Requirement Specification
– Functional Hazard Analysis
– System Design Specification
– Threat Hazard Assessment
– Test Plan and Procedures
– Integrated Hazard Analysis/System Hazard Analysis– Configuration Management Plan
– Safety Requirements/Criteria Assessment
– Health Hazard Assessment
– Capability Development Document
– Concept of Operations
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
34
Ordnance Safety & Security Activity
1
2
3
SFR – Criteria Statements
Has the government's system safety engineering approach been clearly and fully
documented and approved? (MIL-STD-882, NAVSEAINST 5000.8)
Has the developer's system safety engineering approach been clearly and fully
documented and approved? (MIL-STD-882, NAVSEAINST 5000.8)
Has the program developed and approved a plan to manage software safety? (MIL-STD882)
10
Has the PM designated a System Safety Lead/Manager or Principal for Safety (PFS)?
(OPNAVINST 5100.24B)
Is the risk assessment matrix approved by the appropriate authority? (MIL-STD-882)
(NAVSEAINST 5000.8)
Does the program have a plan for managing Hazardous Materials (HAZMAT) in the
system? (DoDI 5000.02)
Have identified hazards been reviewed, assessed, and mitigations identified in
accordance with MIL-STD-882 and have they been updated in the hazard tracking
system? (NAVSEAINST 5000.8)
Have safety critical functions been identified and has a means of mapping to the physical
design been established? (MIL-STD-882)
Have all prescribed and derived safety requirements been documented in the system
functional baseline? (MIL-STD-882)
Has the program identified the characteristics of each potential accident and combat
threat scenario hazards for the system and documented in the hazard tracking system?
11
Have safety related interoperability considerations for the system of systems been
identified?
12
Have health hazards associated with the system been identified?
4
5
6
7
8
9
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
SSMP
SSPP
SwSPP
System Safety Lead
Designation Letter
ESOH Risk Assessment
Matrix
HAZMAT Management Plan
(HMMP)
PHA/Hazard Tracking
System
Functional Hazard Analysis
(FHA)
SR/CA
THA
Integrated Hazard
Analysis(IHA)/System
Hazard Analysis (SHA)
Health Hazard Assessment
(HHA)
35
Ordnance Safety & Security Activity
SFR – Criteria Statements (cont’d)
14
Has the following been updated:
-Hazard Tracking System
- Hazards
-ESOH Risk
-Reports on high and serious ESOH Risk
-Reports on ESOH risk acceptance
(DoDI 5000.02, MIL-STD-882, and NAVSEAINST 5000.8)
Does the CDD include safety/ESOH-related capability statements in Sections 14, 15, and
other sections, as applicable?
15
16
Has the CONOPS been reviewed by safety/ESOH staff to gain insight into the refined
mission and operation of the system?
Has the CARD been updated to contain appropriate safety/ESOH-related cost data?
CONOPS
CARD
17
Have ESOH risks been included in the overall program risk management process?
(NAVSEAINST 5000.8)
Program Risk (Input)
18
Has the Safety Program provided input to the Integrated Master Schedule to include
safety/ESOH activities?
Integrated Master
Schedule
19
Does the Service Cost Position include costs associated with resourcing the safety/ESOH
effort and recommended hazard mitigations? (NAVSEAINST 5000.8)
Service Cost Position
20
Has the Safety Program reviewed the IRS to determine if any safety risk exists or needs to
be mitigated through the requirements process?
IRS
21
Does the Software Requirements Specification (SRS) contain safety-critical software
requirements?
SRS
22
Has safety/ESOH reviewed the System Design Specification (SDS) and provided specific
system safety/ESOH design requirements?
SDS
13
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
Hazard Tracking
System/Risk
Acceptance
CDD
36
Ordnance Safety & Security Activity
23
24
25
26
SFR – Criteria Statements (cont’d)
Does the SDS contain system safety interlocks and assumptions?
Does the Test Plan and Procedures contain specific ESOH and system safety requirements
to conduct testing and specific tests to verify recommended mitigation?
SDS
Test Plan and
Procedures
Do configuration management (CM) plans define the role and involvement of safety/ESOH?
Have the Safety/ESOH analysis tools and processes been evaluated against the CM tools
and process for compatibility?
Configuration
Management Plan
Configuration
Management Plan
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
37
Highlighted text will be talked about in detail
Ordnance Safety & Security Activity
1
2
3
SFR – Criteria Statements
Has the government's system safety engineering approach been clearly and fully
documented and approved? (MIL-STD-882, NAVSEAINST 5000.8)
Has the developer's system safety engineering approach been clearly and fully
documented and approved? (MIL-STD-882, NAVSEAINST 5000.8)
Has the program developed and approved a plan to manage software safety? (MIL-STD882)
10
Has the PM designated a System Safety Lead/Manager or Principal for Safety (PFS)?
(OPNAVINST 5100.24B)
Is the risk assessment matrix approved by the appropriate authority? (MIL-STD-882)
(NAVSEAINST 5000.8)
Does the program have a plan for managing Hazardous Materials (HAZMAT) in the
system? (DoDI 5000.02)
Have identified hazards been reviewed, assessed, and mitigations identified in
accordance with MIL-STD-882 and have they been updated in the hazard tracking
system? (NAVSEAINST 5000.8)
Have safety critical functions been identified and has a means of mapping to the physical
design been established? (MIL-STD-882)
Have all prescribed and derived safety requirements been documented in the system
functional baseline? (MIL-STD-882)
Has the program identified the characteristics of each potential accident and combat
threat scenario hazards for the system and documented in the hazard tracking system?
11
Have safety related interoperability considerations for the system of systems been
identified?
12
Have health hazards associated with the system been identified?
4
5
6
7
8
9
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
SSMP
SSPP
SwSPP
System Safety Lead
Designation Letter
ESOH Risk Assessment
Matrix
HAZMAT Management Plan
(HMMP)
PHA/Hazard Tracking
System
Functional Hazard Analysis
(FHA)
SR/CA
THA
Integrated Hazard
Analysis(IHA)/System
Hazard Analysis (SHA)
Health Hazard Assessment
(HHA)
38
Highlighted text will be talked about in detail
Ordnance Safety & Security Activity
SFR – Criteria Statements (cont’d)
14
Has the following been updated:
-Hazard Tracking System
- Hazards
-ESOH Risk
-Reports on high and serious ESOH Risk
-Reports on ESOH risk acceptance
(DoDI 5000.02, MIL-STD-882, and NAVSEAINST 5000.8)
Does the CDD include safety/ESOH-related capability statements in Sections 14, 15, and
other sections, as applicable?
15
16
Has the CONOPS been reviewed by safety/ESOH staff to gain insight into the refined
mission and operation of the system?
Has the CARD been updated to contain appropriate safety/ESOH-related cost data?
CONOPS
CARD
17
Have ESOH risks been included in the overall program risk management process?
(NAVSEAINST 5000.8)
Program Risk (Input)
18
Has the Safety Program provided input to the Integrated Master Schedule to include
safety/ESOH activities?
Integrated Master
Schedule
19
Does the Service Cost Position include costs associated with resourcing the safety/ESOH
effort and recommended hazard mitigations? (NAVSEAINST 5000.8)
Service Cost Position
20
Has the Safety Program reviewed the IRS to determine if any safety risk exists or needs to
be mitigated through the requirements process?
IRS
21
Does the Software Requirements Specification (SRS) contain safety-critical software
requirements?
SRS
22
Has safety/ESOH reviewed the System Design Specification (SDS) and provided specific
system safety/ESOH design requirements?
SDS
13
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
Hazard Tracking
System/Risk
Acceptance
CDD
39
Highlighted text will be talked about in detail
Ordnance Safety & Security Activity
23
24
25
26
SFR – Criteria Statements (cont’d)
Does the SDS contain system safety interlocks and assumptions?
Does the Test Plan and Procedures contain specific ESOH and system safety requirements
to conduct testing and specific tests to verify recommended mitigation?
SDS
Test Plan and
Procedures
Do configuration management (CM) plans define the role and involvement of safety/ESOH?
Have the Safety/ESOH analysis tools and processes been evaluated against the CM tools
and process for compatibility?
Configuration
Management Plan
Configuration
Management Plan
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
40
Ordnance Safety & Security Activity
Safety Artifacts for Preliminary
Design Review (PDR)
 Preliminary Design Review (PDR) - A formal review that confirms the preliminary design logically
follows the SFR findings and meets the requirements. It normally results in approval to begin detailed
design.
– System Safety Lead Designation Letter
– ESOH Risk Acceptance
– Laser Safety Review Board
– ESOH Risk Matrix
– Preliminary Hazard Assessment
– Hazard Tracking System
– Programmatic ESOH Evaluation (PESHE)
– Hazardous Materials Management Plan
– Functional Hazard Analysis
– Acquisition Strategy
– Configuration Steering Board
– Failure Modes, Effects and Criticality Analysis
– HSI Plan
– PDR Results
– Request for Proposal
– System Requirements/Criteria Assessment
– Demilitarization and Disposal Plan
– Code Level Hazard Analysis
– Requirements Tracking System
– Integrated Hazard Analysis
– Test and Evaluation Master Plan
– System Hazard Analysis
– Trade Studies
– Subsystem Hazard Analysis
– Interface Requirements Specification
– Operating & Support Hazard Analysis
– Corrosion Prevention and Control Plan
– WSESRB Technical Data Package
– Configuration Management Plan
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
41
Ordnance Safety & Security Activity
1
2
3
4
5
6
7
8
9
10
11
PDR – Criteria Statements
Is the System Safety Lead/Manager or PFS chairing System Safety Working Groups on a
regular basis with documented results? (OPNAVINST 5100.24B)
System Safety Lead
Designation Letter
Are all ESOH hazards assessed using the program's approved ESOH risk matrix? (MIL-STD882, NAVSEAINST 5000.8)
Have identified hazards been assessed in accordance with MIL-STD-882 and have they been
documented in the hazard tracking system? (MIL-STD-882, NAVSEAINST 5000.8)
ESOH Risk Assessment
Matrix
PHA/ Hazard Tracking
System
Have design alternatives for eliminating hazards or reducing their impact been considered for
each potential hazard? (MIL-STD-882, NAVSEAINST 5000.8)
Has the expected effectiveness of each alternative risk mitigation been documented in the
hazard tracking system? (MIL-STD-882, NAVSEAINST 5000.8)
Does the program maintain a National Environmental Policy Act (NEPA)/Executive Order (EO)
12114 Compliance Schedule for all system-related NEPA/EO 12114 analyses? (DoDI 5000.02)
Does the program maintain a Programmatic ESOH Evaluation document that identifies ESOH
responsibilities, and how the program will integrate system safety-ESOH considerations into the
systems engineering process, the ESOH risk management process, the hazard tracking system,
and ESOH hazards and their associated risks? (DoDI 5000.02, NAVSEAINST 5000.8)
Has the program reported the current status of all high and serious ESOH risks and applicable
ESOH technology requirements at program reviews? (Include in Risk Management Board
(RMB), GATES and Milestone Reviews) (NAVSEAINST 5000.8)
Has the plan for managing HAZMAT been approved? (MIL-STD-882)
Have hazards associated with HAZMAT been identified, analyzed and documented in the hazard
tracking system? (MIL-STD-882, NAVSEAINST 5000.8)
Has the program identified safety critical functions and have they been allocated to the
subsystem? (MIL-STD-882)
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
PHA
PHA
PESHE
PESHE
PESHE
HMMP
HMMP
FHA
42
Ordnance Safety & Security Activity
PDR – Criteria Statements (cont’d)
Have safety aspects of design features and safety critical functions been identified and analyzed,
and have mitigations been identified? (MIL-STD-882, NAVSEAINST 5000.8)
FHA
14
Have all safety requirements been assigned a method of verification? (MIL-STD-882)
Has the program identified the level of analytical rigor required for each software subsystem?
(MIL-STD-882)
SR/CA
Code Level Hazard
Analysis
15
Have hazards associated with the integrated system/subsystems been assessed, mitigated, and
documented in the hazard tracking system? (MIL-STD-882, NAVSEAINST 5000.8)
IHA/SHA
16
Has the program identified and analyzed hazards associated with subsystems and are they
documented in the hazard tracking system? (NAVSEAINST 5000.8)
17
Has safety/ESOH reviewed and assessed all environmental critical processes and components?
(NAVSEAINST 5000.8)
Sub-System Hazard
Analysis (SSHA)
SSHA; SHA; Operating
and Support Hazard
Analysis (O&SHA)
18
Have hazards associated with combat threat scenarios been identified, analyzed, and
documented in the hazard tracking system? (MIL-STD-882)
THA
19
Have hazards associated with operation and support of the system been identified and
documented in the hazard tracking system? (MIL-STD-882, NAVSEAINST 5000.8)
O&SHA
20
Has the program been reviewed by the Weapon System and Explosives Safety Review Board
(WSESRB)? (as applicable) (NAVSEAINST 8020.6)
WSESRB Technical
Data Package (TDP)
21
Has the program presented to the Laser Safety Review Board (LSRB) (as required)?
LSRB
12
13
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
43
Ordnance Safety & Security Activity
22
23
24
25
PDR – Criteria Statements (cont’d)
Has the process for ESOH risk acceptance and user representative concurrence (for
high and serious risk) been established and implemented? (DoDI 5000.02,
NAVSEAINST 5000.8)
Has the following been updated:
-Hazard Tracking System
-Hazards
-ESOH Risk
-Reports on high and serious ESOH Risk
-Reports on ESOH risk acceptance
(DoDI 5000.02, MIL-STD-882, and NAVSEAINST 5000.8)
Does the Acquisition Strategy include a summary of the Programmatic ESOH
Evaluation?
ESOH Risk Acceptance
Hazard Tracking System/Risk
Acceptance
Acquisition Strategy
Configuration Steering Board
26
Is safety/ESOH a voting member of the Configuration Steering Board?
Has a Failure Modes and Effect Criticality Analysis (FMECA) been performed to
review the potential failure modes and determine if they create or contribute to a
safety risk?
27
Has safety/ESOH been included in the Human Systems Integration (HSI) Plan?
HSI Plan
28
Has Safety/ESOH been included in the PDR?
PDR Results
29
Does the RFP specify safety/ESOH-related requirements and CDRLs?
RFP
30
Does the Demilitarization and Disposal Plan include safety and environmental hazard
data (e.g. hazardous materials)?
Demilitarization and Disposal
Plan
31
Does the Requirement Tracking System include system safety/ESOH requirements?
Requirements Tracking System
FMECA
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
44
Ordnance Safety & Security Activity
PDR – Criteria Statements (cont’d)
32
Has the TEMP been updated to address government and contractor safety/ESOH testing?
TEMP
33
TEMP
34
Does the TEMP been updated to address environmental planning for test events?
Has Commercial Off the Shelf (COTS)/Non-developmental Items (NDI) been assessed for
safety/ESOH impact?
35
Has safety/ESOH identified all safety critical and safety related interfaces in the IRS?
IRS; SSHA; SHA
38
Does the Corrosion Prevention and Control (CPC) Plan include hazard analyses of
alternative corrosion prevention materials and processes?
Are safety/ESOH roles and responsibilities defined for each change control process or
change control board?
Have all CM processes, products and tools captured necessary safety/ESOH fields and
flags?
39
Are Safety/ESOH analysis tools and processes compatible with CM tools and processes?
36
37
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
Trade Studies, PHA
CPC Plan
Configuration
Management (CM) Plan
CM Plan
CM Plan
45
Highlighted text will be talked about in detail
Ordnance Safety & Security Activity
1
2
3
4
5
6
7
8
9
10
11
PDR – Criteria Statements
Is the System Safety Lead/Manager or PFS chairing System Safety Working Groups on a
regular basis with documented results? (OPNAVINST 5100.24B)
System Safety Lead
Designation Letter
Are all ESOH hazards assessed using the program's approved ESOH risk matrix? (MIL-STD882, NAVSEAINST 5000.8)
Have identified hazards been assessed in accordance with MIL-STD-882 and have they been
documented in the hazard tracking system? (MIL-STD-882, NAVSEAINST 5000.8)
ESOH Risk Assessment
Matrix
PHA/ Hazard Tracking
System
Have design alternatives for eliminating hazards or reducing their impact been considered for
each potential hazard? (MIL-STD-882, NAVSEAINST 5000.8)
Has the expected effectiveness of each alternative risk mitigation been documented in the
hazard tracking system? (MIL-STD-882, NAVSEAINST 5000.8)
Does the program maintain a National Environmental Policy Act (NEPA)/Executive Order (EO)
12114 Compliance Schedule for all system-related NEPA/EO 12114 analyses? (DoDI 5000.02)
Does the program maintain a Programmatic ESOH Evaluation document that identifies ESOH
responsibilities, and how the program will integrate system safety-ESOH considerations into the
systems engineering process, the ESOH risk management process, the hazard tracking system,
and ESOH hazards and their associated risks? (DoDI 5000.02, NAVSEAINST 5000.8)
Has the program reported the current status of all high and serious ESOH risks and applicable
ESOH technology requirements at program reviews? (Include in Risk Management Board
(RMB), GATES and Milestone Reviews) (NAVSEAINST 5000.8)
Has the plan for managing HAZMAT been approved? (MIL-STD-882)
Have hazards associated with HAZMAT been identified, analyzed and documented in the hazard
tracking system? (MIL-STD-882, NAVSEAINST 5000.8)
Has the program identified safety critical functions and have they been allocated to the
subsystem? (MIL-STD-882)
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
PHA
PHA
PESHE
PESHE
PESHE
HMMP
HMMP
FHA
46
Highlighted text will be talked about in detail
Ordnance Safety & Security Activity
PDR – Criteria Statements (cont’d)
Have safety aspects of design features and safety critical functions been identified and analyzed,
and have mitigations been identified? (MIL-STD-882, NAVSEAINST 5000.8)
FHA
14
Have all safety requirements been assigned a method of verification? (MIL-STD-882)
Has the program identified the level of analytical rigor required for each software subsystem?
(MIL-STD-882)
SR/CA
Code Level Hazard
Analysis
15
Have hazards associated with the integrated system/subsystems been assessed, mitigated, and
documented in the hazard tracking system? (MIL-STD-882, NAVSEAINST 5000.8)
IHA/SHA
16
Has the program identified and analyzed hazards associated with subsystems and are they
documented in the hazard tracking system? (NAVSEAINST 5000.8)
17
Has safety/ESOH reviewed and assessed all environmental critical processes and components?
(NAVSEAINST 5000.8)
Sub-System Hazard
Analysis (SSHA)
SSHA; SHA; Operating
and Support Hazard
Analysis (O&SHA)
18
Have hazards associated with combat threat scenarios been identified, analyzed, and
documented in the hazard tracking system? (MIL-STD-882)
THA
19
Have hazards associated with operation and support of the system been identified and
documented in the hazard tracking system? (MIL-STD-882, NAVSEAINST 5000.8)
O&SHA
20
Has the program been reviewed by the Weapon System and Explosives Safety Review Board
(WSESRB)? (as applicable) (NAVSEAINST 8020.6)
WSESRB Technical
Data Package (TDP)
21
Has the program presented to the Laser Safety Review Board (LSRB) (as required)?
LSRB
12
13
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
47
Highlighted text will be talked about in detail
Ordnance Safety & Security Activity
22
23
24
25
PDR – Criteria Statements (cont’d)
Has the process for ESOH risk acceptance and user representative concurrence (for
high and serious risk) been established and implemented? (DoDI 5000.02,
NAVSEAINST 5000.8)
Has the following been updated:
-Hazard Tracking System
-Hazards
-ESOH Risk
-Reports on high and serious ESOH Risk
-Reports on ESOH risk acceptance
(DoDI 5000.02, MIL-STD-882, and NAVSEAINST 5000.8)
Does the Acquisition Strategy include a summary of the Programmatic ESOH
Evaluation?
ESOH Risk Acceptance
Hazard Tracking System/Risk
Acceptance
Acquisition Strategy
Configuration Steering Board
26
Is safety/ESOH a voting member of the Configuration Steering Board?
Has a Failure Modes and Effect Criticality Analysis (FMECA) been performed to
review the potential failure modes and determine if they create or contribute to a
safety risk?
27
Has safety/ESOH been included in the Human Systems Integration (HSI) Plan?
HSI Plan
28
Has Safety/ESOH been included in the PDR?
PDR Results
29
Does the RFP specify safety/ESOH-related requirements and CDRLs?
RFP
30
Does the Demilitarization and Disposal Plan include safety and environmental hazard
data (e.g. hazardous materials)?
Demilitarization and Disposal
Plan
31
Does the Requirement Tracking System include system safety/ESOH requirements?
Requirements Tracking System
FMECA
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
48
Highlighted text will be talked about in detail
Ordnance Safety & Security Activity
PDR – Criteria Statements (cont’d)
32
Has the TEMP been updated to address government and contractor safety/ESOH testing?
TEMP
33
TEMP
34
Does the TEMP been updated to address environmental planning for test events?
Has Commercial Off the Shelf (COTS)/Non-developmental Items (NDI) been assessed for
safety/ESOH impact?
35
Has safety/ESOH identified all safety critical and safety related interfaces in the IRS?
IRS; SSHA; SHA
38
Does the Corrosion Prevention and Control (CPC) Plan include hazard analyses of
alternative corrosion prevention materials and processes?
Are safety/ESOH roles and responsibilities defined for each change control process or
change control board?
Have all CM processes, products and tools captured necessary safety/ESOH fields and
flags?
39
Are Safety/ESOH analysis tools and processes compatible with CM tools and processes?
36
37
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
Trade Studies, PHA
CPC Plan
Configuration
Management (CM) Plan
CM Plan
CM Plan
49
Ordnance Safety & Security Activity
Questions
 Who is doing the safety engineering work for the program prior to MS B?
 Was safety able to conduct hazard analysis on prototypes, engineering models,
models & simulations, engineering development models?
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
50
BREAK – 10 minutes
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
51
Agenda
Ordnance Safety & Security Activity







Introduction
SETR Policy Requirements
What is SETR
Recommended SETRs
Tailoring
Safety in SETR Process
Acquisition Framework Deep Dive






Material Solution Analysis
Technology Development
Engineering and Manufacturing Development
Production and Deployment
Operations and Support
Phase Overview
Present SETRs
Artifacts
Safety Criteria Statements
Safety Driving Factors
Summary/Conclusion
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
52
Ordnance Safety & Security Activity
Engineering and Manufacturing
Development
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
53
Ordnance Safety & Security Activity
Engineering and Manufacturing
Development Activities (Pre-CDR)
 Purpose: Develop a system or increment of
capability, complete full system integration; develop
an affordable manufacturing process, minimize
logistics footprint; demonstrate system integration
Integrated System Design
 Enter: Mature Technology; Approved Requirements;
Full Funding in Fiscal Year Defense Plan (FYDP)
 Activities: Define System of System Functionality &
Interfaces, Complete Detailed Design, CDR,
Establish Product Baseline,
 Guided by: CDD, Acquisition Strategy, SEP & TEMP
 Exit: Complete System-Level CDR and Post-CDR
Assessment
 SETR Events: Critical Design Review
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
54
Ordnance Safety & Security Activity
Engineering and Manufacturing
Development Activities (Post-CDR)
 Purpose: Finalize a system or increment of
capability, develop an affordable manufacturing
process, minimize logistics footprint
System Capability & Manufacturing Process
Demonstration
 Enter: Mature Post-CDR Assessment and
Establishment of initial Product Baseline
 Activities: Developmental Testing (DT) Assesses
Progress Against Technical Parameters, and
Operational Assessments (OA) Against CDD
 Guided by: CDD, Acquisition Strategy, SEP & TEMP
 Exit: System Demonstrated in Intended Environment
using production-representative articles;
Manufacturing Processes Demonstrated; Meets Exit
Criteria and MS C Entrance Requirements
 SETR Events: Integration Readiness Review (IRR),
Test Readiness Review (TRR), System Verification
Review (SVR)
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
55
Ordnance Safety & Security Activity
Artifacts for Critical Design
Review (CDR)
 Critical Design Review (CDR) – A formal review conducted to evaluate the completeness of the
design and its interfaces.
– Hazard Tracking System
– Cost estimates
– PESHE
– Demilitarization and Disposal Plan
– Hazardous Material Management Plan
– Lifecycle Sustainment Plan
– ESOH Risk Acceptance
– Operational Risk Management
– Code Level Hazard Analysis
– Program Risk (Input to other processes)
– Functional Hazard Analysis
– Integrated Master Schedule
– Health Hazard Assessment
– Interface Requirement Specification
– Sub-System Hazard Analysis
– Software Requirement Specification
– Integrated Hazard Analysis/System Hazard Analysis
– Systems Engineering Management Plan
– Critical Safety Items
– Systems Engineering Plan
– Operating and Support Hazard Analysis
– Test and Evaluation Management Plan
– Safety Assessment Report
– Test Plan and Procedures
– Weapon Systems Explosives Safety Review Board
– Total Ownership Cost
Technical Data Packages
– Configuration Management
– Acquisition Strategy
– Capability Production Document
– CDR Results
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
56
Ordnance Safety & Security Activity
1
2
3
CDR – Criteria Statements
Has the hazard tracking system been updated and maintained with current ESOH hazard
and risk assessment data? (MIL-STD-882, NAVSEAINST 5000.8)
Has the program reported the current status of all high and serious ESOH risks and
applicable ESOH technology requirements at program reviews? (Include in Risk
Management Board, GATES and Milestone Reviews) (NAVSEAINST 5000.8)
Have hazardous materials, wastes, and pollutants (discharges/emissions/noise) associated
with the system been documented in the system safety hazard tracking system? (DoDI
5000.02)
Hazard Tracking System
PESHE
PESHE
5
Have hazards associated with HAZMAT been identified, analyzed and mitigation controls
implemented? (MIL-STD-882)
Is the process for ESOH risk acceptance and user representative concurrence (for high and
serious risk) being executed? (DoDI 5000.02, NAVSEAINST 5000.8)
6
Has a safety analysis of the software been completed to the identified level of rigor and
documented? (Joint Software Systems Safety Engineering Handbook)
Code Level Hazard
Analysis
7
Have safety aspects of design and safety critical functions been allocated and have the
mitigations been incorporated into the design? (MIL-STD-882)
FHA
4
9
Have mitigations associated with the integrated or interoperable system and subsystems
been verified in the design? (System of Systems Guidebook and MIL-STD-882)
Has the program completed hazard analyses associated with all subsystems? (MIL-STD882)
10
Have risks associated with health hazards been finalized and mitigations incorporated into
the design? (MIL-STD-882)
8
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
HMMP
ESOH Risk Acceptance
IHA/SHA
SSHA
HHA
57
Ordnance Safety & Security Activity
11
12
13
14
CDR – Criteria Statements
Have risks associated with operation and support of the system been finalized and mitigations
incorporated into the design? (MIL-STD-882)
Has the program updated all Critical Safety Items and safety related Critical Application Items?
(DFARS 209.270)
Has the program compiled and documented an overall assessment of safety of the system that
covers system operations, hazard and associated risk data, mitigations, and states that the system is
ready to test, operate, or proceed to the next acquisition phase? (MIL-STD-882, NAVSEAINST
5000.8)
O&SHA
Critical Safety Items/Critical
Application Items
Safety Assessment Report
(SAR)/ Hazard Tracking
System
WSESRB Technical Data
Package
16
Has the program been reviewed by the WSESRB? (as applicable) (NAVSEAINST 8020.6)
Has the following been updated:
-Hazard Tracking System
-Hazards
-ESOH Risk
-Reports on high and serious ESOH Risk
-Reports on ESOH risk acceptance
(DoDI 5000.02, MIL-STD-882, and NAVSEAINST 5000.8)
Does the Acquisition Strategy contain an updated summary of the Programmatic ESOH Evaluation as
required?
17
Does the Capabilities Production Document (CPD) include safety/ESOH-related capability statements
in sections 14, 15, and other sections, as applicable?
CPD
18
Has safety/ESOH been included in the Critical Design Review (CDR)?
CDR results
19
Have the cost estimates been updated to reflect any safety/ESOH related cost data?
Cost Estimates
20
Has the Demilitarization and Disposal Plan been updated to include safety and environmental hazard
data (e.g. hazardous materials)
Demilitarization and Disposal
Plan
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
58
15
Hazard Tracking System/Risk
Acceptance
Acquisition Strategy
Ordnance Safety & Security Activity
CDR – Criteria Statements
21
Does the LCSP include safety and environmental requirements?
LCSP
Operational Risk
Management
22
Is Safety/ESOH included in the management of operational risks?
23
Is Safety/ESOH included in the Risk Management Plan? (NAVSEAINST 5000.8)
24
Are safety/ESOH milestones included in the Integrated Master Schedule?
Program Risk (Input to
other processes)
Integrated Master
Schedule
25
Has the Safety Program reviewed the Interface IRS to determine if any safety risk exists or needs to
be mitigated through the requirements process?
IRS
26
Has the SRS been updated to include any additional safety-critical software requirements?
SRS
30
Has the SEMP been updated to explain how safety/ESOH is integrated into the systems engineering
process?
Has the SEP been updated to explain how safety/ESOH is addressed and integrated into systems
engineering?
Does the TEMP include safety/ESOH compliance requirements (such as NEPA) and specific safety
testing requirements?
Has the Test Plan and Procedures been drafted to include specific ESOH and system safety
requirements to conduct testing and include specific tests to verify recommended mitigation
measures?
31
Is safety/ESOH included in the updated TOC?
TOC
32
Is there evidence that the Safety/ESOH team is actively involved in the CM processes and using the
CM tools as described in the CM plans?
CM
27
28
29
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
SEMP
SEP
TEMP
Test Plan and Procedures
59
Highlighted text will be talked about in detail
Ordnance Safety & Security Activity
1
2
3
CDR – Criteria Statements
Has the hazard tracking system been updated and maintained with current ESOH hazard
and risk assessment data? (MIL-STD-882, NAVSEAINST 5000.8)
Has the program reported the current status of all high and serious ESOH risks and
applicable ESOH technology requirements at program reviews? (Include in Risk
Management Board, GATES and Milestone Reviews) (NAVSEAINST 5000.8)
Have hazardous materials, wastes, and pollutants (discharges/emissions/noise) associated
with the system been documented in the system safety hazard tracking system? (DoDI
5000.02)
Hazard Tracking System
PESHE
PESHE
5
Have hazards associated with HAZMAT been identified, analyzed and mitigation controls
implemented? (MIL-STD-882)
Is the process for ESOH risk acceptance and user representative concurrence (for high and
serious risk) being executed? (DoDI 5000.02, NAVSEAINST 5000.8)
6
Has a safety analysis of the software been completed to the identified level of rigor and
documented? (Joint Software Systems Safety Engineering Handbook)
Code Level Hazard
Analysis
7
Have safety aspects of design and safety critical functions been allocated and have the
mitigations been incorporated into the design? (MIL-STD-882)
FHA
4
9
Have mitigations associated with the integrated or interoperable system and subsystems
been verified in the design? (System of Systems Guidebook and MIL-STD-882)
Has the program completed hazard analyses associated with all subsystems? (MIL-STD882)
10
Have risks associated with health hazards been finalized and mitigations incorporated into
the design? (MIL-STD-882)
8
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
HMMP
ESOH Risk Acceptance
IHA/SHA
SSHA
HHA
60
Highlighted text will be talked about in detail
Ordnance Safety & Security Activity
11
12
13
14
CDR – Criteria Statements
Have risks associated with operation and support of the system been finalized and mitigations
incorporated into the design? (MIL-STD-882)
Has the program updated all Critical Safety Items and safety related Critical Application Items?
(DFARS 209.270)
Has the program compiled and documented an overall assessment of safety of the system that
covers system operations, hazard and associated risk data, mitigations, and states that the system is
ready to test, operate, or proceed to the next acquisition phase? (MIL-STD-882, NAVSEAINST
5000.8)
O&SHA
Critical Safety Items/Critical
Application Items
Safety Assessment Report
(SAR)/ Hazard Tracking
System
WSESRB Technical Data
Package
16
Has the program been reviewed by the WSESRB? (as applicable) (NAVSEAINST 8020.6)
Has the following been updated:
-Hazard Tracking System
-Hazards
-ESOH Risk
-Reports on high and serious ESOH Risk
-Reports on ESOH risk acceptance
(DoDI 5000.02, MIL-STD-882, and NAVSEAINST 5000.8)
Does the Acquisition Strategy contain an updated summary of the Programmatic ESOH Evaluation as
required?
17
Does the Capabilities Production Document (CPD) include safety/ESOH-related capability statements
in sections 14, 15, and other sections, as applicable?
CPD
18
Has safety/ESOH been included in the Critical Design Review (CDR)?
CDR results
19
Have the cost estimates been updated to reflect any safety/ESOH related cost data?
Cost Estimates
20
Has the Demilitarization and Disposal Plan been updated to include safety and environmental hazard
data (e.g. hazardous materials)
Demilitarization and Disposal
Plan
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
61
15
Hazard Tracking System/Risk
Acceptance
Acquisition Strategy
Highlighted text will be talked about in detail
Ordnance Safety & Security Activity
CDR – Criteria Statements
21
Does the LCSP include safety and environmental requirements?
LCSP
Operational Risk
Management
22
Is Safety/ESOH included in the management of operational risks?
23
Is Safety/ESOH included in the Risk Management Plan? (NAVSEAINST 5000.8)
24
Are safety/ESOH milestones included in the Integrated Master Schedule?
Program Risk (Input to
other processes)
Integrated Master
Schedule
25
Has the Safety Program reviewed the Interface IRS to determine if any safety risk exists or needs to
be mitigated through the requirements process?
IRS
26
Has the SRS been updated to include any additional safety-critical software requirements?
SRS
30
Has the SEMP been updated to explain how safety/ESOH is integrated into the systems engineering
process?
Has the SEP been updated to explain how safety/ESOH is addressed and integrated into systems
engineering?
Does the TEMP include safety/ESOH compliance requirements (such as NEPA) and specific safety
testing requirements?
Has the Test Plan and Procedures been drafted to include specific ESOH and system safety
requirements to conduct testing and include specific tests to verify recommended mitigation
measures?
31
Is safety/ESOH included in the updated TOC?
TOC
32
Is there evidence that the Safety/ESOH team is actively involved in the CM processes and using the
CM tools as described in the CM plans?
CM
27
28
29
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
SEMP
SEP
TEMP
Test Plan and Procedures
62
Ordnance Safety & Security Activity
Artifacts for Integration Readiness
Review (IRR)
 Integration Readiness Review (IRR) - A formal review that assesses readiness of software
systems for integrated configuration item testing
– Programmatic ESOH Evaluation (PESHE)
– Integrated Hazard Analysis/System Hazard Analysis
– Hazard Tracking System
– ESOH Risk Acceptance
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
63
Ordnance Safety & Security Activity
1
2
3
IRR – Criteria Statements
Has the program reported the current status of all high and serious ESOH risks and
applicable ESOH technology requirements at program reviews? (Include in Risk
Management Board, GATES and Milestone Reviews) (NAVSEAINST 5000.8)
Have hazard mitigations associated with the integrated system/subsystems been
assessed, mitigated, and documented in the hazard tracking system? (MIL-STD-882,
NAVSEAINST 5000.8)
Has the following been updated:
-Hazard Tracking System
-Hazards
-ESOH Risk
-Reports on high and serious ESOH Risk
-Reports on ESOH risk acceptance
(DoDI 5000.02, MIL-STD-882, and NAVSEAINST 5000.8)
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
PESHE
IHA/SHA
Hazard Tracking
System/Risk Acceptance
64
Ordnance Safety & Security Activity
Artifacts for Test Readiness
Review (TRR)
 Test Readiness Review (TRR) – A formal review of contractors’ readiness to begin testing on both
hardware and software configuration items.
– PESHE
– Safety Release
– ESOH Risk Acceptance
– Health Hazard Assessment
– Sub-System Hazard Analysis
– Integrated Hazard Analysis/System Hazard Analysis
– Safety Requirements/Criteria Assessment
– Operating and Support Hazard Analysis
– Safety Assessment Report
– Weapon Systems Explosives Safety Review Board Technical Data Packages
– Laser Safety Review Board
– Hazard Tracking System/Risk Acceptance
– Operational Risk Management
– Safety Review – ECP/SCN/SPR/PTR/STR
– Test Plan and Procedures
– Requirements Tracking System
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
65
Ordnance Safety & Security Activity
TRR – Criteria Statements
Has the program reported the current status of all high and serious ESOH risks and
applicable ESOH technology requirements at program reviews? (Include in Risk
1 Management Board, GATES and MS Reviews) (NAVSEAINST 5000.8)
Has the PM ensured that a safety release(s) that covers the system/subsystem as
configured for test events has been provided to testers prior to testing; does the safety
release identify hazards, associated risks, provide warnings and cautions, and restrictions
2 placed on testing? (DoDI 5000.02)
Have the test plan and environment been considered when assessing the hazards
3 applicable for test events? (DoDI 5000.02)
Have all of the ESOH risk associated with the test been accepted at the appropriate level?
4 (DoDI 5000.02, NAVSEAINST 5000.8)
Have risks associated with health hazards been assessed and mitigations been
5 implemented prior to testing? (MIL-STD-882, NAVSEAINST 5000.8)
Are the identified hazard mitigations associated with system and subsystems traceable to
6 test procedures? (MIL-STD-882)
7 Does the test plan and criteria address interoperability hazards? (MIL-STD-882)
Are the identified safety requirements associated with system and subsystems tests
8 traceable to procedures? (MIL-STD-882)
Are hazards associated with test, operation, maintenance and support of the system and
components included in the procedures, warnings, cautions, and manuals? (MIL-STD9 882)
Has the program compiled and documented an overall assessment of safety for testing?
10 (MIL-STD-882)
PESHE
Safety Release
Safety Release
ESOH Risk Acceptance
HHA
SSHA
IHA/SHA
SR/CA
O&SHA
SAR
WSESRB Technical Data
11 Has the program been reviewed by the WSESRB? (As applicable) (NAVSEAINST 8020.6) Package
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
66
Ordnance Safety & Security Activity
TRR – Criteria Statements
12
Has the program presented to the LSRB? (As required)
13
Has the following been updated:
-Hazard Tracking System
- Hazards
-ESOH Risk
-Reports on high and serious ESOH Risk
-Reports on ESOH risk acceptance
(DoDI 5000.02, MIL-STD-882, and NAVSEAINST 5000.8)
14
16
Is the updated safety/ESOH data been included in the management of operational risks?
Is safety/ESOH included in the formal Engineering Change Proposal (ECP)/Safety Change Notice
(SCN)/Software Problem Report (SPR)/Program Trouble Report (PTR)/Software Trouble Reports
(STR) change review process?
Has the Test Plan and Procedures been updated to include specific ESOH and system safety
requirements to conduct testing and include specific tests to verify recommended mitigation
measures?
17
Does the requirement tracking system include results of the safety/ESOH analyses and previously
conducted testing?
15
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
LSRB
Hazard Tracking
System/Risk Acceptance
Operational Risk
Management
Safety Review ECP/SCN/SPR/PTR/STR
Test Plan and Procedures
Requirements Tracking
System
67
Highlighted text will be talked about in detail
Ordnance Safety & Security Activity
TRR – Criteria Statements
Has the program reported the current status of all high and serious ESOH risks and
applicable ESOH technology requirements at program reviews? (Include in Risk
1 Management Board, GATES and MS Reviews) (NAVSEAINST 5000.8)
Has the PM ensured that a safety release(s) that covers the system/subsystem as
configured for test events has been provided to testers prior to testing; does the safety
release identify hazards, associated risks, provide warnings and cautions, and restrictions
2 placed on testing? (DoDI 5000.02)
Have the test plan and environment been considered when assessing the hazards
3 applicable for test events? (DoDI 5000.02)
Have all of the ESOH risk associated with the test been accepted at the appropriate level?
4 (DoDI 5000.02, NAVSEAINST 5000.8)
Have risks associated with health hazards been assessed and mitigations been
5 implemented prior to testing? (MIL-STD-882, NAVSEAINST 5000.8)
Are the identified hazard mitigations associated with system and subsystems traceable to
6 test procedures? (MIL-STD-882)
7 Does the test plan and criteria address interoperability hazards? (MIL-STD-882)
Are the identified safety requirements associated with system and subsystems tests
8 traceable to procedures? (MIL-STD-882)
Are hazards associated with test, operation, maintenance and support of the system and
components included in the procedures, warnings, cautions, and manuals? (MIL-STD9 882)
Has the program compiled and documented an overall assessment of safety for testing?
10 (MIL-STD-882)
PESHE
Safety Release
Safety Release
ESOH Risk Acceptance
HHA
SSHA
IHA/SHA
SR/CA
O&SHA
SAR
WSESRB Technical Data
11 Has the program been reviewed by the WSESRB? (As applicable) (NAVSEAINST 8020.6) Package
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
68
Ordnance Safety & Security Activity
Questions
 What happens if you have not completed your hazard analyses and gotten your risk
accepted at the appropriate level?
 What IPTs do you find most effective in helping to integrate safety into the design?
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
69
BREAK – 10 minutes
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
70
Ordnance Safety & Security Activity
Artifacts for System Verification
Review (SVR)
 System Verification Review (SVR) - Verifies that the actual item (which represents the
production configuration) complies with the performance specification.
– Programmatic ESOH Evaluation (PESHE)
– Integrated Hazard Analysis
– System Hazard Analysis
– System Requirements/Criteria Assessment
– Operating & Support Hazard Analysis
– Hazard Tracking System
– ESOH Risk Acceptance
– Cost Analysis Requirements Document
– Request for Proposal
– Requirements Tracking System
– Statement of Work
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
71
Ordnance Safety & Security Activity
1
2
3
4
5
6
7
8
9
10
11
SVR – Criteria Statements
Has the PESHE been updated to reflect test results to date? (DoDI 5000.02)
Has the program reported the current status of all high and serious ESOH risks and applicable
ESOH technology requirements at program reviews? (Include in Risk Management Board
(RMB), GATES and Milestone Reviews) (NAVSEAINST 5000.8)
Have the interface and interoperability hazards and mitigations been updated to reflect test
results to date? (MIL-STD-882)
Have safety requirements been updated to reflect the test failures and design changes affected
the safety requirements? (MIL-STD-882)
Have the operating and support hazards and mitigations been updated to reflect test results to
date? (MIL-STD-882)
PESHE
Has safety analysis and regression testing been conducted on all changes?
Has the following been updated:
-Hazard Tracking System
-Hazards
-ESOH Risk
-Reports on high and serious ESOH Risk
-Reports on ESOH risk acceptance
(DoDI 5000.02, MIL-STD-882,and NAVSEAINST 5000.8)
SR/CA
Has the cost associated with safety/ESOH changes been included in the final CARD?
Has the Low Rate Initial Production (LRIP) RFP been updated to include ESOH/system safety
requirements? (NAVSEAINST 5000.8)
Has the Requirement Tracking System been updated to include changes to the system
safety/ESOH requirements?
Has the LRIP SOW been updated to include system safety/ESOH requirements and CDRLs?
(NAVSEAINST 5000.8)
CARD
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
PESHE
IHA/SHA
SR/CA
O&SHA
Hazard Tracking
System/Risk
Acceptance
RFP
Requirement
Tracking System
SOW
72
Highlighted text will be talked about in detail
Ordnance Safety & Security Activity
1
2
3
4
5
6
7
8
9
10
11
SVR – Criteria Statements
Has the PESHE been updated to reflect test results to date? (DoDI 5000.02)
Has the program reported the current status of all high and serious ESOH risks and applicable
ESOH technology requirements at program reviews? (Include in Risk Management Board
(RMB), GATES and Milestone Reviews) (NAVSEAINST 5000.8)
Have the interface and interoperability hazards and mitigations been updated to reflect test
results to date? (MIL-STD-882)
Have safety requirements been updated to reflect the test failures and design changes affected
the safety requirements? (MIL-STD-882)
Have the operating and support hazards and mitigations been updated to reflect test results to
date? (MIL-STD-882)
PESHE
Has safety analysis and regression testing been conducted on all changes?
Has the following been updated:
-Hazard Tracking System
-Hazards
-ESOH Risk
-Reports on high and serious ESOH Risk
-Reports on ESOH risk acceptance
(DoDI 5000.02, MIL-STD-882,and NAVSEAINST 5000.8)
SR/CA
Has the cost associated with safety/ESOH changes been included in the final CARD?
Has the Low Rate Initial Production (LRIP) RFP been updated to include ESOH/system safety
requirements? (NAVSEAINST 5000.8)
Has the Requirement Tracking System been updated to include changes to the system
safety/ESOH requirements?
Has the LRIP SOW been updated to include system safety/ESOH requirements and CDRLs?
(NAVSEAINST 5000.8)
CARD
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
PESHE
IHA/SHA
SR/CA
O&SHA
Hazard Tracking
System/Risk
Acceptance
RFP
Requirement
Tracking System
SOW
73
Ordnance Safety & Security Activity
Artifacts for Production Readiness
Review (PRR)
 Production Readiness Review (PRR) - Determines if the design is ready for production,
production engineering problems have been resolved, and the producer has accomplished
adequate planning for the production phase
– Programmatic ESOH Evaluation (PESHE)
– Safety Assessment Report
– Threat Hazard Assessment
– Hazard Tracking System
– ESOH Risk Acceptance
– Cost Analysis Requirements Document
– Laser Safety Review Board
– Acquisition Strategy
– Capability Production Document
– Capability Production Document Plan
– Safety Review – ECP/SCN/SPR/PTR/STR
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
74
Ordnance Safety & Security Activity
1
2
3
4
5
6
7
8
9
PRR – Criteria Statements
Has the PESHE been updated to reflect the final production configuration? (DoDI 5000.02)
Has the program reported the current status of all high and serious ESOH risks and applicable
ESOH technology requirements at program reviews? (Include in Risk Management Board
(RMB), GATES and Milestone Reviews) (NAVSEAINST 5000.8)
Have hazardous materials, wastes, and pollutants (discharges/emissions/ noise) associated with
the system been documented in the system safety hazard tracking system? (DoDI 5000.02,
NAVSEAINST 5000.8)
Has the mishap risk of all hazards been accepted by the appropriate authorities and been
communicated to parties responsible for production release, operational test and deployment?
(MIL-STD-882, NAVSEAINST 5000.8)
Has the user representative provided formal concurrence with all serious and high safety residual
risk acceptance and informal concurrence with all medium safety residual risks? (DoDI 5000.02,
NAVSEAINST 5000.8)
Have warnings, cautions, workarounds, and administrative controls applicable to safety been
included in training material and technical manuals?
Has the program compiled and documented an overall assessment of safety for production and
release? (MIL-STD-882)
Have the system threat hazards and mitigation results been communicated to the appropriate
organizations? (MIL-STD-882)
Has the following been updated:
-Hazard Tracking System
-Hazards
-ESOH Risk
-Reports on High and Serious ESOH Risk
-Reports on ESOH Risk Acceptance
(DoDI 5000.02, MIL-STD-882, and NAVSEAINST 5000.8)
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
PESHE
PESHE
PESHE
ESOH Risk
Acceptance
ESOH Risk
Acceptance
SAR
SAR
THA
Hazard Tracking
System/Risk
Acceptance
75
Ordnance Safety & Security Activity
10
PRR – Criteria Statements
Has the program presented to the LSRB? (as required)
Does the Acquisition Strategy contain an updated summary of the Programmatic ESOH
Evaluation as required?
LSRB
Acquisition
Strategy
12
Does the final CPD contain safety/ESOH capability statements?
CPD
13
Has the CPC Plan been updated to address hazards during production and sustainment?
14
Has Safety reviewed ECP/SCN/SPR/PTR/STRs changes to the configuration that may create or
worsen a safety/ESOH issue?
CPC Plan
Safety Review ECP/SCN/SPR/PT
R/STR
11
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
76
Highlighted text will be talked about in detail
Ordnance Safety & Security Activity
1
2
3
4
5
6
7
8
9
PRR – Criteria Statements
Has the PESHE been updated to reflect the final production configuration? (DoDI 5000.02)
Has the program reported the current status of all high and serious ESOH risks and applicable
ESOH technology requirements at program reviews? (Include in Risk Management Board
(RMB), GATES and Milestone Reviews) (NAVSEAINST 5000.8)
Have hazardous materials, wastes, and pollutants (discharges/emissions/ noise) associated with
the system been documented in the system safety hazard tracking system? (DoDI 5000.02,
NAVSEAINST 5000.8)
Has the mishap risk of all hazards been accepted by the appropriate authorities and been
communicated to parties responsible for production release, operational test and deployment?
(MIL-STD-882, NAVSEAINST 5000.8)
Has the user representative provided formal concurrence with all serious and high safety residual
risk acceptance and informal concurrence with all medium safety residual risks? (DoDI 5000.02,
NAVSEAINST 5000.8)
Have warnings, cautions, workarounds, and administrative controls applicable to safety been
included in training material and technical manuals?
Has the program compiled and documented an overall assessment of safety for production and
release? (MIL-STD-882)
Have the system threat hazards and mitigation results been communicated to the appropriate
organizations? (MIL-STD-882)
Has the following been updated:
-Hazard Tracking System
-Hazards
-ESOH Risk
-Reports on High and Serious ESOH Risk
-Reports on ESOH Risk Acceptance
(DoDI 5000.02, MIL-STD-882, and NAVSEAINST 5000.8)
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
PESHE
PESHE
PESHE
ESOH Risk
Acceptance
ESOH Risk
Acceptance
SAR
SAR
THA
Hazard Tracking
System/Risk
Acceptance
77
Highlighted text will be talked about in detail
Ordnance Safety & Security Activity
10
PRR – Criteria Statements
Has the program presented to the LSRB? (as required)
Does the Acquisition Strategy contain an updated summary of the Programmatic ESOH
Evaluation as required?
LSRB
Acquisition
Strategy
12
Does the final CPD contain safety/ESOH capability statements?
CPD
13
Has the CPC Plan been updated to address hazards during production and sustainment?
14
Has Safety reviewed ECP/SCN/SPR/PTR/STRs changes to the configuration that may create or
worsen a safety/ESOH issue?
CPC Plan
Safety Review ECP/SCN/SPR/PT
R/STR
11
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
78
Agenda
Ordnance Safety & Security Activity







Introduction
SETR Policy Requirements
What is SETR
Recommended SETRs
Tailoring
Safety in SETR Process
Acquisition Framework Deep Dive






Material Solution Analysis
Technology Development
Engineering and Manufacturing Development
Production and Deployment
Operations and Support
Phase Overview
Present SETRs
Artifacts
Safety Criteria Statements
Safety Driving Factors
Summary/Conclusion
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
79
Ordnance Safety & Security Activity
Production and Deployment
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
80
Ordnance Safety & Security Activity
Production and Deployment Activities
(Pre-FRP)
 Purpose: Achieve an Operational Capability that
satisfies mission needs
 Enter: Acceptable performance in DT & OA; mature
software; no significant manufacturing risks;
approved CPD; refined integrated architecture;
acceptable interoperability and operational
supportability; demonstration of affordability; fully
funded; phased for rapid deployment.
 Activities: IOT&E, LFT&E and Interoperability
Testing of Production or Production-Representative
Articles; IOC possible
 Guided by: CPD and TEMP
 Exit: System Operationally Effective, Suitable and
Ready for Full-Rate Production
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
81
Ordnance Safety & Security Activity
Production and Deployment Activities
(Post-FRP)
 Purpose: Achieve an operational Capability that
satisfies mission needs
 Enter: Beyond LRIP & LFT&E Reports (OSD
T&E/LFT&E programs) Submitted to Congress.
 Activities: Full-Rate Production; Fielding and
Support of Fielded Systems; IOC/FOC
 Guided by: Acquisition Strategy & Life Cycle
Sustainment Plan
 Exit: Full Operational Capability; Deployment
Complete
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
82
Agenda
Ordnance Safety & Security Activity







Introduction
SETR Policy Requirements
What is SETR
Recommended SETRs
Tailoring
Safety in SETR Process
Acquisition Framework Deep Dive






Material Solution Analysis
Technology Development
Engineering and Manufacturing Development
Production and Deployment
Operations and Support
Phase Overview
Present SETRs
Artifacts
Safety Criteria Statements
Safety Driving Factors
Summary/Conclusion
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
83
Ordnance Safety & Security Activity
Operations and Support
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
84
Ordnance Safety & Security Activity
Operations and Support Activities
 Purpose: Execute a support program that meets materiel
readiness and operational support performance
requirements, and sustains the system in the most costeffective manner over its total life cycle.
 Enter: Approved CPD; approved LCSP; successful FRP
Decision
Activities (Sustainment): Performance-Based Life-Cycle
Product Support (PBL) planning, development,
implementation, and management; initiate system
modifications as necessary; continuing reviews of
sustainment strategies
Guided by (Sustainment): Acquisition Strategy/LCSP
Activities (Disposal): Demilitarize and Dispose of Systems
IAW Legal and Regulatory Requirements, Particularly
Environmental Considerations and Explosives Safety
Guided by: Programmatic Environment, Safety, and
Occupational Health Evaluation (PESHE)
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
85
Ordnance Safety & Security Activity
Artifacts for In-Service Review (ISR)
 In-Service Review (ISR) – A formal technical review that is to characterize in-Service technical and
operational health of the deployed system by providing an assessment of risk, readiness, technical
status, and trends in a measurable form that will substantiate in-Service support and budget
priorities.
– Hazard Tracking System
– ESOH Risk Acceptance
– Operating and Support Hazard Analysis
– Safety Assessment Report
– WSESRB Technical Data Package
– Lifecycle Sustainment Plan
– Program Risk (input to other processes)
– Safety Review (ECPSCN/SPR/PTR/STR)
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
86
Highlighted text will be talked about in detail
Ordnance Safety & Security Activity
1
2
3
4
5
6
7
8
9
10
ISR – Criteria Statements
Has a safety evaluation been performed on new hazards or recommended mitigations from in
service evaluations, industrial hygiene survey, routine safety evaluations and mishap reports
and have the results been documented in the hazard tracking system? (DoDI 5000.02,
NAVSEAINST 5000.8)
Has the program notified the user community of the changes in safety residual risk? (MILSTD-882, NAVSEAINST 5000.8)
Has a hazard analysis been completed for all Class A and B mishap investigations associated
with the system? (DoDI 5000.02, NAVSEAINST 5000.8)
Have hazards and mitigations associated with operation and support of the system been reevaluated based on user, maintainer, test and training community feedback? (MIL-STD-882,
NAVSEAINST 5000.8)
Has the program compiled and documented an overall assessment of the changes in safety
since the last review? (MIL-STD-882)
Has the program been reviewed by the WSESRB? (as applicable) (NAVSEAINST 8020.6)
Has the following been updated:
-Hazard Tracking System
-Hazards
-ESOH Risk
-Reports on High and Serious ESOH Risk
-Reports on ESOH Risk Acceptance
(DoDI 5000.02, MIL-STD-882, and NAVSEAINST 5000.8)
Does the LCSP include safety and environmental requirements and processes?
(NAVSEAINST 5000.8)
Are operational safety/ESOH issues documented and communicated back to safety/ESOH?
(NAVSEAINST 5000.8)
Is safety/ESOH included in the formal ECP/SCN/SPR/PTR/STRs change review process?
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
Hazard Tracking System
ESOH Risk Acceptance
Hazard Tracking System
O&SHA
SAR
WSESRB Technical Data
Package
Hazard Tracking
System/Risk Acceptance
LCSP
Program Risk (Input to
other processes)
Safety Review ECP/SCN/SPR/PTR/STR
87
Ordnance Safety & Security Activity
Questions
 Do you maintain the hazard tracking system once the system is fielded and track new
or changed hazards? What about technology insertion?
 What is the method used for receiving feedback from the field to incorporate mishaps,
ECPs and lessons learned?
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
88
Agenda
Ordnance Safety & Security Activity







Introduction
SETR Policy Requirements
What is SETR
Recommended SETRs
Tailoring
Safety in SETR Process
Acquisition Framework Deep Dive






Material Solution Analysis
Technology Development
Engineering and Manufacturing Development
Production and Deployment
Operations and Support
Phase Overview
Present SETRs
Artifacts
Safety Criteria Statements
Safety Driving Factors
Summary/Conclusion
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
89
Ordnance Safety & Security Activity




Summary/Conclusion
Participation in engineering and program document development helps
safety engage in the plan, design, test, and sustainment processes.
Safety’s participation in the technical review process is critical to an
effective system safety program.
Early participation in technical reviews is important now that PDR is set
prior to MS B (MDAPs only) and competitive prototyping is required by
law. This tutorial provides you with extensive material to accomplish
this.
Safety SETR criteria helps to ensure a robust safety program and
provides standardization across the Naval Enterprise.
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
90
Questions??
Ordnance Safety & Security Activity
 Contact Information
– Mr. Arch McKinlay, Naval Ordnance Safety and Security Activity,
[email protected]
– Ms. Peggy Rogers, Naval Ordnance Safety and Security Activity,
[email protected]
– Mr. Stuart Whitford, Naval Ordnance Safety and Security Activity,
[email protected]
– Ms. Karen Gill, Booz Allen Hamilton, [email protected]
– Ms. Kristin Thompson, Booz Allen Hamilton, [email protected]
PROVIDING ORDNANCE SAFETY FOR THE NAVAL ENTERPRISE
91
References
Ordnance Safety & Security Activity









MIL-STD-882D, Department of Defense Standard Practice for System Safety.
Chief of Naval Operations, Navy System Safety Program Policy, OPNAVINST 5100.24B, 6 February 2007.
Chief of Naval Operations/Commandant of the Marine Corps, Navy Laser Hazards Control Program,
OPNAVINST 5100.27B/MCO 5104.1C, 2 May 2008.
Commanders, Naval Air Systems Command, Naval Sea Systems Command, Naval Supply Systems Command,
Naval Facilities Systems Command, Space and Naval Warfare Systems Command and Marine Corps Systems
Command (2010). MARCORSYSCOM Order 5400.5, SPAWARINST 5000.1, NAVFACINST 5000.15,
NAVSUPINST 5000.21, NAVSEAINST 5000.9, and NAVAIRINST 5000.24, Naval SYSCOM Systems
Engineering Policy, 19 January 2010.
Commanders, Naval Air Systems Command, Naval Sea Systems Command, Naval Supply Systems Command,
Naval Facilities Systems Command, Space and Naval Warfare Systems Command and Marine Corps Systems
Command (2010). MARCORSYSCOM Order 5400.5, SPAWARINST 5000.1, NAVFACINST 5000.15,
NAVSUPINST 5000.21, NAVSEAINST 5000.9, and NAVAIRINST 5000.24, Naval Systems Engineering
Technical Review Handbook Version 1.0, 19 January 2010.
Commanders, Naval Air Systems Command, Naval Sea Systems Command, Naval Supply Systems Command,
Naval Facilities Systems Command, Space and Naval Warfare Systems Command and Marine Corps Systems
Command (2010). MARCORSYSCOM Order 5000.3, SPAWARINST 3058.1, NAVFACINST 5000.15,
NAVSUPINST 5000.20, NAVSEAINST 5000.8, and NAVAIRINST 5000.24, Naval SYSCOM Risk Management
Policy, 20 July 2008.
Naval Sea Systems Command, Department of the Navy Weapon Systems Explosive Safety Review Board, 11
March 2008.
Department of Defense Instruction (DoDI) 5000.02, Operation of the Defense Acquisition System. United States
of America: Department of Defense, 8 December 2008.
Department of Defense Directive (DoDD) 5000.01, Operation of the Defense Acquisition System. United States
of America: Department
of Defense,
20 November
2007.
92
PROVIDING
ORDNANCE
SAFETY
FOR THE NAVAL ENTERPRISE