Transcript No Slide Title

Inventing the Operational Safety Assessment
ATN’99
Steve Paasch
Federal Aviation Administration
Aircraft Engineering Division
Avionics Branch
AIR-130 c/o ANM-100S
1601 Lind Avenue SW
Renton, WA 98055-4056
phone: 425-227-1186
fax: 425-227-1181
email: [email protected]
•
•
•
•
Operational Safety
Assessments - How they
became to be
The purpose of conducting
formal safety assessments
The processes and methods
involved in performing safety
assessments
Controller Pilot Data Link
Communications (CPDLC)
program Operational Safety
Assessment (OSA)
Inventing the Operational Safety Assessment
Contents
2
Operational Safety Assessments How they became to be
Technology Aging
& Evolution
Aviation Expansion
& Globalization
Need for New
Operational Capabilities
Airspace
Modernization
Safety
RTCA TASK
FORCE IV
Need For
End-to-End
“Certification” Certification
Efficiencies
OSA
Digital Communications
Requirements (SC-189/WG-53)
Inventing the Operational Safety Assessment
3
RTCA SC-189 / EUROCAE WG-53
• Chartered to develop safety, performance, &
•
interoperability requirements for air traffic
services supported by communications
Subgroup 2 is working on methods and
examples for developing operational
environment descriptions and performing
operational safety assessments
• air-ground end-to-end safety assessment from an
operational viewpoint
Inventing the Operational Safety Assessment
4
Website for SC-189
http://www.mews.org/atssir//
Inventing the Operational Safety Assessment
5
RTCA Task Force IV
• Opportunities to Reduce the Time, to Reduce
•
•
•
•
•
the Cost, and to Provide Better Certification
Service
Achieving Operational Benefits
Human Performance
End-to-End Aviation Systems Considerations
Regulation, Policy, and Guidance
Development
Authority Organization, Processes, and
Industry Interface
Inventing the Operational Safety Assessment
6
End-to-End Aviation Systems
Considerations
•
The Task Force heard many concerns that systems were not being properly
considered overall, or "from end to end." The introduction of new elements
into the ground or airborne parts of the system are not generally preceded
by appropriate systems engineering practices, including definition of
operations concepts and requirements. It is clear that overall system
performance is rarely specified and that authorities often do not take a
structured approach to establishing the requirements for International
Airspace System (INAS) systems and components. It is common for new
ground or airborne components to have specifications or performance that
are not matched to the other elements of the system with which they work
to perform their function. One consequence may be that the new system
element is over-specified, and therefore more expensive than it should be
to achieve the incremental improvement in performance. Another possible
consequence is that the new system element is not properly specified in
light of the performance of other system elements, and the expected
improvement in efficiency from the new system element is not attained.
Inventing the Operational Safety Assessment
7
Task Force IV Recommendations
•
•
Recommendation 2: The authorities should establish and maintain
a systems engineering capability. This function should be used to
establish overall performance requirements for all advanced
systems and their subsystems, in conjunction with the user
community. As part of this effort, the authorities should consider
developing clear approval standards and processes for ground
system elements that are integrated, to the degree necessary, with
airborne system element certification. (Section 3.4)
Recommendation 5: The authorities should broadly implement a
process where the regulators and applicants come to an early and
clear agreement on their respective roles, responsibilities,
expectations, schedules, and standards to be used in certification
projects. The process should apply broadly across airborne and
ground systems, allow non-applicant equipment suppliers to engage
in certification programs, and provide greater opportunity to approve
components or processes independent of the airplane. (Section 3.6)
Inventing the Operational Safety Assessment
8
The purpose of conducting formal
safety assessments
Inventing the Operational Safety Assessment
9
Starting from what we have and going
to what we need
• We have a traditional aircraft-related system
safety assessment process
• What is it?
• What is for?
Inventing the Operational Safety Assessment
10
What is the traditional, aircraft related
system safety assessment process?
• It is a systems engineering activity to
*
*
*
assure that safety objectives are
met......by identifying where systems
requirements are needed to eliminate or
mitigate potential safety problems
Systems engineering is a two sided coin
- optimistic vs pessimistic
SSA turns the systems engineering
perspective of performance,
functionality, form, etc., around
“Do this” vs “what if it doesn’t do this?”
Inventing the Operational Safety Assessment
11
What is the traditional, aircraft related
system safety assessment process for?
• In a nutshell--
*
*
*
To have a systematic way to analyze aircraft
and aircraft systems function-related failure
conditions, as well as failure condition
contributors and mitigators, in order to:
Set safety objectives for failure conditions
Identify systems safety requirements to meet
safety objectives
Assure systems safety requirements (and
thus safety objectives) are met
Inventing the Operational Safety Assessment
12
A Systems Engineering Discipline
• The System Safety Assessment side of the
*
*
*
*
systems engineering coin:
has its own methods for discovering
requirements
has its own processes to organize the
methods
has its own vocabulary to facilitate the
processes
has its own guidance materials for passing
knowledge on
Inventing the Operational Safety Assessment
13
System safety assessments are tied to
aircraft of a type, and the installed
systems and equipment, or engines and
engine systems.
Inventing the Operational Safety Assessment
14
...But the aircraft isn’t the only
player in the airspace game...
Inventing the Operational Safety Assessment
15
Broadening our horizons beyond
an aircraft.....to the airspace system
* multiple aircraft
* multiple capabilities
* ground systems
* signal networks
* operational procedures
* ad hoc evolution
* modernization program
Inventing the Operational Safety Assessment
16
What is the Operational Safety
Assessment Process for?
• In a nutshell-- A systematic way to analyze
*
*
*
airspace and air traffic management servicerelated operational hazards, and operational
hazard contributors and mitigators, in order
to:
Set safety objectives for operational hazards
Identify systems and procedural safety
requirements to meet safety objectives
Assure systems and procedural safety
requirements (and thus safety objectives)
are met
Inventing the Operational Safety Assessment
17
An Airspace Planning Discipline
• The Operational Safety Assessment side of
*
*
*
*
the airspace planning coin:
should have its own methods for
discovering requirements
should have its own processes to organize
the methods
should have its own vocabulary to facilitate
the processes
should have its own guidance material for
passing knowledge on
Inventing the Operational Safety Assessment
18
The processes and methods involved
in performing safety assessments
Inventing the Operational Safety Assessment
19
Inventing a vocabulary
• Starting with the system safety assessment
vocabulary
Inventing the Operational Safety Assessment
20
What to say when good systems
go bad
• Failure Condition
• Failure
• Failure Mode
• Fault
• Error
Inventing the Operational Safety Assessment
21
What can we do with our specialized
vocabulary?
• We can organize
our concepts into
relationships
Inventing the Operational Safety Assessment
22
Aircraft designer’s view
Aircraft
FAILURE CONDITION
ERROR
FAILURE
FAILURE
FAULT
FAULT
ERROR
PHYSICS
PHYSICS
FAULT
FAILURE
MODES
FAULT
PHYSICS
ERROR
ERROR PHYSICS ERROR
Inventing the Operational Safety Assessment
23
Terminology comparison
SSA:
• Functions
• Failure conditions
• Failures
• Failure Modes
• Faults
• Errors
Inventing the Operational Safety Assessment
OSA:
• Air Traffic Services
• Operational hazards
• Failures
• Failure Modes
• Faults
• Errors
24
Airspace designer’s view
Operational
Environment
Operational
 Hazard
Airspace characteristics:
Separation minima
Traffic density

Traffic complexity
Operational
boundary
ATSU
 Effect on
ATS
System
boundary
ATS
boundary
Risk
mitigation
strategy
 Failure
AOCU

Fault
ATSU
 Procedural error
Aircraft
CNS/ATM
System

Fault
Supporting
ATS
ATS
Procedures
Denotes institutional boundaries
Inventing the Operational Safety Assessment
25
Inventing a process
• Starting with the system safety assessment
process
Inventing the Operational Safety Assessment
26
A metaphor for systems engineering?
Inventing the Operational Safety Assessment
27
System Safety Assessment Process
• Identify aircraft or systems functions
• Identify failure conditions
• Determine failure condition severity
• Set safety objectives based on failure
•
•
•
condition severity
Determine system safety requirements to
meet safety objectives
Allocate safety requirements across systems
and components
Assure safety requirements are met
Inventing the Operational Safety Assessment
28
What can we do with our specialized
process?
• We can organize
our activities to
be systematic
and thorough
Inventing the Operational Safety Assessment
29
System Safety Assessment Process discovering safety requirements
as-built
objectives
Preliminary System Safety
Assessments
Functional Hazard Assessment
Aircraft or System
Function Definition
Inventing the Operational Safety Assessment
Common
Cause
Analyses
strategies & refinement
System Safety Assessments
30
Process comparison
SSA:
• Identify aircraft or
•
•
•
systems functions
Identify failure
conditions
Determine failure
condition severity
Set safety
objectives based on
failure condition
severity
Inventing the Operational Safety Assessment
OSA:
• Identify air traffic
•
•
•
services
Identify operational
hazards
Determine oper.
hazard severity
Set safety
objectives based on
operational hazard
severity
31
Process comparison (continued)
SSA:
• Determine systems
•
•
safety requirements
to meet safety
objectives
Allocate safety
requirements
across systems and
components
Assure safety
requirements are
met
Inventing the Operational Safety Assessment
OSA:
• Determine
•
•
operational safety
reqts to meet safety
objectives
Allocate safety reqts
across institutions
and airspace
components
Assure safety
requirements are
met
32
Operational Safety Assessment Process discovering safety requirements
Ground System Safety Assessments
as-built
Aircraft System Safety Assessments
Institutional Safety Assessments
strategies & refinement
Allocation of Safety Objectives
and Requirements
Operational Hazard Assessment
Operational Environment
Definition (OED -- Services and
airspace characteristics that
may affect hazard severity)
Inventing the Operational Safety Assessment
Common
Cause
Analyses
objectives
33
How do operational
safety assessments
and system safety
assessments relate?
Inventing the Operational Safety Assessment
34
Planners & Implementers
Other assessments
Operational
safety assessment
(OSA)
Coordination
Operational
environment
definition (OED)
Operational
hazard
assessment
(OHA)
Coordination
Allocation of
safety objectives
& requirements
Operational &
Safety Objectives
Safety Rqmts
CNS/ATM System & Procedures Developers
Development
Activities
Safety Rqmts,
Proposed
Revisions
Development
Assurance
Institutional
safety
assessment
Rqmts, Design,
‘As built’
Operational
Capability
Entry into service
CNS/ATM Service Providers and Users (Operators)
Operations
Coordination
Monitoring
requirements
Monitoring
Inventing the Operational Safety Assessment
Collected data
Continued
operational
safety
35
Inventing methods?
• Starting with system safety assessment
methods?
Inventing the Operational Safety Assessment
36
System Safety Assessment Methods
• Inverse relationship for classifying failure
•
•
•
•
•
•
conditions and setting assurance levels
Fail Safe Principles
Fault Tree Analysis
Failure Modes and Effects Analysis
Markov Analysis
Dependence Diagramming
Mathematics of failure rates, probability, and
Boolean algebra
Inventing the Operational Safety Assessment
37
What can we do with our specialized
methods?
• We can discover
cause and
measure effect in
a relatively
precise fashion
with tabular,
graphical,
mathematical,
logical means
Inventing the Operational Safety Assessment
38
System Safety Assessment hazard
classification scheme
Effect on Airplane
No effect on
operational
capabilities or
safety
Slight reduction in
functional
capabilities or
safety margins
Effect on
Occupants
excluding Flight
Crew
Inconvenience
Physical
discomfort
Effect on Flight
Crew
No effect on flight
crew
Slight increase in
workload
Physical
discomfort or a
significant
increase in
workload
Allowable
Qualitative
Probability
No Probability
Requirement
<---Probable---->
Allowable
Quantitative
Probability:
Average
Probability per
Flight Hour on the
Order of:
No Probability
Requirement
Classification of
Failure
Conditions
Significant
reduction in
functional
capabilities or
safety margins
Physical distress,
possibly including
injuries
Large reduction in
functional
capabilities or
safety margins
Normally with
hull loss
Serious or fatal
injury to a small
number of
passengers or
cabin crew
Physical distress
or excessive
workload impairs
ability to perform
tasks
Multiple fatalities
<----Remote----->
Extremely
<------------------>
Remote
Extremely
Improbable
<------------------>
<------------------>
<------------------>
<10-3
<10-5
<10-7
<-----Major------>
<--Hazardous--->
Fatalities or
incapacitation
<10-9
Note 1
No Safety Effect
<-----Minor------>
Inventing the Operational Safety Assessment
Catastrophic
39
Inventing the Operational Safety Assessment
40
Methods comparison
SSA:
•
•
•
•
•
•
•
OSA:
Inverse relationship for • Inverse relationship for
classifying operational
classifying failure
hazards and setting
conditions and setting
assurance levels
assurance levels
• Otherwise, we’re
Fail Safe Principles
working on it
Fault Tree Analysis
• Matrix and templates
Failure Modes and Effects
• Institutional methods
Analysis
at institutional levels
Markov Analysis
Dependence Diagramming • CPDLC
Mathematics of failure
rates, probability, and
Boolean algebra
Inventing the Operational Safety Assessment
• OED
• Hazard table
• FTA
• Reqts & allocation
41
Operational Safety Assessment hazard
classification scheme
Hazard Classification
Effect on Operations
2
3
4
Inability to sustain flight,
placing the aircraft into an
unavoidable fatal event.
Total loss of aircraft control.
1 (most severe)
Condition places aircraft(s) in
position where either a
survivable accident or
significant incident occurs.
Hull loss/multiple fatalities in
unsurvivable crash.
Effect on Aircrew
Aircrew unable to avoid
accident.
Survivable accident with less
than full hull loss, fatalities
possible. Significant incident
events with fatalities or
injuries.
Physical distress or higher
workload such that the aircrew
cannot be relied upon to
perform their tasks.
Loss of efficiency, possible
violation of separation
standards, increases in missed
approaches, requirement to
execute changes in altitude,
direction or speed to assure
safety margin.
Physical discomfort
Failure conditions
have no safety
significance.
Effect on Occupants
Significant increase in risk of
accident with possible injuries
to aircrew and/or occupants.
Requirement to execute
significant changes in altitude,
direction or speed to assure
safety margin.
Physical distress, possibly
including injuries
Sudden inability to provide
any degree of service within
one or more airspace sectors
for a significant period of
time where service denial, or
where sufficiently misleading
information given to the
service provider and/or to the
aircrew causing an aircraft
accident.
Sudden inability to maintain
safety margins in one or more
ATC sectors for a significant
period of time.
Through either a high increase
in workload for the controller,
or the provision of misleading
information, the flight
operations would result in an
accident or incident.
Increase in aircrew workload
that can be accommodated and
well within their capabilities
to perform. Some loss of
efficiency.
Failure conditions that slightly
impair the ability to maintain
safe ATS within one or more
airspace sectors without
warning and for a significant
period of time. Procedures are
able to compensate for the loss
of function, but controller
workload is likely to be high
until the overall system
demand is reduced.
Nuisance failure not
requiring action by
flight crew during
flight.
Effect on Service
Provider
Accident/Incident
Occurrence
Total loss of flight control,
mid-air collision,
controlled flight into terrain,
or surface movement
collision.
Increase in critical near midair collisions, or runway
incursions, or repeated loss of
separation where the distance
between the aircraft involved
is less than ½ the standard.
Significant increase in aircrew
workload with impaired
efficiency and significant
concern over the consequences
of failure.
Compromised ability to
maintain safe ATS within one
or more sectors without
warning. Significant increase
in controller workload.
Possible access to less
information than required for
normal operations, leading to
the loss of separation or
provision of erroneous
information that significantly
decreases aircrew’s ability to
perform required tasks.
Contingency measures can be
applied but safety risk in the
airspace is high and multiple
losses of separation are likely
until system demand is
reduced.
Increase in near mid-air
collisions, or runway
incursions, or repeated loss of
separation.
Occasional loss of separation.
Failure conditions
have no safety
significance
Inventing the Operational Safety Assessment
5 (least severe)
No events.
Nuisance failure
requiring routine
restoration of
capability.
42
Inverse relationship
Likelihood
of Occurrence
Hazard
Class
Probable
Remote
Extremely
Remote
Extremely
Improbable
1
2
3
4
5
Risk Acceptance Cases
Unacceptable
Acceptable with Review
Acceptable
Acceptable with Review - Unacceptable with
Single Point Failures and Common-Cause Failures
Inventing the Operational Safety Assessment
43
Inventing guidance material
• Adding to system safety assessment
guidance
Inventing the Operational Safety Assessment
44
Safety assessment guidance material
• AC 23.1309-1C for Normal, Utility,
Acrobatic, Commuter Airplanes
• AC/AMJ 25.1309-1B for Transport
Airplanes
• AC 27-1A for Normal Rotorcraft
• AC 29-2B for Transport Rotorcraft
• SAE ARP 4754 for all
• SAE ARP 4761 for all
Inventing the Operational Safety Assessment
45
What can we do with our specialized
guidance?
• We can pass the
vocabulary,
processes, and
methods on to the
community of
airspace planners,
developers, service
providers, and users
Inventing the Operational Safety Assessment
46
Guidance comparison
SSA:
• System Safety
Assessments
* AC 23.1309-1C
* AC 25.1309-1B
* AC 27-1A
* AC 29-2B
* SAE ARP 4754
* SAE ARP 4761
Inventing the Operational Safety Assessment
OSA:
• Operational
Safety
Assessment
* RTCA SC-189
ED-DO docs
–Guidance
–Methodology
–SPR
–FANS
47
SC-189/WG-53 Summary
Inventing the Operational Safety Assessment
48
Publication overview
ED/DOGUID
Crossregional/area
planning
ED/DOSPR
Objs/Rqmts
Homogeneou
s ATM Area
Planning
Objs/rqmts
ED/DOMETH
Inventing the Operational Safety Assessment
Implementation
•Aircraft certification
•ATS system commissioning
•ATS operational approval
•User operational approval
•Airspace approval
CNS/ATM
System,
Procedures,
& Airspace
Development
ED/DOINTEROP
•ARINC 622
Coordination
•ARINC 623
•ATN
•MIX
CNS/ATM
Service
Operation
49
Planning
ED/DO-GUID
Requirements
Determination
OED
OSA
Operational capability
Air traffic services
Functions
Tech
Choice
RCP
Allocated requirements
Interop
Objective: To agree
on approach
To establish
requirements
Activities:
Definition,
Assessment,
Allocation
Validation
Coordination
Aircraft
Ops App
Aircraft
Cert
ATS Prov
Sys App
Inventing the Operational Safety Assessment
ATS Prov
Ops App
Airspace
App
Evidence: Approval
plan(s)
Assessments
Requirements
Traceability
50
189/53 pub group
•
Objective: To produce first drafts of all publications with high
level of maturity on:
•
•
•
•
•
PUB-4 D: Outline for the RTCA / EUROCAE Documents
PUB-20 (METH) : Method for Operational environment description
and for evaluating operational environment
PUB-22 (GUID) : Guidelines for qualifications and operations of
advanced ATS
PUB-23 (INTEROP (622)) : A622 Interoperability Document
PUB-24 (SPR (Procedural Control Airspace)) : Characterization of
operational environment - safety and performance requirements
Inventing the Operational Safety Assessment
51
Oh boy!
189/53 pub group
Serge Bagieu
Tom Kraft
Wil Struck
Lionel Bertin
Kevin Grimm
Bob Granville
Steve Paasch
Gilles Surlaive
Roy Oishi
Gary Morton
Don Streeter
Chester Studzinski
Jim Coyne
Frank Cheshire
Mike Cuddy
189-53
189-53/CAG
189-53/CAG
SG1
SG1
SG2
SG2
SG3
SG3
CAG/SG2
CAG/SG1
CAG/SG2
CAG/SG2
PUB
PUB
Inventing the Operational Safety Assessment
A/C mfgr
A/C cert
A/C cert
A/C mfgr
ATS provider
ATS provider
A/C cert
ATS provider
Com provider
ATS regulator
A/C ops app
ATS regulator
A/C cert
A/C Oper
A/C Oper
France
USA
USA
France
USA
UK
USA
France
USA
UK
USA
Canada
Australia
USA
USA
52
Ugh!
189/53 pub group schedule
Cmts
D1 due
14 Jun
email/fax
Plenary
ACTION
Apr
May
1st mtg
19-23 Apr
Seattle
ED/DO
Draft 1
17 May
email/web
Jun
Cmts
D2 due
23 Aug
email/fax
Jul
2nd mtg
21-27 Jun
Canberra
Aug
3rd mtg
30 Aug3 Sep
Toulouse
ED/DO
Draft 2
12 Jul
email/web
Inventing the Operational Safety Assessment
Cmts
D3 due
11 Oct
email/fax
Sep
Oct
4th
18-24 Oct
Wash, DC
ED/DO
Draft 3
6 Sep
email/web
ED/DO
Final
25 Oct
email/web
53
Controller Pilot Data Link
Communications (CPDLC)
• Items being developed
• Operational Environment Description
(OED)
• Operational Hazard Assessment (OHA)
• Allocation of Safety Requirements (ASOR)
• Includes fault tree analysis and
requirements allocation matrix
• SPR (Safety & Performance Requirements)
Inventing the Operational Safety Assessment
54
Mode C Aircraft,
Fly ing Level Below
Transition Alt, Flies
Inc orrect Altitude
ALTS001
g=1x10E-07
n=Hazard Class 2 Ev ent Requires DAL B
Undetected Corruption
of Altimeter Setting,
Does Not Pass Pilot
Chec k
Undetected Corruption
of Altimeter Setting,
Pas ses Pilot Check
ALTS001A
ALTS001B
g=1x10E-07
n=DAL B Required
g=1x10E-07
n=DAL B Required - Mutually exc lus iv e from ALTS001A
Page 4
CPDLC Altimeter
Setting Transaction
is Corrupted
Pilot Unable to Chec k
Altimeter Setting
Against Those Brdcs t
to Other Aircraft
Mode C Surveillance
Fails to Detec t &
Warn About Altitude
Disc repancy
Flight Crew Fails to
Compare New Altimeter
Setting to Current
Altimeter Setting
ALTS002
ALTS003
ALTS004
ALTS005
g=1x10E-03
n=DAL E Allowed
Page 4
g=1x10E-05
n=DAL C Required
Page 2
g=1x10E-05
n=DAL C Required
g=1x10E-09
n=DAL A Credit is Tak en
Page 3
Pilot Fails to
Compare new Altimeter
Setting to Current
Alk imeter Setting
PILCOM****
p=
s=
Inventing the Operational Safety Assessment
55
Wrap Up and Questions?
Thank You!
Inventing the Operational Safety Assessment
56