Transcript System Safety and Software Assurance

Considerations in the Preference
for and Application of
RTCA/DO-178B in the Australian
Military Avionics Context
SQNLDR Derek Reinhardt
Systems Certification and Integrity (SCI)
Directorate of Aircraft Engineering
(DAIRENG)
DGTA-ADF
Directorate General Technical Airworthiness
1
Overview
•
•
•
•
•
•
Introduction to criticisms of RTCA/DO-178B
Background and structure of DO-178B
Criticisms that RTCA/DO-178B is insufficient
Criticisms that RTCA/DO-178B is too onerous
ADF’s preference for RTCA/DO-178B
Application issues of RTCA/DO-178B in the
Australian Military Avionics Context
DGTA-ADF
Directorate General Technical Airworthiness
2
Introduction
• RTCA/DO-178B is the centre of much debate or
criticism – insufficient, too onerous, etc
• Avoidable software failures have already been
responsible for aircraft mishaps
– cockpit displays go blank
– engines throttle back during takeoff
– contradictory airspeed readings
• This presentation/paper
– examines these criticisms
– how do they influence the ADF’s preference for and
application of RTCA/DO-178B
DGTA-ADF
Directorate General Technical Airworthiness
3
ADF Preference for
RTCA/DO-178B
• RTCA/DO-178B is the ADF’s preferred software
assurance standard for safety critical and safety
related airborne software development
– Refer AAP7001.054 Airworthiness Design Requirements
Manual
• ADF recognises the FAA processes and standards
– widely used and accepted by many international
airworthiness authorities
• Many military aviation systems have a civil heritage
with software developed under RTCA/DO-178B
– AEW&C B737, MRTT A330, etc
DGTA-ADF
Directorate General Technical Airworthiness
4
Background of RTCA/DO-178B
• 14 CFR 25.1309
– often referred to as FAR 25.1309
– SAE ARP 4754 and SAE ARP 4761
• aircraft level FHA, FTA
• system level FMEA, FTA, etc.
• common cause analysis – PRA, ZSA, CMA
– RTCA/DO-178B for software design assurance
– RTCA/DO-254 for hardware design assurance
• Five failure condition severities are assigned design
assurance levels (DALs)
– Catastrophic (A), Hazardous / Severe Major (B), Major
(C), Minor (D), No Effect (E)
DGTA-ADF
Directorate General Technical Airworthiness
5
Structure of RTCA/DO-178B
• 66 objectives in 10 tables
– + several implicit objectives
– satisfaction tailored by software level
– several prescriptive objectives
• statement coverage, decisions coverage, modified condition
decision coverage
• Lifecycle phases
– planning, development, requirements, design, coding and
integration, verification
• Integral processes
– configuration management, quality assurance
• Certification Authority liaison
• RTCA/DO-248B, FAA Order 8110.49, CAST5
DGTA-ADF
Directorate General Technical Airworthiness
6
Intricacies of DO-178B
• Common misconception that RTCA/DO-178B
completely specifies the process and activities
• Yes – objectives are coupled to the software
lifecycle
• Yes – they don’t distinguish between requirements
– functional, software safety, etc
• Fidelity of the objectives presents challenges for
COTS and PDS
• With exception of 3 objectives – flexibility is
permitted in how the objectives are satisfied
• Objectives can be broadly classified as contributing
to requirements validity, requirements satisfaction,
and requirements traceability
DGTA-ADF
7
Directorate General Technical Airworthiness
Criticisms of RTCA/DO-178B
• Divided into several positions
– those that believe RTCA/DO-178B is insufficient
• academics, researchers, or consultants
– those that believe RTCA/DO-178B is too onerous
• development contractors with cost and schedule driven imperative
• Criticisms are at odds with each other
– central to the criticisms, then is it about right?
• Widely accepted by FAA, EASA
– NTSB reports that it is effective in those contexts
• How does the ADF address the criticisms?
DGTA-ADF
Directorate General Technical Airworthiness
8
RTCA/DO-178B is Insufficient
•
•
•
•
Absence of mandatory formal methods
Absence of mandatory static code analysis
Ineffectiveness of MC/DC testing
Absence of specific requirements or
objectives relating to software safety analysis
and software safety requirements
• Assumptions underlying the design
assurance level definition are questionable
DGTA-ADF
Directorate General Technical Airworthiness
9
Absence of Mandatory Formal
Methods
• Does not prohibit formal methods
– acceptable method to satisfy objectives
• Application to problem domain
– not universally applicable to problem domains and technologies used in
critical systems
– only partially applicable to problem scope
– are closed languages, no inherent real world meaning, natural language is
still required
• Formal methods and safety
– does not address significant sources of error WRT the safety of systems
– little evidence that it would prevent a number of mishaps attributable to
software
• Complementary to testing
– testing is required to demonstrate real world behaviours, on real hardware, in
the target environment
• Formal methods is not the ‘silver bullet’ for safety software
DGTA-ADF
Directorate General Technical Airworthiness
10
Absence of Mandatory Static
Code Analysis
• Does not prohibit static code analysis
– it is an acceptable method used to satisfy objectives
• Static analysis won’t find all the faults (requirements
and implementation) most related to the safety of
the software
• Permits greater focus on those activities related to
identifying requirements validity and satisfaction
problems affecting safety
– prevents developers being overwhelmed in code reviews
and testing identifying inadvertent implementation
problems, which static code analysis tools readily detect
• Limitations to applying static analysis retrospectively
DGTA-ADF
Directorate General Technical Airworthiness
11
Ineffectiveness of MC/DC Testing
• Exercise each data flow that directly affects a
control flow to identify as many faults as possible
• Widely debated objective
– studies confirm that MCDC does find faults that other DO178B testing approaches do not find
– other studies found “no significant difference”
• RCDC address some of these limitations
• MCDC not finding additional faults is not the
concern
– if normal and robustness testing has been
comprehensive, then it is possible that the gap in MCDC
will be small, and NOT safety related
– adequacy of normal and robustness testing
DGTA-ADF
Directorate General Technical Airworthiness
12
Absence of Specific Requirements or Objectives Relating
to Software Safety Analysis and Software Safety
Requirements
• Is not explicit in objectives relating to software
safety analysis and software safety requirements
– but they are not absent!
– number of objectives contribute to requirements validity,
including that of safety requirements
• However, systematic software safety analysis are
not always proposed to show that the identified and
allocated set of software safety requirements is
complete and correct
• DGTA recommends an IEEE1228 Software Safety
Plan be used to document the planned software
safety activities – and their outcomes
– or the RTCA/DO-178B PSAC can be used
DGTA-ADF
Directorate General Technical Airworthiness
13
Assumptions Underlying the Design Assurance
Level Definition are Questionable
• Issues with integrity/assurance levels
– little evidence that software of different integrity levels does have failure rates
of integrity level order
– debate regarding the philosophy and rules for allocating integrity levels
• significant differences in the processes recommended by standards
• Inconsistent application
– misunderstanding of the differences between reviews versus analyses
• some objectives being presumed to be satisfied solely by reviews
• intent is combination of analysis and reviews of the outputs
– variable
• normal and robustness testing
• software architecture is appropriate
– avoids design constructs that would not comply with system safety objectives
• software safety requirements are identified/allocated
• Apportionment and adequacy of objectives
– objectives fundamental to the validity and satisfaction – from Level C
– Level A and B provide additional evidence - trustworthiness
DGTA-ADF
Directorate General Technical Airworthiness
14
RTCA/DO-178B is Too Onerous
• Excessive requirements specification and
traceability fidelity requirements
• Excessive verification requirements
DGTA-ADF
Directorate General Technical Airworthiness
15
Excessive Requirements Specification and
Traceability Fidelity Requirements
• RTCA/DO-178B motivation for fidelity
– each behaviour that constitutes the requirements at level of
abstraction is systematically accounted for
– design tool to assist developer produce a good design
• Why should all the behaviours be accounted for?
–
–
–
–
–
evidence that all behaviours of the software are acceptable
do not lead to unacceptable failure conditions
all the behaviours of the software should be disclosed
permits reasoning about their suitability for the safety of the system
arguing non-interference with the behaviours that are important to the
safety of the system
• Questionable disagreements
– Intellectual Property constraints
– software development is a creative process, not itself compatible with
such rigour requirements
DGTA-ADF
Directorate General Technical Airworthiness
16
Excessive Verification
Requirements
• Testing will always be required to gain confidence in the behaviour of
software on the target hardware in the intended operating environment
• Completion of testing
– defensible engineering argument as to why testing is complete
– with evidence to support it
• Not based on the following factors:
–
–
–
–
cost and schedule
consensus of program managers
broad consensus of the programmers and testers
any other non-engineering based arguments
• RTCA/DO-178B provides a defensible argument as to when testing is
complete by specifying:
– requirements completeness criteria
– requirements coverage criteria
• extent of normal and robustness tests
• extensiveness of requirements based low level testing
– coupled with additional implementation related coverage criteria (structural
coverage) to elicit certain properties
DGTA-ADF
Directorate General Technical Airworthiness
17
Application of RTCA/DO-178B
• RTCA/DO-178B is not applied in isolation
• Test coverage objectives
• Use of RTCA/DO-178B as a benchmark for
assessing software assurance practices
• COTS with RTCA/DO-178B
• Migrating to RTCA/DO-178B
DGTA-ADF
Directorate General Technical Airworthiness
18
Not Applied in Isolation
• System Safety Program - MIL-STD-882C or FAR 25.1309
• Key Issues
– A Software Safety Program (SwSP) should be established to
coordinate hazard identification and mitigation efforts for hazards with
software-related causal factors.
• IEEE1228 Software Safety Plan
• Software Safety Analysis, Generic Software Safety Requirements
– A Software Assurance standard is required for the development of all
software that is safety related
– Software process standards should be applied to the development of
software for airborne and related systems
• An assessment of the applicant’s software development
capability, including domain knowledge, should be conducted
as part of the tender evaluation and contract negotiation
process
– examines the safety culture
– determine if non-systematic (experience) activities can be trusted
DGTA-ADF
Directorate General Technical Airworthiness
19
Test coverage objectives
• Some organisations believe that DGTA does not require
structural code analysis – by default – THIS IS NOT TRUE!
• In cases where DGTA has not required it
– additional activities to ensure that the coverage is comprehensive
•
•
•
•
fidelity of requirements
explicitness of traceability
extent of normal and robustness testing
additional activity to identify dead and deactivated code
– often exception related code
– mission systems, with no series hazards
– extenuating circumstances – legacy constraints
• Negotiate through the PSAC –expect to have a compelling
argument if you are going to proposed not doing it
DGTA-ADF
Directorate General Technical Airworthiness
20
Software Assurance Benchmark
• Assess software products and their development practices
– software agencies that do not use RTCA/DO-178B
– confused that DGTA is applying DO-178B where not contracted
• RTCA/DO-178B objectives help with the assessment of
– requirements validity, satisfaction, and traceability
• RTCA/DO-178B provide clear measures of
– fidelity in the specification and traceability of requirements
• no gap between required behaviours and executable code
• all software behaviours are appropriate with respect to safety
– extent of test based verification of requirements and implementation
on the target computer in the intended environment
– development and review rigour
• independence and oversight
• assures that the evidence presented contains an acceptably small
number of faults
DGTA-ADF
Directorate General Technical Airworthiness
21
COTS with RTCA/DO-178B
• COTS fall under the scope of PDS
• PDS criteria are demanding, but not
excessive
– but often the evidence is just not available
• Alternate proposals for use of COTS
• Provide evidence to demonstrate the
following types of properties
– Partitioning (containment and/or mediation)
– Non-Interference
– Acceptable Behaviours
DGTA-ADF
Directorate General Technical Airworthiness
22
Migrating to DO-178B
• Acquired where no software assurance standard
applied
– DOD-STD-2167A, MIL-STD-498, IEEE12207
• Two options – transition to RTCA/DO-178B or
develop a Software Assurance Task Matrix
• FAA guidelines
– Notice 8110.49 Chapter 10
– Intended for systems developed to RTCA/DO-178 and
RTCA/DO-178A
– Careful management of the scope of retrospective
production of software assurance evidence is required
– DGTA has assessed the approach as acceptable
DGTA-ADF
Directorate General Technical Airworthiness
23
Summary
•
•
•
•
•
•
Introduction to criticisms of RTCA/DO-178B
Background and Structure of DO-178B
Criticisms that RTCA/DO-178B is insufficient
Criticisms that RTCA/DO-178B is too onerous
ADF’s preference for RTCA/DO-178B
Application of RTCA/DO-178B in the
Australian Military Avionics Context
– RTCA/DO-178B applied within the ADF
framework addresses relevant criteria for
producing safety software systems
DGTA-ADF
Directorate General Technical Airworthiness
24
Questions
DGTA-ADF
Directorate General Technical Airworthiness
25