Transcript Introduction to Airworthiness
Migrating to a Software Assurance Standard
2008 ADF Software Symposium FLTLT Patrick Redmond SCI-DGTA
DGTA-ADF
•
Overview
The Problem with Legacy Software
•
Migrating to a Software Assurance Standard
•
The Potential Challenges
Bounding the Change Low-Level Requirements Traceability Structural Coverage •
Case Study
MIL-STD-498 to DO-178B Considered throughout.
DGTA-ADF
The Problem with Legacy Software
• Numerous ADF platforms acquired where no software assurance standard has been explicitly applied.
• Only MIL-STD-498 or DOD-STD-2167A.
• Development standards do not define how well software has to be constructed.
• During development, somebody made the decision to stop testing.
Why?
• During development, somebody made the decision to review source code.
Why?
• To what extent should the ADF rely on this software?
• To what extent does the ADF rely on this software?
DGTA-ADF
• • • • •
Migrating to a Software Assurance Standard
DGTA requires that a software assurance standard be applied to legacy software systems.
All development methods apply a level of assurance, but is that level acceptable?
Applying a software assurance standard to each modification of in-service software progressively increases the integrity.
Applies ‘Cancer Theory’ to Legacy Software Two methods:
Service History Argument and Application of Recognised Standard e.g. RTCA DO-178B Software Assurance Task Matrix Negotiate a custom software assurance “standard” with DGTA
DGTA-ADF
How does migration work?
DGTA considers the software to be compliant with the applied software assurance standard from this point forward.
?
DGTA-ADF
?
?
?
?
?
?
?
Unknown Integrity ?
?
?
?
?
Build x ?
?
?
Build x ?
?
?
• • • •
Service History Argument / Recognised Standard
Establish a Service History Argument that demonstrates that the current software is acceptably safe.
Need to consider, among others: Configuration Management Problem Reporting Change Control Relevance of Product Service History Operating Environments Safety Related Problems Design and Code Errors Error Rates
No need to create a service history argument for in-service software, ADF has already determined that it is acceptably safe.
Only applies to in-service, legacy software. Does not apply to acquisitions.
Does not apply if there is a substantial change of context in which the software is used.
Apply a recognised software assurance standard to the next modification.
DGTA then consider the software to be compliant with that standard.
DGTA-ADF
Software Assurance Task Matrix
Source code is complies with coverage.
Design OFP B
Negotiate with SCI
Code Test
Decision plus untested code coverage.
OFP C OFP D
DGTA-ADF
•
The Potential Challenges
Bounding the Change
How much “re-assurance” needs to be done?
•
Traceability
•
Low-Level Requirements
•
Structural Coverage
DGTA-ADF
•
The Easy Ones
Planning
•
Additional Considerations
Tool Qualification •
Development of High-Level Requirements, Source Code and Executable Object Code
•
Verification of High-Level Requirements
•
Configuration Management
•
Quality Assurance
DGTA-ADF
• • •
Bounding the Change
Software assurance standard only needs to be applied to the scope of the modification.
What is the scope of the modification?
The things that are changed, and The things affected by the things that are changed.
Change Impact Analysis:
Traceability Memory Margin Timing Margin Data Flow Control Flow Input/Output Development Environment and Process Operational Characteristics Certification Maintenance Requirements Partitioning Analysis
DGTA-ADF
Traceability
• Full traceability data was not recorded for many legacy software systems.
• Software assurance standards generally require traceability to source code.
• When applying a software assurance standard to a legacy software system, how much traceability data needs to be generated?
• DGTA Position: Trace down and up once.
• Trace down from new or modified high-level requirements to affected and new low-level requirements, to affected and new code.
• Trace up from affected and new code to low-level requirements and to high-level requirements.
DGTA-ADF
•
Example – MIL-STD-498 to DO-178B
This example will be considered a number of times.
•
A legacy software system developed to MIL-STD-498 with typical artefacts.
•
Can cause or contribute to Major hazards.
•
Migrating to DO-178B Level C objectives.
DGTA-ADF
HLR1 LLR1 LLR2 LLR3
Traceability
HLR1 LLR1 LLR2 LLR2 LLR3 SCM1 SCM2 SCM3 SCM4 SCM1 SCM2 SCM3 SCM4 HLR1 LLR1 LLR2 LLR3 HLR1 LLR1 LLR2 LLR3 LLR1 HLR1 LLR2 LLR3 HLR1 LLR1 LLR2 LLR3 HLR1 LLR1 LLR2 LLR3 SCM1 SCM2 SCM3 SCM4 HLR1 SCM1 SCM2 SCM3 SCM4 HLR1 LLR1 LLR2 LLR3 LLR1 LLR2 LLR3 SCM1 SCM2 SCM3 SCM4 HLR1 SCM1 SCM2 SCM3 SCM4 HLR1 SCM1 SCM2 SCM3 SCM4 HLR1 LLR1 LLR2 LLR3 LLR1 LLR2 LLR3 LLR1 LLR2 LLR3 LLR1 HLR1 LLR2 LLR3 LLR1 LLR2 LLR3 HLR1 LLR1 LLR2 LLR3 SCM1 SCM2 SCM3 SCM4 HLR1 LLR1 LLR2 LLR3 LLR1 HLR1 LLR2 LLR3 HLR1 LLR1 LLR2 HLR1 LLR1 LLR2 LLR3 SCM1 SCM2 SCM3 SCM4 LLR1 HLR1 LLR2 LLR3 SCM1 SCM2 SCM3 SCM4 SCM1 SCM2 SCM3 SCM4 SCM1 SCM2 SCM3 SCM4 LLR1 HLR1 LLR2 LLR3 SCM1 SCM2 SCM3 SCM4 SCM1 SCM2 SCM3 SCM4 LLR1 HLR1 LLR3 LLR1 HLR1 LLR2 LLR3 HLR1 LLR1 LLR2 LLR1 HLR1 LLR2 LLR3 LLR1 HLR1 LLR2 LLR1 HLR1 LLR2 SCM1 SCM2 SCM3 SCM4 Requirement HLR1 LLR3 LLR1 LLR2 LLR3 SCM1 SCM2 SCM3 SCM4 HLR1 LLR2 LLR3 SCM1 SCM2 SCM3 SCM4 SCM1 SCM2 SCM3 SCM4 Source Code HLR1 HLR1 HLR1 LLR1 LLR2 LLR3 LLR2 LLR3 LLR2 LLR3 SCM1 SCM2 SCM3 SCM4 SCM1 SCM2 SCM3 SCM4
DGTA-ADF
SCM1 SCM2 SCM3 SCM4 SCM1 SCM2 SCM3 SCM4 SCM1 SCM2 SCM3 SCM4 SCM1 SCM2 SCM3 SCM4
Low-Level Requirements
• Some legacy systems have no low-level requirements.
• Others have design descriptions that are not refined enough to be low-level requirements.
• Others have design descriptions that do not drive source code development (source code is developed from requirements).
• How many low-level requirements need to be defined or redefined?
• DGTA Position: Each affected low-level requirement and each low-level requirement identified by the down and up trace.
DGTA-ADF
Low-Level Requirements
HLR1 LLR2 SCM1
DGTA-ADF
•
Low-Level Requirements
Verify that low-level requirements LLR1, LLR3 and LLR4:
Comply with high-level requirements.
Are accurate and consistent.
Conform to standards.
Are traceable to high-level requirements.
•
Ensure that low-level requirements LLR1, LLR3 and LLR4 are sufficiently refined to be directly translatable to source code.
May need to further refine one low-level requirement into several.
•
Leave LLR2 as is.
DGTA-ADF
• • • •
Structural Coverage
Software assurance standards generally require assessment of structural coverage in order to demonstrate that testing is complete.
Purpose of Structural Coverage Objectives is to:
Identify shortcoming in requirements based test cases or procedures, Identify inadequacies in software requirements, Identify dead code, and Identify deactivated code.
How do these measures apply to modification of legacy software systems?
DGTA Position:
For requirements based measures: each new, modified or affected requirement needs to be tested.
For structure based measures: each new or modified source code module.
DGTA-ADF
Structural Coverage
Normal Range and Robustness Tests LLR2 Normal Range and Robustness Tests Statement Coverage SCM1
DGTA-ADF
• •
Structural Coverage
Normal Range and Robustness Tests
High-level requirements that have been added or modified.
High-level requirements where a dependent low level requirement has been added or modified.
High-level requirements where the implementation (source code) of a dependent low-level requirement has been added or modified.
Low-level requirements that have been added or modified.
Low-level requirements where the implementation (source code) has been added or modified.
Structural Coverage Objectives:
Structural coverage of all new or modified source code modules.
DGTA-ADF
•
Additional Testing
Also need to consider:
Data Dependencies Has the change impacted data that other functions rely upon?
Control Flow Dependencies Has the change inadvertently/adversely disrupted control flow or coupling?
Timing Dependencies Has the change violated a timing constraint?
Memory Space Dependencies Has the change violated memory constraints or used memory space assigned to other functions?
DGTA-ADF
•
What about bug fixes?
A large part of in-service support is the rectification of bugs.
•
Bug fixes may not commence at the requirements level, may start with an identified fault in the source code.
•
To what extent should software assurance standards be applied to bug fixes?
•
DGTA Position: Trace up from the modified source code.
DGTA-ADF
What about bug fixes?
LLR6 Low-Level Requirements Normal Range and Robustness Tests LLR8 Additional traceability data Structural Coverage SCM6 SCM7
DGTA-ADF
SCM9 SCM10
• • • • • •
Conclusions
DGTA requires the application of a software assurance standard to legacy software systems.
All development methods provide a level of assurance, writing it down can identify gaps.
A software assurance standard can be applied either by:
A Product Service History Argument and application of a recognised software assurance standard, or A Software Assurance Task Matrix.
The software assurance standard need only be applied to the current modification.
Determine extent through change impact analysis.
For legacy software systems, a large number of assurance objectives are probably already being achieved.
Meeting a recognised software assurance standard will probably require additional effort in the areas of:
Traceability, Low-Level Requirements, and Structural Coverage