Safety Critical Systems and Certification Issues DO-178B Airborne Standard

SC190 / WG-52 Application Guidelines For RTCA DO-178b/ED-12b

RTCA EUROCAE SC-167 WG-12 CNS/ATM Community Avionics Industry CAST (Certification Authority Software Team) Cast Position Papers 3 SC-190 / WG-52 SC-190 Products

DO-178B in a Nutshell

 Highly Process Oriented  Requires Traceability  Requirements  High Level Design  Detailed Design  Source Code  Test Procedures  Test results  Test & Test & Test & Test and …..

4

DO-178B Safety Levels

 Level A   Catastrophic Failure Prevents Continued Safe Flight and Landing  Level B   Hazardous/Severe-Major Potential Fatal Injuries to a Small Number of Occupants  Level C   Major Discomfort to Occupants or Possible Injuries  Level D  Minor 

Development Team

 What is COTS?

 How can objectives of DO-178B be satisfied, using COTS?

 There is much variation in applicants for COTS certification credit  Is DO-178B clear on the interpretations?

Product is COTS + Certification Evidence Available as COTS

Together these satisfy safety objectives

6

Testing without Source Code

Wrappers to Validate Parameters Application Commercial O/S This cannot be trusted unless O/S is verified 7

Special Considerations

Use of Service History for Certification

Air Traffic Control system  Worked Correctly in US for years  Transferred to U.K.

Actual Plane 2 Plane 1 Displayed Plane 2 8 Greenwich Meridian

Use of Service History

Developed under a less stringent standard (Military?) Safety Critical System Used for 4 years Problems tracked Quality Good!

Dead Code(Unintended Function) Residual Error Is this system safe for the next 4 years? At Level A, B, C?

We can bound inputs, but we cannot check internal states without “looking inside” 9

Black Box Testing

 No single failure should prevent “Continuous safe flight and landing.”  Statistical testing cannot show absence of a single state that will cause a failure  Software has discontinuities There is no foundation for statistical reasoning about software faults or safety  Software does not follow Gaus/Normal Distribution



Verification Team examples of issues

What are low level requirements? How can they be tested  Data and Control flow coupling  Use of higher level test results for lower level requirements  What is the intent of structural coverage?

 Traceability of source to object code for structural coverage  What is statement, decision, condition and MCDC coverage testing ( Modified Condition/Decision Code)  Verification tool qualification  etc..

11

Coverage Analysis at Level B and C

 Statement Coverage  Decision Coverage  Entry Points  Exit Points  All Decisions  All Outcomes 12

Fixing anomalies example for level B/Library

Filter Compiler B := 3; A := Filter (B); X := X + A; Object Code Source level coverage required Even for Filter 13

Boundary Level testing not enough!

A := B; -- A and B are packed Boolean arrays Run-time call to: Bit_Block_Move (From, To, Size); -- size in bits  Min, Mid, Max in combination gives 27 (useless) test cases  5 Bits size  32 Bit size  67 Bit Size Interesting test cases based on actual code i.e. White Box Testing  From overlaps To  To overlaps From 14

Coverage at Level A

 Coverage required at Machine Code level or  Show source to object code traceability  and test at source level or Use different language/compilers  and use voting system MCDC testing required  each condition must have effect on outcome  Tools which modify source for traceability problem at level A  Mitigation method : use 3 different compilers

Conditions/Decisions

Conditions if A=B and C or D<3 then Boolean Variable Decision 16 Boolean Operators

Ada : C : C : C++ :

What are Conditions/Decisions

if

(A=B

and

C)

or

D<3

then if

((A==B)

&

C )

|

(D<3)

then if

((A==B)

*

C) + (D<3)

then if

((A==B)

and

C)

or

(D<3)

then MCDC Coverage Requires all Branches AND all Conditions Be Covered

17

Ada : C :

More Boolean Conditions

X if

:= (A=B

and

C)

or

D<3; X

then

-- X is Boolean Cannot hide from Testing Obligations

X

= ((A==B)

*

C) + (D<3));

if

X

then

/* X can be any Integer ‘*’ and ‘+’ are Boolean Operators! 18

Ada :

Condition coverage

X

:= (A=B

and if

X

then

C)

or

D<3; -- X is Boolean Coverage of “Basic-Block” may not capture condition results 19

Avoiding MCDC Testing

Modified Condition/Decision Code Use Ada’s short-circuit conditions: if A=0

and then

B< 2

and then

C>5 then Or in C write: if A== 0 && B < 2 && C < 5 { 20

Why short-circuit conditions eliminate MCDC

if A=0 and then B< 2 and then C>5 then if A=0 then if B<2 then if C>5 then P; end if; end if; end if; MCDC not required for this code 21

Testing strategy must evaluate conditions

if A=0 and then B< 2 and then C>5 then if A=0 then if B<2 then if C>5 then P; end if; end if; end if; BUT !!!

Testing must show that each ‘then’ part has been tested True and False MCDC not required for this code 22

Inlining code

 If decisions/conditions introduced  Decisions must be identified and verified (level B)  Conditions must be identified and verified (level A)  Verification may be done by analysis  Traced to derived requirements  ensure safety is not compromised  Code may be “deactivated”  As inlined code depends on local state it may be very hard to test the conditions in accordance with standards requirements  Intent - absence of unintended funtion 

Use of Tools

 Tool Qualification is required if tool replaces a step of development process  Development tools  Examples - Compiler, Design to code generator  May introduce an error  In general - NOT qualified, not trusted  Verification tools  Examples - Coverage analyser  May conceal an error  May be qualified - Trusted for verification purposes  Additional verification process required if the tool is not trusted

CNS/ATM Process Integration

 Information matrix  Regulators  Committees  Standards Bodies  Standard  Software Integrity Assurance Standards vs. Software Development Standards  Relationships between DO-178B and IEC 61508 25

Ground Based Community

 Communications/ Navigation/Surveilance and Air Traffic Management - in the loop  Ground Based Systems affect airborne software  DO-178B addresses airborne only  Guidance being prepared to encompass needs of CNS/ATM community (SC-190 committee)  Standards tightening up 26

Certification Standards are Improving

 “Holes” in documents are being fixed  Understanding of Certification Requirements is spreading  Industry and Certification Authorities are collaborating on Guidance materials  It will get more difficult to “shop around” for a more lenient signature 27

( Legal) Safety Systems

Laws Regulations Standards Guidelines

PROCESS

Case Law Precedence Interpretations Standards Guidelines Visibility Traceability EVIDENCE / RECORD

Confidence / Safety

28

When Is Software Safe

29

When Is Software Safe

We Don’t Know !!

30

What is our best guess about the safety

 When applicable processes have been followed  When we have verified the code “from within”  When this has been checked and checked and checked and checked and checked and checked 31

The FAA/JAA Rules are Strict

To Date: “no fatalities have been attributed to Software Failure” Have we been lucky?

Have we been safe?

32