Transcript Click to Edit Master Title

Superior Products Through Innovation
Validation & Verification of
Intelligent and Adaptive
Control Systems (VVIACS)
Jim Buffington
Lockheed Martin Aeronautics Company
Outline
• Introduction
• Motivation
• Approach
• Assessment
• Development
• Evaluation
• Summary
Lockheed Martin Aeronautics Company
2
Team
Vince Crum – AFRL - Government PM
Jim Buffington – LM Aero - Contractor PM
LM Aero – Greg Tallant, Peter Stanfill
LM M&FC - Clinton Plaisted, Barry Frazier, Rich Hull, Guy Rowlands
LM SS - Prasanta Bose
Carnegie Mellon University - Bruce Krogh
General Electric Global Research - Tim Johnson, Hunt Sutherland
Scientific Systems Company, Inc - Ravi Prasanth, Sanjeev Seereeram
Lockheed Martin Aeronautics Company
3
Scope
•Safety-Critical Flight Systems
•Military Certification
•Advanced Controls
•System Development
•All Development Phases
•Process, Tools, and Methods
•Verification & Validation
Lockheed Martin Aeronautics Company
4
Emerging Software Size and Complexity
 Advanced system attributes (on-board
intelligence and adaptive control laws) will
be required to accommodate emerging
functional requirements.
 This will increase the size and complexity
of control systems beyond the capability
of current V&V practices.
Inter-System
Communication
& Dependencies
UAV - Projected
F-35 – Est
F-22
F-16 IDF YF-22
JSF CDA
• Increasing system
integration
requirements and
Complexities
Block 60
Projected Exponential Increase in SW Size and Complexity
Lockheed Martin Aeronautics Company
5
Emerging System Costs
 SW size and system complexity
lead to growth system
development and certification
costs
 Test automation insufficient for
emerging control systems
Size and Complexity Increase Costs
Lockheed Martin Aeronautics Company
6
Compressed Development Schedules
Time to First Flight
60
Months
50
40
30
20
10
0
AC1
AC2
AC3
AC4 AC5: AC5: Future
(1990) (1996) (2000) (2002) plan actual UAV
(2001) (2002) (??)
Vehicle (Year)
• Customers requiring 12 to 36 month development schedules
• Baseline is 48 months to first flight
Emerging Program Requirements Increase Risk
Lockheed Martin Aeronautics Company
7
Purpose
GOAL:
Enable affordable development of future safety-critical flight systems with
prescribed levels of safety and reliability.
OBJECTIVE:
Study, develop, and demonstrate effective V&V technologies for advanced
safety-critical control system flight certification.
• Classify emerging safety-critical control systems according to fundamental
attributes
• Develop and demonstrate preliminary V&V strategies that focus on critical
flight certification schedule and cost points
• Identify high-payoff V&V process, tool, and method technologies for further
development
APPROACH:
• Use extensive experience-base and diverse team to develop innovative
concepts
• Evaluate concepts in a realistic framework to maximize transition success
Lockheed Martin Aeronautics Company
8
Tasks
 Task 1
 Emerging Control System Study
1. Assessment  Task 2
 Control Characteristics and V&V
Needs Study
 Task 3
2. Development
 Innovative Flight Certification
Strategies Development
 Task 4
 Proof of Concept
3. Evaluation
 Task 5
 Technology Development
Planning and Reporting
Lockheed Martin Aeronautics Company
9
Tasks 1 & 2 – Assessment
 Emerging Control Systems
Study
 Develop control system
project database
 Down select to subset of
projects for additional
analysis
 Control Characteristics and
V&V Needs Study
 Analysis to define emerging
fundamental properties
 Identification of V&V
drivers
 System Development Model
 System Development
impact assessment
% Development Cost By Functional Discipline
SYS
OTHER
S&C
HW
CLAW
HWPA
SW
TEST
TTD
SIM
V&V NEEDS
Lockheed Martin Aeronautics Company
10
Task 1 – Emerging Control Systems
ECS PROJECT
DESCRIPTION
AIMSAFE / RESTORE
Integrated Management, Adaptive Control
ICARUS
Intelligent Autonomy
PCA
Polymorphous Computing Architectures
LOCAAS
Autonomous Control
Swarm Intelligence
Bio-inspired Multi-vehicle Control
XACT
Adaptive Failure Management
Software Enabled Control
Optimal Trajectory Generation
Autonomous Propulsion System Tech
Intelligent Failure Management
Intelligent Engine PHM
Model-based Health Management
Distributed Space Systems
Distributed Multi-vehicle Control
Characterized 10 Emerging Control System (ECS) projects
• 6 ECS projects used to form the Single-Vehicle ECS (AIMSAFE/RESTORE, PCA,
LOCAAS, XACT, Autonomous Propulsion System, and Intelligent Engine PHM)
• 5 ECS projects used to form the Multi-Vehicle ECS (ICARUS, PCA, Swarm
Intelligence, Software Enabled Control, and Distributed Space Systems)
Lockheed Martin Aeronautics Company
11
Task 2 - Emerging Systems Analysis
•Identified 100+ fundamental properties (FPs) for the ECS projects
•Categorized FPs in 5 Views:
• Requirements, System, Algorithm, SW, HW
•Identified 28 FPs as Emerging Fundamental Properties (EFPs)
•Identified 4 primary V&V Drivers
• Difficulty, Complexity, Unfamiliarity, External factors
•System development model
• Based on LM Process
Weights
Process
• Multiple levels
• Contains over 800 tasks
• MS Project / Excel
System Design - 753.1
Task
System Operational Requirements
SSS & SSDD Documentation
Update SSDD Documentation
Update SSDD Documentation
Update SSDD Documentation
Update SSDD Documentation
Schedule
Task Duration
Start Day
(Days)
Duration
Cost
1.00
1.15
1.00
1.00
1.00
1.00
1.00
1.15
1.00
1.00
1.00
1.00
30
86
5
5
5
5
1.00
30
2.00
1.00
3.00
4.50
1.15
1.50
1.00
2.00
1.00
3.00
4.50
1.25
2.00
3.00
4.50
1.25
1.25
Martin
2.00
15
75
225
300
29
38
15
8
75
150
300
88
8
150
300
38
50
Aeronautics
8
Input to Software Development Plan System/Software Development Plan
1.00
753.2
Flight Control Software Release 1 (FCSW01) Requirements Definition
1.50
FCSW01 Operating System Software
1.00
FCSW01 Flight Control Application Software
3.00
FCSW01 System Integrity Management Software
3.00
FCSW01 OS/FCA Integration & Validation
1.15
FCSW01 IM Integration & Validation
1.50
FCSW01 Documentation
1.00
Flight Control Software Release 2 (FCSW02) Requirements Update
1.50
FCSW02 Operating System Software
1.00
FCSW02 Flight Control Application Software
3.00
FCSW02 System Integrity Management Software
3.00
Flight Software Requirements 753.3
FCSW02 Integration, OFP01 SPARS, Validation, & Documentation
1.25
Flight Control Software Release 3 (FCSW03) Requirements Update
1.50
FCSW03 Flight Control Application Software
3.00
FCSW03 System Integrity Management Software
3.00
FCSW03 Incorporate OFP02 SPARS
1.25
FCSW03 Integration, Validation, & Documentation
1.25
Lockheed
Flight Control Software Release 4 (FCSW04) Requirements Update
1.50
End Day
1
31
712
1113
1476
1910
30
116
716
1117
1480
1914
31
60
344
358
359
433
359
583
359
658
584
612
659
696
697
711
717
724
725
799
725
874
725
1024
1025
1112
1118
1125
1126
1275
1126
1425
1168
1205
1426
1475
Company
1777
1784
Cost
Category
Total Cost
($K)
V&V Cost
($K)
SYS
SYS
SYS
SYS
SYS
SYS
288.00
828.00
48.00
48.00
48.00
48.00
0.00
24.84
0.48
0.48
0.48
0.48
OTHER
248.28
12.41
SW
SW
SW
SW
SW
SW
SW
SW
SW
SW
SW
SW
SW
SW
SW
SW
SW
SW
150.47
564.26
1692.79
3385.58
216.30
282.13
112.85
75.24
564.26
1128.53
3385.58
658.31
75.24
1128.53
3385.58
282.13
376.18
75.24
75.24
310.34
931.03
1862.07
205.49
268.03
0.00
37.62
310.34
620.69
1862.07
625.39
37.62
620.69
1862.07
155.17
357.37
12
37.62
Extensible Analysis Framework based on Fundamental Properties
Task 2 - Impact Analysis Results
ECS Impact on System Development Cost by Functional Discipline
ECS Impact on Critical Path
TEST
Development Cost
HW
TTD
CLAW
HWPA
SIM
S&C
Critcal Path Length
SW
OTHER
SYS
Baseline
Single-Vehicle
Emerging Control System
Multi-Vehicle
Baseline
Single-Vehicle
Multi-Vehicle
Emerging Control System
• Single-Vehicle ECS Increases Development Costs ~ 50%, V&V Costs ~ 100%,
and Critical Path Length ~ 50%
• Multiple-Vehicle ECS Increases Development Costs ~ 100%, V&V Costs ~ 150%,
and Critical Path Length ~ 125%
• Software: Single-Vehicle 100% Increase and Multiple-Vehicle 200% Increase in
V&V Costs
• Test: Single-Vehicle 150% Increase and Multiple-Vehicle 250% Increase in V&V
Costs
Significant Cost/Schedule Increase Projected Due to Complexity
Lockheed Martin Aeronautics Company
13
Task 2 - Pareto Analysis
Functional Discipline Impact on V&V Cost: Single-Vehicle ECS
100%
 Single-Vehicle ECS V&V Cost
Impact
40%
20%
0%
SYS
OTHER
S&C
HWPA
CLAW
SIM
TTD
SW
Functional Discipline
Functional Discipline Impact on V&V Duration: Single-Vehicle ECS
 Single-Vehicle ECS V&V Duration
Impact
100%
80%
Percentage of Total
 80%: SW, TEST
 90%: 80% + TTD
 SW dominates at 53%
60%
TEST
 80%: TEST, SW
 90%: 80% + TTD
 TEST largest component (41%)
followed by SW (28%)
Percentage of Total
80%
60%
40%
20%
0%
SYS
OTHER
SIM
S&C
CLAW
HWPA
TTD
TEST
SW
Functional Discipline
Pareto Analysis Identifies the Critical Functional Disciplines
Lockheed Martin Aeronautics Company
14
Task 3 – Flight Certification Strategies
Development
V&V impact drivers are critical drivers
 Requirements Development
 Requirements/Guidelines
 V&V Drivers Mapping
 Flight Certification Refinement
 Current Process Development
 Evolution of Process Development
for V&V
Defects
Require
Rework
Requirements (Textual)
Design
OLD
Build
Test Planning and Development
Test Execution
Fewer Defects
Requirements Modeling
 V&V Methods Development
 Identify Current V&V Methods
 Recommend Approaches
Less Rework
Verification Modeling
Design & Analysis
NEW
Lower Risk
Test Generation & Analysis
Implementation & Integration
Shorter
Development
Schedule
Test Execution & Analysis
Highly Integrated Verification
V&V Test
Aero Data
Model Based
System Synthesis
Design
Req’mts
System Development And Verification
Aero Data
Linear
Models
Non-Realtime Simulation
A B
C D
Analysis tools
Vehicle
Integration
Facility (VIF)
• Trim
• Linearize
• Simulate
• CPR/CPA
Autocode
Model
Simulation
CLAW
Realtime Simulation
Embedded
Software
Product
O/S
Prototyping
Tools
Process
V&V STRATEGY EVOLUTIONS
Te s t - A - L
Aero
Stability & Control
CL
CM
Validation/
Verification
Testing
Algorithm
Design/Analysis
Model
Requirements
Aero
CD
it t l e
Implementation
Data
Bu i
ld - A
- L i t t le
Lockheed Martin Aeronautics Company
15
Task 3 – Strategy Evolutions
Near-Term (1-3 yrs) Evolution: System Model-based design now being
implemented
 Auto-Code
 Auto-Test
 Rapid Prototyping
 System Model-Based
 Automated Verification Management
 Simulation-Based design
Mid-Term (4-6 yrs) Evolution: Formal Foundations in advanced development
• Formal Requirements Specs
• Requirements and Traceability Analysis
• Formal Methods
• Computer-Aided System Engineering
Far-Term (7-9 yrs) Evolution: V&V Awareness throughout  still in research
• Run-Time V&V
• Rigorous Analysis for Test Reduction
• Requirements & Design Abstraction
• Probabilistic/Statistical Test
• Testing Metrics
Lockheed Martin Aeronautics Company
16
Task 4 – Advanced Technology Impact
to System Development
Advanced Tools, Methods, & Processes Impact on System Development Cost
 System Development Cost
Reductions:
Current
Cost
 Baseline: 25%
 Single-Vehicle: 33%
 Multi-Vehicle: 35%
Advanced
Baseline
Single-Vehicle
Multi-Vehicle
Emerging Control System
Advanced Tools, Methods, & Processes Impact on Crtical Path
 Baseline: 12%
 Single-Vehicle: 29%
 Multi-Vehicle: 30%
Critcal Path Length
 System Development Critical
Path Reductions:
Current
Advanced
Baseline
Single-Vehicle
Multi-Vehicle
Emerging Control System
Lockheed Martin Aeronautics Company
17
Task 5 – Technology Development
Planning and Reporting
 Risk Waterfall Planning  ROM Development Cost
 EFP Coverage Analysis + Task 4 System Development Impact 
Net Benefit
 Priority of V&V technologies was established on the basis of a
cost-benefit analysis performed on each technology
 Cost-Benefit Ratio (CBR) = ROM Development Cost/Net Benefit
 All near-term technologies except Automated Verification
Management were eliminated from the CBR analysis because
these technologies are relatively mature (moderate technical risk)
and significant industry investment in these technologies is
ongoing and is expected to continue.
Lockheed Martin Aeronautics Company
18
Task 5 – Technology Cost / Benefit
 ROM Cost – Risk Waterfall Analysis
Total ROM Cost: $M
TRL
1
• Task 1
• Task 2
• Task 3
$M
High
Technical Risk
2
$M
$M
3
Mod.
• Task 1
• Task 2
• Task 3
$M
4
• Task 1
• Task 2
• Task 3
5
Low
• Task 1
• Task 2
• Task 3
$M
• Task 1
• Task 2
• Task 3
6
Near-Term (1-3 years)
Mid-Term (4-6 years)
Far-Term (7-9 years)
Development Year
 Net Benefit is defined as the product of the technology’s EFP Coverage
Benefit and the System Development Benefit
 BNET = BEFPBSD
 EFP Coverage Benefit is a measure of the technology’s applicability across
the set of EFPs (large BEFP implies good coverage)
 System Development Benefit is a measure of the technology’s impact on
system development cost and schedule
Lockheed Martin Aeronautics Company
19
Task 5 – Final Prioritized Technologies
Based on Cost-Benefit Analysis
1. Automated Verification Management
2. Formal Requirements Specifications
3. Requirements and Traceability Analysis
4. Formal Methods
5. Probabilistic / Statistical Test
6. Requirements and Design Abstraction
7. Run-Time V&V
8. Testing Metrics
9. Rigorous Analysis for Test Reduction
10. Computer-Aided System Engineering
Increasing
Cost-Benefit
Ratio
• EFP weighting:
• WEFP = 1 (all equal)
• ECS weighting:
• 10% Baseline
• 50% Single-vehicle
• 40% Multi-vehicle
Near-Term (1-3 years) Technology
Mid-Term (4-6 years) Technology
Far-Term (7-9 years) Technology
Lockheed Martin Aeronautics Company
20
Summary
 Developed emerging control system (ECS) database and populated with an extensive
set of past, present, and future ECS projects.
 Assessed impact of future systems on current development process and identified
“long-pole” functional disciplines with highest adverse impact (i.e., SW, TEST, TTD).
 Identified 15 key V&V technologies to address system development impact and
characterized in terms of near- (1-3 years), mid- (4-6 years), and far-term (7-9 years)
strategy evolutions.
 Demonstrated effectiveness of each strategy evolution using the system development
model and the system development impact assessment tool (i.e., cumulative 25%
reduction in V&V cost and 12% reduction in V&V effort for all strategies).
 Developed technology maturation plans for each V&V technology identified, and
prioritized the technologies by performing a cost-benefit analysis.
Lockheed Martin Aeronautics Company
21