Transcript Document

Software Quality Assurance
Dr. Linda H. Rosenberg
Assistant Director For Information Sciences
Goddard Space Flight Center, NASA
301-286-5710
[email protected]
V&V 10/2002
Slide 1
Agenda
•
•
•
•
•
Introduction
Defining Software Quality Assurance
Quality Assurance and Software Development
IV&V within SQA
Summary
QA
V&V 10/2002
Slide 2
Introduction
V&V 10/2002
Slide 3
“Traditional” Development
TIME
REQUIREMENTS
TESTING
V&V 10/2002
Q
U
A
L
I
T
Y
Slide 4
Results in
V&V 10/2002
Slide 5
Quality Assurance
V&V 10/2002
Slide 6
Why SOFTWARE Assurance
Software
Hardware
time
cost
Software
Hardware
time
V&V 10/2002
Slide 7
Software Quality Assurance
IEEE 12207 - Standard for Information Technology Software Life Cycle Processes
“The Quality assurance process is a process for providing
adequate assurance that the software products and
processes in the project life cycle conform to their
specified requirements and adhere to their established
plans. “
IEEE 730 - Quality Assurance Plans
“Quality Assurance - a planned and systematic pattern of all
actions necessary to provide adequate confidence that the
time or product conforms to established technical
requirements.”
V&V 10/2002
Slide 8
Quality Attributes
Maintainability - Can I fix it?
Flexibility - Can I change it?
Testability - Can I test it?
Product Product
Revision Transition
Portability - Will I be able to use
on another machine?
Reusability - Will I be able to
reuse some of the software?
Interoperability - Will I be able to
interface it with another machine?
Product
Operations
Correctness - Does it do what I want?
Reliability - Does it do it accurately all the time?
Efficiency - Will it run on my machine as well as it can?
Integrity - Is it secure?
Usability - Can I run it?
V&V 10/2002
Slide 9
SQA Life CYCLE
Concept/ Requirements
Reviews (SCR. SRR)
Requirement trace
SW Development Plans
Define success criteria
Prototyping
Metrics
Safety Considerations
IV&V
Design
Reviews (PDR, CDR )
Requirement trace
Support tools
Metrics
Safety Considerations
IV&V
Deployment
Devel. & Coding
Walkthrough and reviews
Requirement trace
SW Devel. Folders
Capture deficiencies
Metrics
Safety Considerations
IV&V
Test
Witnessing
Requirement trace
Monitoring
Reliability metrics
Metrics
Safety Considerations
IV&V
Capture anomalies
Report trending
Sustaining engineering
Metrics
Safety Considerations
IV&V
V&V 10/2002
Slide 10
SQA Across the Life Cycle
Concept/ Requirements
Design
Devel. & Coding
Test
Deployment
IV&V
V&V 10/2002
Safety
Reliability
Metrics
Risk Management
Slide 11
Why IV&V at NASA
MARS
V&V 10/2002
Slide 12
Independent Verification &
Validation
Software IV&V is a systems engineering process employing
rigorous methodologies for evaluating the correctness and quality
of the software product throughout the software life cycle
Independent
– Technical: IV&V prioritizes its own efforts
– Managerial: Independent reporting route to Program Management
– Financial: Budget is allocated by program and controlled at high level such
that IV&V effectiveness is not compromised
Verification (Are we building the product right?)
Validation (Are we building the right product?)
V&V 10/2002
Slide 13
IV&V Approach
Traditional Software Development
Req
Req
Design
Design
Code
Code
Test (Verification & Validation)
Unit
Integration
Acceptance
Testing
Unit
V&V
Clean Room Approach
iV&V
Test (Verification & Validation)
Integration
Acceptance
Req
Design
Code
Test (Verification & Validation)
Unit
Integration
Acceptance
IV&V
IV&V Implementation
V&V 10/2002
Slide 14
IV&V Activities
Requirements Phase
•System Reqts
Analysis
•S/W Reqts
Analysis
•Interface Analysis
•Process Analysis
•Technical
Reviews & Audits
Design Phase
•Design Analysis
•Interface Analysis
•Test Program
Analysis
•Supportability
Analysis
•Process Analysis
•Technical
Reviews & Audits
Code Phase
•Code Analysis
•Test Program
Analysis
•Supportability
Analysis
•Process Analysis
•Technical
Reviews & Audits
Test Phase
•Test Program
Analysis
•Independent Test
•Supportability
Analysis
•Technical
Reviews & Audits
Verify
Verify
Validate
Verify
Catastrophic/Critical/High Risk Functions List
Traceability Analysis
Issues Tracking
Metrics Assessment
Loading Analysis
Change Impact Analysis
Special Studies
V&V 10/2002
Slide 15
Implementing IV&V at NASA
V&V 10/2002
Slide 16
IV&V Criteria
IV&V is intended to mitigate risk
Probability
of an undesired event
Consequences
if that event should occur
Risk = Probability * Consequence
 IV&V must be based on Risk Probability & Consequence
V&V 10/2002
Slide 17
IV&V Probability Risk Factors
Factors that impact the difficulty of the development
–
–
–
–
–
–
–
–
–
V&V 10/2002
Software Team Complexity
Contractor Support
Organization Complexity
Schedule Pressure
Process Maturity of Software Provider
Degree of Innovation
Level of Integration
Requirement Maturity
Software Lines of Code
Slide 18
IV&V Probability Risk Factors
Factors
Un-weighted probability of failure score
contributing
to probability
of software
failure
Software
team
complexity
Contractor
Support
1
2
4
Up to 5 people Up to 10
Up to 20
at one location people at one people at one
location
location or 10
people with
external
support
None
Contractor with
minor tasks
Organization One location
Complexity*
No deadline
Process
Maturity of
Software
Provider
Independent
Independent
Independent
assessment of assessment of assessment of
Capability
CMM Level 3 CMM Level 2
Maturity Model
(CMM) Level
4, 5
Proven and
Proven but
accepted
new to the
development
organization
Simple - Stand
alone
Level of
Integration
Less than 50K
8
Up to 50
people at one
location or 20
people with
external
support
Contractor with
major tasks
Deadline is
negotiable
Requirement Well defined
Well defined
Maturity
objectives - No objectives unknowns
Few unknowns
Software
Lines of
Code***
Likelyhood of
failure
rating
16
More than 50 X2
people at one
location or 20
people with
external
support
Contractor with X2
major tasks
critical to
project
success
Two locations Multiple
Multiple
Multiple
X1
but same
locations but
providers with providers with
reporting chain same reporting prime sub
associate
chain
relationship
relationship
Schedule
Pressure**
Degree of
Innovation
Weighting
Factor
Non-negotiable X2
deadline
CMM Level 1
with record of
repeated
mission
success
Preliminary
objectives
Over 500K
CMM Level 1
or equivalent
X2
Cutting edge
X1
Extensive
X2
Integration
Required
Changing,
X2
ambiguous, or
untestable
objectives
Over 1000K
X2
Total
V&V 10/2002
Table 1 Likelihood of Failures Based on Software Environment
Slide 19
Consequence Factors
•
•
•
•
•
•
•
•
V&V 10/2002
GRAVE
SUBSTANTIAL
MARGINAL
INSIGNIFICANT
Potential for loss of life
Potential for serious injury
Potential for catastrophic mission failure
Potential for partial mission failure
Potential for loss of equipment
Potential for waste of software resource investmentPotential for adverse visibility
Potential effect on routine operations
Slide 20
Consequence of Software Failure
Criteria Determination for
IV&V
IV&V
Grave
IV&V
Substantial
IV&V
Marginal
96
Insignificant
16
32
64
128
250
Total Likelihood of Failure based on Software Environment
High Risk - IV&V Required
V&V 10/2002
Intermediate Risk - Evaluate for IV&V
Slide 21
Summary
V&V 10/2002
Slide 22
SQA vs. IV&V
PROJECT X
SQA
IV&V
Risk
 SQA  IV&V
V&V 10/2002
Slide 23
IV&V Benefits
Technical
•Better software/system
Performance
•Better Visibility into
Development
•Higher Confidence in
Software Reliability
•Better Decision Criteria
•Compliance between
Specs & Code
•Criteria for Program
Acceptance
V&V 10/2002
Management
•Second Source Technical
Alternative
•Reduced maintenance cost
•Reduced Frequency of
Operational Change
Slide 24
Conclusion
• Applied early in the software development process, IV&V
can reduce overall Project cost.
• NASA policy provides the management process for
assuring that the right level of IV&V is applied.
• IV&V Implementation Criteria provide a quantitative
approach for determining the right level based on mission
risk
• IV&V CANNOT replace Quality assurance but must
supplement it to be successful
• IV&V Requires a strong Quality assurance base
V&V 10/2002
Slide 25
References
IV&V Facility, Fairmont, WV
Director – Ned Keeler
[email protected]
Deputy Director - Bill Jackson
[email protected]
V&V 10/2002
Slide 26