Transcript Introduction to Risk Software Architecture Risk Assessment ميحرلا نمحرلا الله مسب

‫بسم هللا الرحمن الرحيم‬
‫ والصالة والسالم على رسول هللا‬، ‫الحمد هلل‬
Introduction to Risk Management And
Software Architecture Risk Assessment
Hany H. Ammar
LANE Department of Computer Science and Electrical Engineering
West Virginia University, Morgantown, West Virginia, USA, and
Faculty of Computers and Information, Cairo University, Cairo, Egypt
1
OUTLINE
• Risk Management
• Software Architecture Risk Assessment
– Maintainability-based risk
• Conclusions
• Next Steps
SW Architecture Risk Assessment Keynote Presentation MySEC’06
2
Risk Management
SW Architecture Risk Assessment Keynote Presentation MySEC’06
3
SW Architecture Risk Assessment Keynote Presentation MySEC’06
4
For NASA Programs
• RISK MANAGEMENT: An organized, systematic
decision-making process that efficiently identifies
risks, assesses or analyzes risks, and effectively
reduces or eliminates risks to achieving program
goals.
• RISK: A Program “Risk” is any circumstance or
situation that poses a threat to: crew or vehicle safety,
Program controlled cost; Program controlled
schedule; or major mission objectives, and for which
an acceptable resolution is deemed unlikely without a
focused management effort
SW Architecture Risk Assessment Keynote Presentation MySEC’06
5
Risk Management Cycle
Identify: Identify that a risk exits and give it a meaningful name.
Analyze: Determine the severity of the risk according to the risk matrix. If the risk is
negligible (low to medium severity, low likelihood of occurrence), stop here. However,
if the risk could cause damage to the system or the system's users, continue.
Plan: Decide how to combat the risk based on the risk's severity and likelihood of
occurrence.
Mitigate: Follow the plan formulated in the previous phase as closely as possible to
combat the risk. If this approach does not work, return to the previous phase and
make a new plan. If the plan does work, continue analyzing the risk to determine
whether it has been reduced to an acceptable severity level.
Track: Once the risk has been mitigated to an acceptable severity level, the risk
should be tracked to ensure the continued control of the risk. If at any time the risk
seems to resurface, the risk management cycle should begin again, starting with the
analysis phase.
SW Architecture Risk Assessment Keynote Presentation MySEC’06
6
SW Architecture Risk Assessment Keynote Presentation MySEC’06
7
Risk Definition
• According to NASA Software Safety Technical
Standard, risk is defined as:
“exposure to the chance of injury or loss. It is a
function of the possible frequency of occurrence of
the undesired event, of the potential severity of
resulting consequences, and of the uncertainties
associated with the frequency and severity”.
•
For software intensive systems, a risk is a
combination of a likelihood of occurrence of an
abnormal event or failure and the potential
consequences or severity of that event or failure to a
system's operators, users, or environment
SW Architecture Risk Assessment Keynote Presentation MySEC’06
8
Risk Matrix
Likelihood of Occurrence
Severity
Probable
Occasional
Catastrophic
High
High
Critical
High
HighMedium
Marginal
Negligible
Remote
HighMedium
Improbable
Medium
Medium
MediumLow
HighMediu
m
Medium
MediumLow
Low
Medium
MediumLow
Low
Low
SW Architecture Risk Assessment Keynote Presentation MySEC’06
9
SW Architecture Risk Assessment Keynote Presentation MySEC’06
10
NASA IV&V Facility
NPD 2820.1C for Software IV&V Policy states:
"Task the IV&V Facility in Fairmont, West Virginia to manage the
performance of all IV&V for software identified per the
established criteria, and for any other safety critical software (as
defined in NASA-STD-8719.13)"
SW Architecture Risk Assessment Keynote Presentation MySEC’06
11
IV&V Function
• Software Independent Verification & Validation
(IV&V) is a systems engineering process
employing rigorous methodologies for
evaluating the correctness and quality of the
software product throughout the software life
cycle.
• Software IV&V is adapted to the characteristics
of the project. Different projects require
different level of IV&V
SW Architecture Risk Assessment Keynote Presentation MySEC’06
12
IV&V Lifecycle Activities
System
Preliminary
Requirements Design
Review
Review
Initial
IVVP
Signed
Baseline
IVVP
Signed
Concept
Phase
2.0
Critical
Design
Review
S/W
FQT
System
Test
- IV&V provides support and reports for Project milestones
- Technical Analysis Reports document major phases
- IVVP is updated to match changes in Project
Requirements
Phase
3.0
Design
Phase
4.0
Implementation
Phase
5.0
IV&V Phase Independent Support
1.0
Test
Phase
6.0
Mission
System
Readiness
Retirement
Launch
Review
IV&V IV&V Final
Provides Report
CoFR
Operations &
Maintenance Phase
7.0
Note: numbers correspond to IV&V WBS
• Life-cycle IV&V is designed to mesh with the Project schedule and
provide timely inputs to mitigate risk
• Dialog between the IV&V Facility and the Project must begin before
SRR
• For most Projects, IV&V ends (and the Final Report is delivered) on
or about MRR. Some Projects have extended S/W development
post-launch or major upgrades/maintenance (e.g. Shuttle, MER)
SW Architecture Risk Assessment Keynote Presentation MySEC’06
13
Software Project Resolution
Project Resolution is commonly categorized into three
resolution types:
1. Successful Projects
– Completed and operational, and:
• On Schedule
• On Cost
• With all originally specified features and functions
2. Challenged Projects
– Completed and operational, but:
• Behind Schedule------- Project Risk
• Over Cost--------------- Project Risk
• With fewer features and functions than originally specified
------------ Product Risk
3. Failed Projects:
– Cancelled before completion or never implemented
SW Architecture Risk Assessment Keynote Presentation MySEC’06
14
Software CHAOS
The Standish Group has examined 30,000 Software Projects in the US
since 1994. This "CHAOS" research has revealed a decided improvement
in IT project management with the implementation of standards and
practices such as IV&V. This improvement correlates with the rise in
project success depicted in the chart below:
Project Resolution History (1994-2000)
2000
1998
1996
1994
0%
28%
49%
26%
27%
16%
46%
28%
33%
40%
Successful
Challenged
Failed
40%
53%
20%
23%
31%
60%
80%
100%
The Standish Group International, Inc.: Extreme CHAOS (2001) - The 2001 update to the
CHAOS report. http://www.standishgroup.com/sample_research/PDFpages/extreme_chaos.pdf
SW Architecture Risk Assessment Keynote Presentation MySEC’06
15
Error Detection/Correction
Early error detection and correction are vital. The cost to correct
software errors multiplies during the software development lifecycle.
Early error detection and correction reduce costs and save time.
Relative Cost to Fix Defects per Phase Found
50
45
40
35
30
Defect
Type
25Relative
20
Cost
to Fix
15
Requirements
Design
10
5
Code
0
Test
s
ent
m
e
r
qui
Re
n
s ig
De
de
Co
t
Tes
Phase Found
Test
Code
Design
Requirements
Direct Return on Investment of Software Independent Verification and Validation:
Methodology and Initial Case Studies, James B. Dabney and Gary Barber, Assurance Technology
Symposium, 5 June 2003.
SW Architecture Risk Assessment Keynote Presentation MySEC’06
16
IV&V Characteristics
• Includes Risk Identification and Mitigation Techniques
• Provides Independent Evaluation / Assessment of:
– Are we building the product right? = Verification
– Are we building the right product? = Validation
• Requires Technical, Managerial and Financial Independence
• Makes a value added contribution, everyone shares the same
mission success objective
– For NASA Management - Provides Mission Assurance
– For Project Management - Provides Unbiased Source of Help
• Helps deliver
– Risk Identification and Mitigation
– Increased Quality and Safety
– Improved Timeliness and Reliability
– Reduced Rework Cost
NPD 8730.4: Requires NASA programs and projects that contain mission or safety
critical software to document decisions concerning the use of IV&V.
SW Architecture Risk Assessment Keynote Presentation MySEC’06
17
OUTLINE
• Risk Management
• Software Architecture Risk Assessment
– Reliability-based risk
– Performance-based risk
– Maintainability-based risk
– Component Ranking
• Conclusions
• Next Steps
SW Architecture Risk Assessment Keynote Presentation MySEC’06
18
Software Architecture Risk
Assessment
This work is funded in part by grants to West Virginia University
Research Corp. from the NSF (ITR) Program, and from the NASA
Office of Safety and Mission Assurance (OSMA) Software
Assurance Research Program (SARP) managed through the NASA
Independent Verification and Validation (IV&V) Facility, Fairmont
SW Architecture Risk Assessment Keynote Presentation MySEC’06
19
Project Overview
• Risk Assessment of software
architecture components, usage
What keeps satellites working 24/7 ?
scenarios, and requirements
• Risk definition is based on
* Frequency of abnormal events
* Severity or consequences of events
•
•
•
•
•
Reliability-based risk,
Performance-based risk
Requirement–based risk
Severity Analysis
Maintainability-based risk
SW Architecture Risk Assessment Keynote Presentation MySEC’06
20
Software Architecture Risk Assessment
• An architecture-based approach for risk assessment
Components\connectors, requirements, and scenario risk,
• Define several types of risk factors
Reliability-based Risk [IEEE Trans. on Rel 2001, on SE, 2002, 2003]
Probability of failure
* Severity or Consequences of this failure
Maintainability-based Risk [RAMS 06, ICSM 05, ICSM 04]
Probability of performing maintenance task
* Cost of performing this task
• The losses caused by low system maintainability can be:
– High cost of maintenance effort
– Loss of the system by aging
Performance-based Risk [IEEE Trans. SE, Jan. 2005]
Probability of missing timing or performance requirements
* Severity or Consequences
SW Architecture Risk Assessment Keynote Presentation MySEC’06
21
Importance / Benefits
Components Risk
Risk
Factor
Components
• Identify the high risk components of the system in terms of
Reliability/Maintainability/Performance
SW Architecture Risk Assessment Keynote Presentation MySEC’06
22
Software Architecture Risk Assessment
Methodology
Requirements Model
System Architecture Model
Software Architecture Risk Assessment
Reliabilitybased
Risk Analysis
Maintainabilitybased
Risk Analysis
Components Risk Factors
Performancebased
Risk Analysis
Components
Ranking
SW Architecture Risk Assessment Keynote Presentation MySEC’06
23
OUTLINE
• Risk Management
• Software Architectures
• Software Architecture Risk Assessment
– Maintainability-based risk
• Conclusions
• Next Steps
SW Architecture Risk Assessment Keynote Presentation MySEC’06
24
Maintenance Effort
Corrective
20%
Perfective
55%
Adaptive
25%
Maintainability-based Risk
[ICSM 05] AbdelMoez, Goseva, Ammar, Mili, Fuhrman,
“Architectural level Maintainability Based Risk Assessment”
[RAMS 06] AbdelMoez, Goseva, Ammar,” Methodology for
Maintainability Based Risk Assessment”, Jan 2006.
SW Architecture Risk Assessment Keynote Presentation MySEC’06
25
Importance / Benefits
Maintainability-based Risk
• According to Pigoski, 60%-80% of the system budget
is spent on maintenance

Enhancements
(perfective/
adaptive
maintenance)
account for 78%83% of the
maintainer effort
Maintenance Effort
Corrective
20%
Perfective
55%
Adaptive
25%
SW Architecture Risk Assessment Keynote Presentation MySEC’06
26
Importance / Benefits
Maintainability-based Risk
• Unisys holds the NASA contract to
maintain and support 14 million
lines of ground software for the
space shuttle
• There were 3,800 requirement
changes made to the software
after the loss of Challenger. These
changes resulted in 900 software
releases, of which 30 applied to
the mission-control center with 3
of these being major upgrades • Reference:
IEEE Software, Vol.6, No.1, pp. 116-119
http://hebb.cis.uoguelph.ca/~dave/343/Lectures/introduction.html
SW Architecture Risk Assessment Keynote Presentation MySEC’06
27
Importance / Benefits
Trade off Analysis for Perfective
Maintenance
Risk
Factor
Change in Risk
Components Risk
0.25
0.2
Change in Risk
0.15
0.1
0.05
0
-0.05
Components
-0.1
apptgui banner
borg
btgui
calgui calmodel errmsg helpscrn model propgui srchgui taskgui taskmodel tdgui
Components
Patterns
Maintainability risk for perfective maintenance (open source case study Borg)
SW Architecture Risk Assessment Keynote Presentation MySEC’06
28
Software Architecture Risk Assessment
Methodology: Maintainability-based Risk
Requirments maturity Index
/ change / error reports
(1) Estimate components
Initial Change Probability
(ICP)
SW Architecture
(2) Estimate Change
Propagation (CP)
probabilities
(3) Estimate
Size of Change
(SC)
CP=[cpi/j]
SC=[sci/j]
ICP=[icpi]
(4) Estimate component risk factor



MR  mri     icp j . cpi / j . sci / j .  cp j / i . sc j / i 
 j
  j i

SW Architecture Risk Assessment Keynote Presentation MySEC’06
29
Maintenance change
propagation
Incoming maintenance
Outgoing maintenance
SW Architecture Risk Assessment Keynote Presentation MySEC’06
30
Estimating Change Propagation
C1
Change in
Provided
Service
V1
V11
 vij
=1
V12
V13
C2
.
.

ij
v
V13
Required
Services
=0
• Change Propagation Probabilities matrix CP=[cpij ]
cpij is the probability that a change in Ci due to corrective/
perfective maintenance requires a change in Cj while
maintaining the overall function of a system S
cpij = P([Cj]  [Cj'] | [Ci]  [Ci'] ^ [S] = [S'] )
 cpij is estimated by
1
cpij =
 vij

| Vi |  Vi
SW Architecture Risk Assessment Keynote Presentation MySEC’06
31
Estimating Size of Change
C1
Change in
Provided
Service
V1
V11
V12
V13
C2
M1
M2
M3
…
M7
Receiving
Component
methods
 Size of change SC=[scij ]
scij is defined as the ratio between the number of affected
methods of the receiving component caused by the
changes in the interface elements of the providing
components and the total number of methods in the
receiving component
1 |Vi | ij
v
scij =
 scij is estimated by

| M i |  1
SW Architecture Risk Assessment Keynote Presentation MySEC’06
32
CM1 Maintainability-Based Risk
in Adaptive Maintenance Context
SW Architecture Risk Assessment Keynote Presentation MySEC’06
33
Case Study: NASA CM1 UML Model
Structure Diagram
• The UML-RT
Model of CM1 was
Developed by
WVU students
(Nathan, Tom and
Rajesh, Summer
2004) based on the
CM1 software
design specification
SW Architecture Risk Assessment Keynote Presentation MySEC’06
34
Change Propagation Probabilities for CM1
• The Change
Propagation
probabilities CP is
estimated using the
CM1 UML model
• The Change
Propagation
probabilities CP can
be automatically
estimated from
UML-RT models, or
java source
SW Architecture Risk Assessment Keynote Presentation MySEC’06
35
Size of Change for CM1
• The Size of Change
metrics SC is
estimated using the
CM1 UML model
• The Size of Change
metrics SC
Probabilities CP can
be automatically
estimated from
UML-RT models, or
Java source
SW Architecture Risk Assessment Keynote Presentation MySEC’06
36
Software Architecture Risk Assessment
Methodology: Maintainability-based Risk
Requirments maturity Index
/ change / error reports
(1) Estimate components
Initial Change Probability
(ICP)
SW Architecture
(2) Estimate Change
Propagation (CP)
probabilities
(3) Estimate
Size of Change
(SC)
CP=[cpi/j]
SC=[sci/j]
ICP=[icpi]
(4) Estimate component risk factor



MR  mri     icp j . cpi / j . sci / j .  cp j / i . sc j / i 
 j
  j i

SW Architecture Risk Assessment Keynote Presentation MySEC’06
37
Maintainability-based Risk
For corrective maintenance
(case study CM1)
ICP is estimated
using error reports
data
SW Architecture Risk Assessment Keynote Presentation MySEC’06
38
Prioritizing Corrective Maintenance
Tasks for CM1
Components
Severity
Level
BIT
CCM
DCI
DCX
DPA
EDAC
ICUI
1553
SCUI
SSI
TIS
TMALI
Minor
Cat.
Cat.
Minor
Major
Major
Critical
Cat.
Critical
Cat
Major
Critical
SW Architecture Risk Assessment Keynote Presentation MySEC’06
39
Maintainability-based Risk
Maintainability risk for Adaptive
maintenance (case study CM1)
ICP is estimated
using change reports data
SW Architecture Risk Assessment Keynote Presentation MySEC’06
40
Tool Support
SW Architecture Risk Assessment Keynote Presentation MySEC’06
41
Technology Readiness Level
The Software Architecture Risk Assessment
Tool Support
SW Architecture Risk Assessment Keynote Presentation MySEC’06
42
Conclusions
• Risk Management is vital to the success of projects
and products
• A risk Assessment process is needed
• Software Architecture is a major determinant of
software quality
• Software Architecture can be used to manage
project and product risks
• Development of a methodology and a process for
software architecture risk assessment
• Continued development of a software architecture
risk assessment tool to support the methodology
SW Architecture Risk Assessment Keynote Presentation MySEC’06
44
Papers Published
1.
2.
3.
4.
5.
6.
7.
8.
9.
Vittorio Cortellessa, Katerina Goseva-Popstojanova, Kalaivani Appukkutty, Ajith R. Guedem, Ahmed
Hassan, Rania Elnaggar, Walid Abdelmoez, Hany H. Ammar, “Model-Based Performance Risk
Analysis, IEEE Transactions on Software Engineering, January 2005, (Vol. 31, No. 1), pp.3-20.
Katerina Goseva-Popstojanova, Ahmed Hassan, Ajith Guedem, Walid Abdelmoez, Diaa Eldin M.
Nassar, Hany Ammar, Ali Mili, "Architectural-Level Risk Analysis Using UML", IEEE Transactions on
Software Engineering, October 2003 (Vol. 29, No. 10), pp.946-960.
S. Yacoub, H. Ammar, “A Methodology for Architectural-Level Reliability Risk Analysis,” IEEE
Transactions on Software Engineering, Vol. 28, No. 6, June 2002.
W. AbdelMoez, K. Goseva-Popstojanova, H.H. Ammar,” Methodology for Maintainability-Based Risk
Assessment”, Proc. of the 52nd Annual Reliability & Maintainability Symposium (RAMS 2006),
Newport Beach, Ca., January 23-26, 2006.
Israr P. Shaik , W. Abdelmoez, R. Gunnalan, M. Shereshevsky, A. Zeid, H.H. Ammar, A. Mili, C.
Fuhrman, “Change Propagation for Assessing Design Quality of Software Architectures”, Proc. of 5th
IEEE/IFIP Working Conference on Software Architecture (WICSA), Pittsburgh, Pa., USA, November
6-9, 2005.
AbdelMoez, W., I. Shaik, R. Gunnalan, M. Shereshevsky, K. Goseva-Popstojanova, H.H. Ammar, A.
Mili, C. Fuhrman, “Architectural level Maintainability Based Risk Assessment”, Proc. of poster papers in
IEEE International Conference on Software Maintenance (ICSM 2005), Budapest, Hungary,
September 25-30,2005.
W. Abdelmoez, D. M. Nassar, M. Shereshevsky, N. Gradetsky, R. Gunnalanm and H. H. Ammar, Bo
Yu, and Ali Mili "Error Propagation in Software Architectures". In Proceedings of the 10th International
Symposium on Software Metrics (METRICS'04), September 11 - 17, 2004 , IEEE Comp. Soc., pp
384-393
Abdelmoez, W., M. Shereshevsky, R. Gunnalan, H. H. Ammar, Bo Yu, S. Bogazzi, M. Korkmaz, A. Mili,
"Software Architectures Change Propagation Tool (SACPT),” 20th IEEE International Conference on
Software Maintenance (ICSM'04) September 11 - 14, 2004 , Chicago, Illinois, IEEE Comp. Soc., pp
517
A. Hassan, K. Goseva-Popstojanova, and H. Ammar, “UML Based Severity Analysis Methodology”,
Proceedings of the 2005 Annual Reliability and Maintainability Symposium (RAMS 2005),
Alexandria, VA, January 2005.
SW Architecture Risk Assessment Keynote Presentation MySEC’06
45