Transcript New Methods of Software Reliability: Estimations and Projections

Software Reliability Estimates/
Projections, Cumulative &
Instantaneous
- Dave Dwyer
Main
Title Slide
With help from:
Ann Marie Neufelder,
John D. Musa, Martin Trachtenberg,
Thomas Downs, Ernest O. Codier and
Faculty of Rivier College Grad. School
Math and Computer Science
Martin Trachtenberg (1985):
• Simulation shows that, with respect to the number
of detected errors:
– Testing the functions of the software system in a random or roundrobin order gives linearly decaying system error rates.
– Testing each function exhaustively one at a time gives flat systemerror rates
– Testing different functions at widely different frequencies gives
exponentially decaying system error rates [Operational Profile
Testing], and
– Testing strategies which result in linear decaying error rates tend to
require the fewest tests to detect a given number of errors.
10/29/2002
2
Thomas Downs (1985):
“In this paper, an approach to the modeling of
software testing is described. A major aim of this
approach is to allow the assessment of the effects
of different testing (and debugging) strategies in
different situations. It is shown how the techniques
developed can be used to estimate, prior to the
commencement of testing, the optimum allocation
of test effort for software which is to be
nonuniformly executed in its operational phase.”
10/29/2002
3
There are Two Basic Types of Software
Reliability Models
• Predictors - predict reliability of software at
some future time. Prediction made prior to
development or test as early as concept
phase. Normally based on historical data.
• Estimators - estimate reliability of software
at some present or future time based on data
collected from current development and/or
test. Normally used later in life cycle than
predictors.
10/29/2002
4
A Pure Approach Reflects the
True Nature of Software
• The execution of software takes the form of the
execution of a sequence of M paths.
• The actual number of paths affected by an
arbitrary fault is unknown and can be treated as a
random variable, c.
• Not all paths are equally likely to be executed in a
randomly selected execution profile.
10/29/2002
5
Start
x1
xN
x2
3
1
2
10/29/2002
M
‘M’ total paths
2 paths affected by x1
‘N’ total faults initially
1 path affected by x2
‘c’ paths affected by an arbitrary fault
6
Further...
 In the operational phase of many large software
systems, some sections of code are executed
much more frequently than others.
 Faults located in heavily used sections of code
are much more likely to be detected early.
10/29/2002
7
Downs (IEEE Trans. on SW Eng. April, 1985)
Showed that Approximations can be Made
 Each time a path is selected for testing, all
paths are equally likely to be selected.
• The actual number of paths affected by an
arbitrary fault is a constant
10/29/2002
8
My Data Assumptions
• Cumulative 8 Hr. test shifts are recorded VS
the number of errors.
• Each first instance is plotted
 The last data point will be at the end of the
test time, even though there was no error,
because a long interval without error is
more significant than an interval with an
error.
10/29/2002
9
Other Assumptions
 Only integration & system test data are used.
 Problems will be designated as priority 1, 2 or 3
(Ref DoD-STD-2167A) where:
 Priority 1: “Prevents mission essential capability”
 Priority 2: “Adversely affects mission essential
capability with no alternative workaround”
 Priority 3: “Adversely affects mission essential
capability with alternative workaround”
10/29/2002
10
Downs Showed  ~ faults/path
• j = (N – j), where:
– N = the total number of faults,
– j = the number of corrected faults,
–  = -r log(1 – c/M),
• r = the number of paths executed/unit time,
• c = the average number of paths effected by each
fault and
• M = the total number of paths
10/29/2002
11
Failure Rate is proportional to failure
number, Downs: j  (N – j)r(c/M).
Given:
N
(0)
j
j
i
T
= total initial number of faults
= initial failure intensity => 0 errors detected/corrected (start of testing)
= cumulative failure intensity after some number of faults is detected, ‘j’
= the number of faults removed over time
= instantaneous failure intensity
= time
N
j
10/29/2002
j = j/T
0
12
Failure rate plots against failure number for a range
of non-uniform testing profiles, M1, M2 paths and
N1, N2 initial faults in those paths. (Logarithmic?)
10/29/2002
13
Imagine two main segments
Segment 1
10/29/2002
Segment 2
14
After testing segment 1, someone asks:
• Given 10 faults found, what’s the reliability
of the code?
• Responses:
– Don’t know how many other faults remain in
section 1, let alone are in section 2
– Don’t know how often sections 1, 2 are used.
– Did we plot failure intensity vs faults?
– Why didn’t we test to the operational profile?
10/29/2002
15
By reference to Duane’s derivation
for hardware reliability,
(Ref. E. O. Codier, RAMS - 1968)
c
= F/T = kT(-m)
F = kT(1 – m)
?F/?T = i = k(1 – m)T(-m) = (1-m) c
i = (1-m) c
10/29/2002
16
Instantaneous Failure Intensity
for Software
j = (N – j) Ф
= j/T
j = T(N – j) Ф
?j/?T = i
= (N – j) Ф + T(- ?j/?T) Ф
i = j + T(-i ) Ф
i(1 + T Ф) = j
i = j/(1 + T Ф)
10/29/2002
17
Priority 1 Data Plotted
45.0
Sum Failures (n)
40.0
35.0
30.0
25.0
20.0
y = -43.964x + 38.803
15.0
10.0
5.0
0.0
0
0.2
0.4
0.6
Failures/8 Hours
10/29/2002
18
Priority 1 and 2 Data Plotted
400.0
Series1
350.0
Series2
Series3
Linear (Series2)
300.0
Linear (Series3)
Sum Failures
250.0
200.0
y = -176.83x + 349.85
150.0
100.0
y = -179.88x + 288.61
50.0
0.0
0
0.2
0.4
0.6
0.8
1
1.2
Failures/8 Hours
10/29/2002
19
Point Estimates vs Instantaneous
FRP Censored
75
70
65
60
55
Critical Failure Number
50
45
Series1
40
Series2
35
Linear (Series1)
Linear (Series2)
30
25
20
15
y = -676.24x + 58.829
R2 = 0.9193
y = -338.12x + 58.829
R2 = 0.9193
10
5
0
0.0000 0.0100 0.0200 0.0300 0.0400 0.0500 0.0600 0.0700 0.0800 0.0900 0.1000 0.1100 0.1200 0.1300 0.1400
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
10/29/2002
Failure Rate
20
From the previous graph, top curve j = -338.12c + 58.89
j
41.98
40.29
43.67
i
c
0.050
0.055
0.045
T= j/c
839.6
732.55
970.44
= (43.67 – 40.29)/(970.44 – 732.55)
= 3.38/237.89
= 0.014
A point estimate over the range including a few points
From the formula for instantaneous failure intensity:
i

T
= c/(1 + T)
= 1/338.12 (-1/slope of curve)
= 839.6
i
= 0.050/(1 + 839/338)
= 0.050/(1 + 2.48)
= 0.050/3.48
= 0.014
A much better estimate, including all points up to c = 0.050
10/29/2002
21
For copy of paper, e-mail request to:
[email protected]