Transcript Confidence bounds in software reliability testing
Confidence intervals in software reliability testing
Alessandro Di Bucchianico (LaQuSo, Eindhoven University of Technology) Ed Brandt and Rob Henzen (Refis, Netherlands) ENBIS-5 Newcastle September 15, 2005 LaQuSo is an activity of Technische Universiteit Eindhoven
Goals of this talk
show how to obtain confidence intervals for software reliability predictions from NHPP models apply results to case study ENBIS-5, September 15, 2005 1
Overview of this talk
Introduction of LaQuSo and Refis Case Dutch Ministry of Transport, Public Works and Water Management Software reliability models Confidence intervals for NHPP models: asymptotics simulation goodness-of-fit tests Conclusions ENBIS-5, September 15, 2005 2
LaQuSo: Laboratory for Quality Software
university based laboratory started at the Eindhoven University of Technology Radboud University (Nijmegen) has recently joined as partner statistics and probability group in math department at TU/e is one of the participating groups started in January 2004: 10 fte; will grow to 50 fte case-study driven in cooperation with industry statistics will be integrated part of testing and verification activities more information: www.laquso.com
ENBIS-5, September 15, 2005 3
Refis
consultancy company in Bilthoven, the Netherlands activities include: software reliability assessments measurements systems for IT sector test audits for more information, see www.refis.nl
ENBIS-5, September 15, 2005 4
Context of case
2/3 of the Netherlands is below sea level protection against sea and rivers by dunes dikes dams sluices … hardware reliability of sluices is well understood and documented control of sluices by huge software systems (reliability??) ENBIS-5, September 15, 2005 5
Sluice (1)
ENBIS-5, September 15, 2005 6
Sluice (2)
ENBIS-5, September 15, 2005 7
Goals of case
Case is project of Dutch Ministry of Transport, Public Works and Water Management (see www.rws.nl
for general information) Obtain information on reliability of software system Registration system for defect detection and repair Predict system reliability with confidence bounds ENBIS-5, September 15, 2005 8
Available data
Data available from three tests: plant acceptation test site acceptation test site acceptation retest Defect counts grouped data severity index repair status … Data was collected manually and checked on consistency etc.
ENBIS-5, September 15, 2005 9
Data assumptions
Assumptions are results from intensive discussions with project and test engineers all test intervals have same effort every test period corresponds to 219 days of actual use immediate correction of errors (gaps between testing periods allowed for this) no new error introduced by correction actions ENBIS-5, September 15, 2005 10
Data (severity 1 FAT)
cumulative errors 14 12 10 8 6 4 2 2 4 6 8 10 12 14 period
ENBIS-5, September 15, 2005 11
Software reliability models
Main differences with hardware reliability: no wear no burn-in exact reproducibility of errors Hundreds of reliability growth models available Dedicated software for software reliability exists (not always reliable, though): Casre Smerfs … ENBIS-5, September 15, 2005 12
Initial Model Selection
Models available in standard software reliability packages (Smerfs, Casre) were judged on several criteria (assumptions or properties), including: upper bound on number of errors interval data length of test intervals distribution of errors shape of failure intensity … The list of selected models included two NHPP models (Goel-Okumoto and Yamada S-shaped) ENBIS-5, September 15, 2005 13
Nonhomogeneous Poisson process
N
(
t
)=4
T 1 T 2 T 3 T 4
0 ( )
k
)
e
t k
!
k
This is a Type II model (cf. Langberg/Singpurwalla (1985)) that in general cannot be described easily in terms of time between failures.
Special case: Poisson process
t
( )
k
)
e
t k k
!
ENBIS-5, September 15, 2005 14
NHPP models
Several choices for have been introduced: expected number of failures time
L
(
t
)
n
( 1
e
at
) Goel-Okumoto, Musa
L
(
t
)
n
( 1 ( 1
at
)
e
at
)
L
(
t
)
n
( 1
e
at
) 1
ye
bt L
(
t
)
i m
1
n i
( 1
e
a i t
)
L
(
t
) 1
a
ln( 1
abt
) delayed S-shaped inflection S-shaped hyperexponential logarithmic ENBIS-5, September 15, 2005 15
NHPP models: inference for grouped data
data consists of counts in time intervals:
n i =
# detected failures in time interval (
t i-1 ,t i
] likelihood function (
t 0 =
0):
L
(
y
1 ,
y
2 ,
L
,
y n
;
t
1 ,
t
2 ,
L
,
t n
)
i n
1 (
L
(
t i
)
L
(
t i
1 ))
y i y i
!
exp(
L
(
t n
)) (
t
) = cumulative hazard rate at time
t
= expected number of failures at time
t
if has parametric form, then maximizing
L
yields ML estimates for parameters (
t
) = d/dt (
t
) = hazard rate at time
t
ENBIS-5, September 15, 2005 16
NHPP models with 2 parameters: inference for parameters
Assume depends on 2 parameters a and e ML-estimators have no closed form asymptotic distribution through Fisher information:
F
E
2 2 ln ln / / 2
a
a
e
2 2 ln ln / /
a
2
e
e V
F
1
AsymVar
(
â AsymCov
(
â
, )
ê
)
AsymCov
(
â
,
AsymVar
(
ê
)
ê
)
â
:
N
(
a
,
AsymVar
(
â
))
ê
:
N
(
e
,
AsymCov
(
ê
)) ENBIS-5, September 15, 2005 17
NHPP models with 2 parameters: inference for function of parameters
assume depends on two parameters
a
and
b
asymptotic distribution of functions of
a
and
b
through Fisher information and delta method:
Var
(
a
,
e
) ( 2
a f
)
a
â AsymVar
(
â
) ( 2
e f
)
e
ê AsymVar
(
ê
) 2
f
2 (
a
e
)
a
â
,
e
ê AsymCov
(
â
,
ê
) examples of functions of parameters include: probability of no failure in certain time period failure intensity at
t
=
t 0
ENBIS-5, September 15, 2005 18
Simulation NHPP process
N
(
t
)=4
T 1 T 2 T 3 T 4
0
k
)
e
k
!
k t
Conditional on the event
N
(
t
)=
n
, the
T 1 ,…,T n
are distributed as the order statistics of a sample of size
n
from a distribution with density
(t
) / (
t
).
Hence, simulating a sample from a distribution with density
(t
) / (
t
) can be used to simulate an NHPP process with intensity
(t
) ENBIS-5, September 15, 2005 19
Goodness-of-fit NHPP process
N
(
t
)=4
T 1 T 2 T 3 T 4
0
k
)
e
k
!
k t
Conditional on the event
N
(
t
)=
n
, the
T 1 ,…,T n
are distributed as the order statistics of a sample of size
n
from a distribution with density
(t
) / (
t
).
Hence, the Kolmogorov goodness-of-fit test based on the empirical distribution function may be used to perform a GOF test.
ENBIS-5, September 15, 2005 20
Back to case study
parameter estimates and 95% confidence intervals for Goel-Okumoto model (a(1-exp(b t)): a : ( 13.2 , 19.5989 ) b = ( 0.000818358 , 0.00318164 ) goodness-of-fit: OK at 5% level important question from Dutch politics: 95% confidence interval for probability of no failure in 1 year: ( 0.799462 , 1 ) (thus confirmation of suspicion by Ministry officials that defect system is not good enough for required probabilities) ENBIS-5, September 15, 2005 21
Conclusions
asymptotic confidence intervals for functions of parameters in NHPP models may obtained from Fisher information testing registration of Dutch water works not sufficient to obtain high-precision estimates of software reliability ENBIS-5, September 15, 2005 22
Literature
Rijkswaterstaat report (confidential) Systematic description of software reliability models, manuscript in progress (ADiB + Refis) Xie and Hong (2001), Handbook of statistics 20 (Advances in Reliability), 707-731.
ENBIS-5, September 15, 2005 23