Transcript Software Testing: A Selection of Papers by Elaine Weyuker

Software Testing:
A Selection of Papers by
Elaine Weyuker
Peter Greenwood
for EEE 599
Elaine Weyuker
• PhD (CS) Rutgers U ‘77
• Software Testing
Researcher at AT&T Bell
Labs, Murray Hill, NJ
• CS Professor Courant
Institute of NYU
• 30+ papers, primarily on
Software Testing
Apr 98
EEE 599 - Weyuker Presentation
2
The Papers
• Axiomatizing Software Test Data
Adequacy (TSE Dec 86)
• The Automatic Generation of Load Test
Suites and the Assessment of the
Resulting Software (TSE Sep 95, with
Avitzer)
• Reliability Testing of Rule-Based
Systems (Software Jun 96, with Avritzer
and Ros)
Apr 98
EEE 599 - Weyuker Presentation
3
Goal of Presentation
• highlight some of Weyuker’s prolific
work in software testing
• contrast the general and theoretical
character of her early work with the
specific and practical nature of her
current work
Apr 98
EEE 599 - Weyuker Presentation
4
Axiomizing Software Test
Data Adequacy (Dec 86)
• How much testing is enough to
determine if a program is acceptable?
– very important question
– relatively little research activity (1986)
– no industry standards
• test until testing time expired
• test until all cases in current test set pass
Apr 98
EEE 599 - Weyuker Presentation
5
When Should Testing Stop?
• Terminate when an adequacy criterion
is met
– a measure of how well the testing process
has been performed
– relates a test set to the program, the
specification, or both
– could relate a test set to the program’s
operational profile
Apr 98
EEE 599 - Weyuker Presentation
6
Constraints of this
Approach
• deals only with dynamic testing (running
the program on test cases)
• does not involve any static testing
(analysis of code, structure, etc.)
Apr 98
EEE 599 - Weyuker Presentation
7
Constraints (con’t)
• Weyuker’s paper deals only with
program-based criteria
– ignores specifications
– most adequacy criteria are of this type
– easier to automate
Apr 98
EEE 599 - Weyuker Presentation
8
Weyuker on Testing
“The purpose of testing is to uncover
errors, not to certify correctness.”
Apr 98
EEE 599 - Weyuker Presentation
9
The Axioms (1 & 2)
• Axiom 1: (Applicability) For every
program, there exists a finite adequate
test set.
• Axiom 2: (Nonexhaustive Applicability)
There is a program P and a test set T
such that P is adequately tested by T,
and T is not an exhaustive test set.
Apr 98
EEE 599 - Weyuker Presentation
10
Axioms 3 & 4
• Axiom 3: (Monotonicity) If T is adequate
for P, and T is a subset of T’, then T’ is
adequate for P.
• Axiom 4: (Inadequate Empty Set) The
empty set is not adequate for any
program.
Apr 98
EEE 599 - Weyuker Presentation
11
Axioms 5 & 6
• Axiom 5: (Anti-extensionality) There are
programs P and Q such that P is
equivalent to Q, T is adequate for P, but
T is not adequate for Q.
• Axiom 6: (General Multiple Change)
There are programs P and Q which are
the same shape, and a test set T such
that T is adequate for P, but T is not
adequate for Q.
Apr 98
EEE 599 - Weyuker Presentation
12
Axiom 7
• Axiom 7: (Anti-decomposition) There
exists a program P and a component Q
such that T is adequate for P, T’ is the
set of vectors of values that variables
can assume on entrance to Q for some t
of T, and T’ is not adequate for Q.
Apr 98
EEE 599 - Weyuker Presentation
13
Axiom 8
• Axiom 8: (Anti-composition) There exist
programs P and Q such that T is
adequate for P and P(T) is adequate for
Q, but P is not adequate for P;Q.
Apr 98
EEE 599 - Weyuker Presentation
14
Summary of Axiomatizing
• axioms are applied to 5 adequacy
criteria
– path, branch, statement, mutations,
modified size
• path, branch, statement fail application
and antidecompositon
• branch and statement also fail
anticomposition
Apr 98
EEE 599 - Weyuker Presentation
15
Summary of Axiomatizing
applicability




modified
size

non-exhaustive
applicability





monotonicity





inadequate
empty set
antiextensionality
general
multiple
change
antidecomposition
anticomposition

























Axiom
Apr 98
statement branch path mutation
EEE 599 - Weyuker Presentation
16
Criticism
• Zweben & Gourlay, On the Adequacy of
Weyuker’s Test Data Adequacy Axioms
(IEEE TSE Apr 89)
– “The properties proposed by Weyuker are
neither fundamental enough nor precise
enough to serve as a useful formal system
for discussing adequacy or comparing
criteria”
Apr 98
EEE 599 - Weyuker Presentation
17
More Criticism
• Professor Norman Fenton, keynote
address to 2nd QUALMS User Group
Meeting, South Bank University, Jul 92
– “Elaine Weyuker … has presented a list of
axioms … Measurement Theory reveals
that some of these axioms are
contradictory.”
Apr 98
EEE 599 - Weyuker Presentation
18
Dr. Weyuker’s Current Work
Apr 98
EEE 599 - Weyuker Presentation
19
Automatic Generation of
Load Test Suites (Sep 95)
• develops 3 algorithms for automatic test
case generation
• applies them to 5 real industrial
software systems
• assess the results
Apr 98
EEE 599 - Weyuker Presentation
20
Failure Types
• Hard failure
– complete loss of traffic ( ie cable break,
power failure, router failure)
• Soft failure
– degraded performance (ie activity burst)
– important that this does not occur
– telecoms designed with excess capacity
– avoid permanent soft failure
Apr 98
EEE 599 - Weyuker Presentation
21
Goals
• a smoothly degrading system
• graceful recovery
Apr 98
EEE 599 - Weyuker Presentation
22
Load Testing Strategies
• system functionality has already been
tested
• assess system performance under a
known load
• black-box oriented
• based on system operational profile
Apr 98
EEE 599 - Weyuker Presentation
23
Typical Telecom Operational
Profile
• number of different types of call
• average service demands and
probability distribution of each call type
• average external arrival rate
• routing probabilities for each node
• this information is usually readily
available
Apr 98
EEE 599 - Weyuker Presentation
24
Purpose of Algorithms
• tests resource allocation mechanisms
associated with software systems
• system behavior under load is not
measured by functionality testing
• each variable vector is a distinct system
state
Apr 98
EEE 599 - Weyuker Presentation
25
Basis of Algorithms
• too many states to exhaustively test
• calculate probability of system being in
each state (from operational profile)
• heuristically determine some threshold 
• generate a test case for those states
where P(S) > 
Apr 98
EEE 599 - Weyuker Presentation
26
The Algorithms
• Deterministic State Testing (DST)
– generate a test case for each state S
where P(S) > 
• DST with State Transition Checking
(TDST)
– as for DST, but also generate test cases for
every S’ reachable from S
Apr 98
EEE 599 - Weyuker Presentation
27
The Algorithms (con’t)
• DST with Length N Cycles (CDST)
– as for DST, but also generate test cases for
every S’ within a cycle of length N from S
with a probability > 
Apr 98
EEE 599 - Weyuker Presentation
28
Target Systems
• Project 1:
• Project 2:
• Project 3:
• Project 4:
• Project 5:
Apr 98
telephone call rerouter
800 number call traffic
monitor
routing automator
transaction processing
system
network interface software
EEE 599 - Weyuker Presentation
29
Testing Results
• automated test generation routines
were developed for all five projects
• automated test results were presented
for projects 1 & 2
Apr 98
EEE 599 - Weyuker Presentation
30
Project 1 Results
• telephone call rerouter software
• test results accurately predicted system
performance
• testing revealed a serious architectural
problem
Apr 98
EEE 599 - Weyuker Presentation
31
Project 2 Results
• 800 number call traffic monitor
• led to better requirements analysis
• used automated test case generation
exclusively, replacing hand-crafting
• more test cases with more variety
• uncovered four potentially severe faults
• minimal additional cost
Apr 98
EEE 599 - Weyuker Presentation
32
Reliability Testing of RuleBased Systems (Sep 96)
• applies same automatic test case
generation algorithm to an industrial
network surveillance system
• system was a reimplementation, so a
large body of operational data was
available from the original system
• system designed to run 24 hours/day;
365 days/year
Apr 98
EEE 599 - Weyuker Presentation
33
Approach
• system has very large state space
• 355 days of operational data from 93
switches were used to identify 857
states entered
• 0.999 coverage required testing 843
states
Apr 98
EEE 599 - Weyuker Presentation
34
State Testing Requirements
Probability
of Coverage
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0.99
0.999
1.0
Apr 98
States to Test
34
51
72
99
137
206
347
721
843
857
EEE 599 - Weyuker Presentation
35
Findings
• historical data is useful for selecting test
cases based on state probabilities
• selection algorithm must consider
– reliability requirements
– size of the (limited) state space
– comprehensiveness of individual test
cases
Apr 98
EEE 599 - Weyuker Presentation
36
Summary
• change of focus from theory to
application
• extension from telecom to general
application
• accomplished, respected, and prolific
researcher in software testing and
reliability
Apr 98
EEE 599 - Weyuker Presentation
37