Transcript SENG 521 Software Reliability & Testing Preparing for Test

SENG 521
Software Reliability &
Testing
Preparing for Test
(Part 6a)
Department of Electrical & Computer Engineering, University of Calgary
B.H. Far
（[email protected]）
http://www.enel.ucalgary.ca/~far/Lectures/SENG521/06a/
SENG521 (Fall 2002)
[email protected]
1
Concepts /1



Run: The smallest division of work that can be
initiated by external intervention. Run is associated
with input state (set of input variables) and runs
with identical input state are of the same run type.
Operation: An operation is a group of run types as
conceived at the development stage.
Test procedure: A controller that sets up
environmental conditions and invokes randomly
selected test cases at random times.
SENG521 (Fall 2002)
[email protected]
2
Concepts /2
Types of test



Feature Test: A single execution of an operation
with interaction between operations minimized.
Load Test: Testing with field use data and
accounting for interactions
Regression Test: Feature tests after every build
involving significant change, i.e., check whether
a bug fix worked.
SENG521 (Fall 2002)
[email protected]
3
Concepts /3

Direct input variable: is a variables that
controls the operation directly


Example: arguments, selection menu, entered
data field.
Indirect input variable: is a variable that
only influences the operations or its effects
are propagated to the operation

Example: traffic load, environmental variable
SENG521 (Fall 2002)
[email protected]
4
Example

Example of direct and indirect input
variables for an operation
SENG521 (Fall 2002)
[email protected]
5
What is a Test Case?



A test case is a partial specification of a run
through the naming of its direct input
variables and their values.
Test case is independent of operational mode
The same test case can execute in different
operational modes. Thus a test case can
generate multiple runs each with different
potential failure behavior
SENG521 (Fall 2002)
[email protected]
6
Properties
A “good” test case satisfies:
 It has a reasonable probability to catch an
error
 It is not redundant
 It is the best of its breed
 It is neither too simple nor too complex
 It makes program failure obvious
SENG521 (Fall 2002)
[email protected]
7
Example


Test case is specified with
its direct input variables
In theory, it is possible to
record all the input
variables needed to
initiate the runs that make
up the whole execution
space of the software
SENG521 (Fall 2002)
[email protected]
8
Test Case & Run



Specification of the indirect input variable gives a
test case the necessary context so it can become a
run.
During feature and regression test, the influence of
indirect input variables should be kept to minimum
to ensure that the operation is reliable.
Indirect input variables are effective during load test.
The load test is divided into a number of operation
modes each derived by a test procedure.
SENG521 (Fall 2002)
[email protected]
9
Procedure
The procedure involves two steps:
1) Preparing test cases (Test case management)

Using either field data recording or operational profiles
2) Preparing test procedures


Test procedure is a controller that sets up environmental
conditions and invokes randomly selected test cases at
random times.
Prepare one test procedure for each operational mode
using operational profile and the operation occurrence
rate
SENG521 (Fall 2002)
[email protected]
10
Test Cases Management
The procedure for preparing test cases involve:
1. Estimate the number of new test cases needed for
the current release
2. Allocate the number of new test cases among the
systems to be tested
3. Allocate the number of new test cases for each
system among its new operations
4. Specify the new test cases
5. Adding the new test cases to the ones already
available (may be from a previous release)
SENG521 (Fall 2002)
[email protected]
11
1. Estimate New Test Cases


Affected by two factors: time and cost
Method:

Compute the number of test cases for the given

Time:
(available time X available staff)/
(average time to prepare a test case)

Cost:
(available budget) / (average preparation cost per test case)

Select the minimum number of the two
SENG521 (Fall 2002)
[email protected]
12
2. Test Case Allocation:
System





Allocate the bulk of the test cases to the product itself.
Give weight to the differing operations that have high
occurrence probabilities.
Do not allocate test cases to acquired components unless the
newly acquired components are substituted for previous ones.
Allocate test cases to acquired components based on their
size relative to the product and the estimated risk.
In the case of Fone Follower (example in Musa’s Book), the
operating system represented a substantial part of the product
and its reliability was unknown. Hence 500 test cases were
allocated to the product and 200 to the operating system.
SENG521 (Fall 2002)
[email protected]
13
3. Test Case Allocation:
Operations
1.
2.
3.
4.
5.
Convert graphical representation of the operational profile
to the tabular representation by walking through all the
paths, obtain an occurrence probability for each operation
(path) by multiplying the branch probabilities together.
Identify the rarely occurring critical new operations and
determine how many test cases to pre-assign to each.
Determine the allocation probabilities for the other new
operations.
Pre-assign one test case to each infrequent other new
operation.
Assign the remaining test cases to the remaining other new
operations in accordance with the allocation probabilities.
SENG521 (Fall 2002)
[email protected]
14
Critical Operations



A critical operation is one for which successful execution
adds a great deal of extra value and failure causes a great
deal of impact with respect to human life, cost, or system
capability.
An example of a critical operation is the SCRAM operation
in nuclear power plants that shuts down a reactor when it
starts to overheat. The operation that handles this condition
is very rarely used, but it is extremely critical.
Identify only the rarely occurring critical operations because
sufficient test cases will be allocated to the frequently
occurring critical operations because of their substantial
occurrence probabilities.
SENG521 (Fall 2002)
[email protected]
15
New Operations



In the case of a new release, set the allocation probabilities
for the new operations equal to the occurrence probabilities
of the system operational profile.
This is a satisfactory approximation; the sum of the
occurrence probabilities for the other new operations will be
very close to 1.
For subsequent releases, divide the system occurrence
probabilities for the other new operations by the total of the
system occurrence probabilities for the other new operations.
SENG521 (Fall 2002)
[email protected]
16
Infrequent Operations






Infrequent operations are those that would not normally be assigned test
cases by virtue of their occurrence probabilities.
Ensure that we assign each operation at least one test case.
The occurrence probability of an operation is often such that, because of
the fact that runs usually substantially outnumber test cases, the operation
might well invoke a run even though the occurrence probability might
not normally result in test case selection.
Thus determine the threshold occurrence probability below which you
would not normally assign a test case to an operation.
This is 0.5 divided by the number of new test cases.
We then pre-assign one test case to each other new operation whose
occurrence probability falls below the threshold.
SENG521 (Fall 2002)
[email protected]
17