Transcript Software Engineering

Software Engineering
Software Testing
The material is this presentation is based on the following
references and other internet resources:
•Ian Sommerville, Software Engineering (Seventh Edition),
Addison-Wesley, 2004.
•Roger Pressman, Software Engineering, A Practitioner
Approach, 6th ed., McGraw Hill, 2005.
Software Engineering
Software Testing
Slide 1
Objectives
• To discuss the distinctions between validation
testing and defect testing
• To describe the principles of system and
component testing
• To describe strategies for generating system
test cases
• To understand the essential characteristics of
tool used for test automation
Software Engineering
Software Testing
Slide 2
Topics covered
•
•
•
•
System testing
Component testing
Test case design
Test automation
Software Engineering
Software Testing
Slide 3
Testability
• Operability—it operates cleanly
• Observability—the results of each test case are
readily observed
• Controllability—the degree to which testing can be
automated and optimized
• Decomposability—testing can be targeted
• Simplicity—reduce complex architecture and logic to
simplify tests
• Stability—few changes are requested during testing
• Understandability—of the design
Software Engineering
Software Testing
Slide 4
What is a “Good” Test?
• A good test has a high probability of finding an error
• A good test is not redundant.
• A good test should be “best of breed”
• A good test should be neither too simple nor too complex
Software Engineering
Software Testing
Slide 5
What Testing Shows
errors
requirements conformance
performance
an indication
of quality
Software Engineering
Software Testing
Slide 6
Who Tests the Software?
Developer
independent tester
Understands the system but, will
test "gently“ and, is driven by
"delivery"
Must learn about the system, but,
will attempt to break it and, is
driven by quality
Software Engineering
Software Testing
Slide 7
Testing phases
Software Engineering
Software Testing
Slide 8
Testing Strategy
• We begin by ‘testing-in-the-small’ and move toward ‘testingin-the-large’
• For conventional software
– The module (component) is our initial focus
– Integration of modules follows
• For OO software
– our focus when “testing in the small” changes from an
individual module (the conventional view) to an OO class
that encompasses attributes and operations and implies
communication and collaboration
Software Engineering
Software Testing
Slide 9
Testing Strategy
integration
test
unit test
system
test
Software Engineering
validation
test
Software Testing
Slide 10
The testing process
• Component testing
– Testing of individual program components;
– Usually the responsibility of the component developer
(except sometimes for critical systems);
– Tests are derived from the developer’s experience.
• System testing
– Testing of groups of components integrated to create a
system or sub-system;
– The responsibility of an independent testing team;
– Tests are based on a system specification.
Software Engineering
Software Testing
Slide 11
Defect testing
• The goal of defect testing is to discover defects
in programs
• A successful defect test is a test which causes a
program to behave in an anomalous way
• Tests show the presence not the absence of
defects
Software Engineering
Software Testing
Slide 12
Testing process goals
• Validation testing
– To demonstrate to the developer and the system customer
that the software meets its requirements.
– A successful test shows that the system operates as
intended.
• Defect testing
– To discover faults or defects in the software where its
behavior is incorrect or not in conformance with its
specification.
– A successful test is a test that makes the system perform
incorrectly and so exposes a defect in the system.
Software Engineering
Software Testing
Slide 13
The software testing process
Software Engineering
Software Testing
Slide 14
Testing policies
• Only exhaustive testing can show a program is free
from defects. However, exhaustive testing is
impossible,
• Testing policies define the approach to be used in
selecting system tests:
– All functions accessed through menus should be tested;
– Combinations of functions accessed through the same
menu should be tested;
– Where user input is required, all functions must be tested
with correct and incorrect input.
Software Engineering
Software Testing
Slide 15
Exhaustive Testing
loop < 20 X
Software Engineering
Software Testing
Slide 16
Selective Testing
Selected path
loop < 20 X
Software Engineering
Software Testing
Slide 17
Component testing
• Component or unit testing is the process of
testing individual components in isolation.
• It is a defect testing process.
• Components may be:
– Individual functions or methods within an object;
– Object classes with several attributes and methods;
– Composite components with defined interfaces
used to access their functionality.
Software Engineering
Software Testing
Slide 18
Unit Testing
results
module
to be
tested
Software engineer
test cases
Software Engineering
Software Testing
Slide 19
Unit Testing
module
to be
tested
Interface
local data structures
boundary conditions
independent paths
error handling paths
results
test cases
Software Engineering
Software Testing
Slide 20
Object class testing
• Complete test coverage of a class involves
– Testing all operations associated with an object;
– Setting and interrogating all object attributes;
– Exercising the object in all possible states.
• Inheritance makes it more difficult to design
object class tests as the information to be
tested is not localised.
Software Engineering
Software Testing
Slide 21
Interface testing
• Objectives are to detect faults due to interface
errors or invalid assumptions about interfaces.
• Particularly important for object-oriented
development as objects are defined by their
interfaces.
Software Engineering
Software Testing
Slide 22
Interface testing
Software Engineering
Software Testing
Slide 23
Interface types
• Parameter interfaces
– Data passed from one procedure to another.
• Shared memory interfaces
– Block of memory is shared between procedures or
functions.
• Procedural interfaces
– Sub-system encapsulates a set of procedures to be called by
other sub-systems.
• Message passing interfaces
– Sub-systems request services from other sub-systems
Software Engineering
Software Testing
Slide 24
Interface testing guidelines
• Design tests so that parameters to a called procedure
are at the extreme ends of their ranges.
• Always test pointer parameters with null pointers.
• Design tests which cause the component to fail.
• Use stress testing in message passing systems.
• In shared memory systems, vary the order in which
components are activated.
Software Engineering
Software Testing
Slide 25
Integration Testing Strategies
Options:
• the “big bang” approach
• an incremental construction
strategy
Software Engineering
Software Testing
Slide 26
Integration testing
• Involves building a system from its components and
testing it for problems that arise from component
interactions.
• Top-down integration
– Develop the skeleton of the system and populate it with
components.
• Bottom-up integration
– Integrate infrastructure components then add functional
components.
• To simplify error localisation, systems should be
incrementally integrated.
Software Engineering
Software Testing
Slide 27
Top Down Integration
A
B
Software Engineering
G
stubs are replaced one at a time,
"depth first"
C
D
F
top module is tested with stubs
E
as new modules are integrated,
some subset of tests is re-run
Software Testing
Slide 28
Bottom-Up Integration
A
B
G
drivers are replaced one at a
time, "depth first"
C
D
F
E
worker modules are grouped into
builds and integrated
cluster
Software Engineering
Software Testing
Slide 29
Sandwich Testing
A
B
F
Top modules are
tested with stubs
G
C
D
E
Worker modules are grouped into
builds and integrated
cluster
Software Engineering
Software Testing
Slide 30
System testing
• Involves integrating components to create a system or
sub-system.
• May involve testing an increment to be delivered to
the customer.
• Two phases:
– Integration testing - the test team have access to the system
source code. The system is tested as components are
integrated.
– Release testing - the test team test the complete system to
be delivered as a black-box.
Software Engineering
Software Testing
Slide 31
Object-Oriented Testing
• begins by evaluating the correctness and consistency of the
OOA and OOD models
• testing strategy changes
– the concept of the ‘unit’ broadens due to encapsulation
– integration focuses on classes and their execution across a
‘thread’ or in the context of a usage scenario
– validation uses conventional black box methods
Software Engineering
Software Testing
Slide 32
OOT Strategy
• class testing is the equivalent of unit testing
– operations within the class are tested
– the state behavior of the class is examined
• integration applied three different strategies
– thread-based testing—integrates the set of classes
required to respond to one input or event
– use-based testing—integrates the set of classes
required to respond to one use case
– cluster testing—integrates the set of classes
required to demonstrate one collaboration
Software Engineering
Software Testing
Slide 33
Incremental integration testing
A
T1
A
T1
T2
A
T2
T2
B
T3
B
T3
B
C
T3
T4
C
T4
D
Test sequence 1
Software Engineering
T1
Test sequence 2
Software Testing
T5
Test sequence 3
Slide 34
Testing approaches
• Architectural validation
– Top-down integration testing is better at discovering errors
in the system architecture.
• System demonstration
– Top-down integration testing allows a limited
demonstration at an early stage in the development.
• Test implementation
– Often easier with bottom-up integration testing.
• Test observation
– Problems with both approaches. Extra code may be
required to observe tests.
Software Engineering
Software Testing
Slide 35
Black-box testing
Software Engineering
Software Testing
Slide 36
Testing guidelines
• Testing guidelines are hints for the testing team to
help them choose tests that will reveal defects in the
system
– Choose inputs that force the system to generate all error
messages;
– Design inputs that cause buffers to overflow;
– Repeat the same input or input series several times;
– Force invalid outputs to be generated;
– Force computation results to be too large or too small.
Software Engineering
Software Testing
Slide 37
Use cases
• Use cases can be a basis for deriving the tests
for a system. They help identify operations to
be tested and help design the required test
cases.
• From an associated sequence diagram, the
inputs and outputs to be created for the tests
can be identified.
Software Engineering
Software Testing
Slide 38
Test case design
• Involves designing the test cases (inputs and
outputs) used to test the system.
• The goal of test case design is to create a set of
tests that are effective in validation and defect
testing.
• Design approaches:
– Requirements-based testing;
– Partition testing;
– Structural testing.
Software Engineering
Software Testing
Slide 39
OOT—Test Case Design
Berard [BER93] proposes the following approach:
1. Each test case should be uniquely identified and should be explicitly
associated with the class to be tested,
2. The purpose of the test should be stated,
3. A list of testing steps should be developed for each test and should contain
[BER94]:
a list of specified states for the object that is to be tested
b. a list of messages and operations that will be exercised as a consequence
of the test
c. a list of exceptions that may occur as the object is tested
d. a list of external conditions (i.e., changes in the environment external to
the software that must exist in order to properly conduct the test)
e. supplementary information that will aid in understanding or
implementing the test.
Software Engineering
Software Testing
Slide 40
Test Case Design
OBJECTIVE to uncover errors
CRITERIA in a complete manner
CONSTRAINT with a minimum of effort and time
Software Engineering
Software Testing
Slide 41
Requirements based testing
• A general principle of requirements
engineering is that requirements should be
testable.
• Requirements-based testing is a validation
testing technique where you consider each
requirement and derive a set of tests for that
requirement.
Software Engineering
Software Testing
Slide 42
Software Testing
black-box
methods
white-box
methods
Methods
Strategies
Software Engineering
Software Testing
Slide 43
Black-Box Testing
requirements
output
input
Software Engineering
events
Software Testing
Slide 44
Black-Box Testing
•
•
•
•
How is functional validity tested?
How is system behavior and performance tested?
What classes of input will make good test cases?
Is the system particularly sensitive to certain input
values?
• How are the boundaries of a data class isolated?
• What data rates and data volume can the system
tolerate?
• What effect will specific combinations of data have
on system operation?
Software Engineering
Software Testing
Slide 45
Equivalence Partitioning
user
queries
Software Engineering
mouse
picks
FK
input
output
formats
prompts
Software Testing
data
Slide 46
Boundary Value Analysis
user
queries
mouse
picks
FK
input
output
formats
prompts
input domain
Software Engineering
Software Testing
data
output
domain
Slide 47
Partition testing
• Input data and output results often fall into
different classes where all members of a class
are related.
• Each of these classes is an equivalence
partition or domain where the program
behaves in an equivalent way for each class
member.
• Test cases should be chosen from each
partition.
Software Engineering
Software Testing
Slide 48
Equivalence partitioning
Software Engineering
Software Testing
Slide 49
Equivalence partitions
Software Engineering
Software Testing
Slide 50
Search routine specification
procedure Search (Key : ELEM ; T: SEQ of ELEM;
Found : in out BOOLEAN; L: in out ELEM_INDEX) ;
Pre-condition
-- the sequence has at least one element
T’FIRST <= T’LAST
Post-condition
-- the element is found and is referenced by L
( Found and T (L) = Key)
or
-- the element is not in the array
( not Found and
not (exists i, T’FIRST >= i <= T’LAST, T (i) = Key ))
Software Engineering
Software Testing
Slide 51
Search routine - input partitions
• Inputs which conform to the pre-conditions.
• Inputs where a pre-condition does not hold.
• Inputs where the key element is a member of
the array.
• Inputs where the key element is not a member
of the array.
Software Engineering
Software Testing
Slide 52
OOT Methods: Random Testing
• Random testing
–
–
–
–
identify operations applicable to a class
define constraints on their use
identify a minimum test sequence
generate a variety of random (but valid) test
sequences
Software Engineering
Software Testing
Slide 53
OOT Methods: Partition Testing
• Partition Testing
– reduces the number of test cases required to test a class in
much the same way as equivalence partitioning for
conventional software
– state-based partitioning
• categorize and test operations based on their ability to change the
state of a class
– attribute-based partitioning
• categorize and test operations based on the attributes that they use
– category-based partitioning
• categorize and test operations based on the generic function each
performs
Software Engineering
Software Testing
Slide 54
Sample Equivalence Classes
• Valid data
–
–
–
•
user supplied commands
responses to system prompts
file names
computational data
• physical parameters
• bounding values
• initiation values
• output data formatting
• responses to error messages
• graphical data (e.g., mouse picks)
• Invalid data
– data outside bounds of the program
– physically impossible data
– proper value supplied in wrong place
Software Engineering
Software Testing
Slide 55
Testing guidelines (sequences)
• Test software with sequences which have only
a single value.
• Use sequences of different sizes in different
tests.
• Derive tests so that the first, middle and last
elements of the sequence are accessed.
• Test with sequences of zero length.
Software Engineering
Software Testing
Slide 56
Search routine - input partitions
Sequence
Single value
Single value
More than 1 value
More than 1 value
More than 1 value
More than 1 value
Input sequence (T)
17
17
17, 29, 21, 23
41, 18, 9, 31, 30, 16, 45
17, 18, 21, 23, 29, 41, 38
21, 23, 29, 33, 38
Software Engineering
Element
In sequence
Not in sequence
First element in sequence
Last element in sequence
Middle eleme nt in sequence
Not in sequence
Key (Key)
17
0
17
45
23
25
Output (Found, L)
true, 1
false, ? ?
true, 1
true, 7
true, 4
false, ? ?
Software Testing
Slide 57
Structural testing
• Sometime called white-box testing.
• Derivation of test cases according to program
structure. Knowledge of the program is used to
identify additional test cases.
• Objective is to exercise all program statements
(not all path combinations).
Software Engineering
Software Testing
Slide 58
Structural testing
Software Engineering
Software Testing
Slide 59
Binary search - equiv. partitions
• Pre-conditions satisfied, key element in array.
• Pre-conditions satisfied, key element not in
array.
• Pre-conditions unsatisfied, key element in array.
• Pre-conditions unsatisfied, key element not in array.
• Input array has a single value.
• Input array has an even number of values.
• Input array has an odd number of values.
Software Engineering
Software Testing
Slide 60
Binary search equiv. partitions
Software Engineering
Software Testing
Slide 61
Binary search - test cases
Input array (T)
17
17
17, 21, 23, 29
9, 16, 18, 30, 31, 41, 45
17, 18, 21, 23, 29, 38, 41
17, 18, 21, 23, 29, 33, 38
12, 18, 21, 23, 32
21, 23, 29, 33, 38
Software Engineering
Key (Key)
17
0
17
45
23
21
23
25
Output (Found, L)
true, 1
false, ? ?
true, 1
true, 7
true, 4
true, 3
true, 4
false, ? ?
Software Testing
Slide 62
White-Box Testing
... our goal is to ensure that all
statements and conditions have
been executed at least once ...
Software Engineering
Software Testing
Slide 63
Path testing
• The objective of path testing is to ensure that
the set of test cases is such that each path
through the program is executed at least once.
• The starting point for path testing is a program
flow graph that shows nodes representing
program decisions and arcs representing the
flow of control.
• Statements with conditions are therefore nodes
in the flow graph.
Software Engineering
Software Testing
Slide 64
Binary search flow graph
Software Engineering
Software Testing
Slide 65
Independent paths
•
•
•
•
1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 14
1, 2, 3, 4, 5, 14
1, 2, 3, 4, 5, 6, 7, 11, 12, 5, …
1, 2, 3, 4, 6, 7, 2, 11, 13, 5, …
• Test cases should be derived so that all of
these paths are executed
• A dynamic program analyser may be used to
check that paths have been executed
Software Engineering
Software Testing
Slide 66
Why Cover?
• logic errors and incorrect assumptions are
inversely proportional to a path's execution
probability
• we often believe that a path is not likely to be
executed; in fact, reality is often counter
intuitive
• typographical errors are random; it's likely
that untested paths will contain some
Software Engineering
Software Testing
Slide 67
Basis Path Testing
First, we compute the cyclomatic
complexity:
number of simple decisions + 1
or
number of enclosed areas + 1
In this case, V(G) = 4
Software Engineering
Software Testing
Slide 68
Cyclomatic Complexity
A number of industry studies have indicated that the higher V(G),
the higher the probability or errors.
modules
V(G)
modules in this range are
more error prone
Software Engineering
Software Testing
Slide 69
Basis Path Testing
Next, we derive the
independent paths:
1
Since V(G) = 4,
there are four paths
2
3
4
5
7
8
Software Engineering
6
Path 1:
Path 2:
Path 3:
Path 4:
1,2,3,6,7,8
1,2,3,5,7,8
1,2,4,7,8
1,2,4,7,2,4,...7,8
Finally, we derive test
cases to exercise these
paths.
Software Testing
Slide 70
Basis Path Testing Notes
you don't need a flow chart,
but the picture will help when
you trace program paths
count each simple logical test,
compound tests count as 2 or
more
basis path testing should be
applied to critical modules
Software Engineering
Software Testing
Slide 71
Control Structure Testing
• Condition testing — a test case design method
that exercises the logical conditions contained
in a program module
• Data flow testing — selects test paths of a
program according to the locations of
definitions and uses of variables in the
program
Software Engineering
Software Testing
Slide 72
Loop Testing
Simple
loop
Nested
Loops
Concatenated
Loops
Software Engineering
Software Testing
Unstructured
Loops
Slide 73
Loop Testing: Simple Loops
• Minimum conditions—Simple Loops
• skip the loop entirely
• only one pass through the loop
• two passes through the loop
• m passes through the loop m < n
• (n-1), n, and (n+1) passes through the loop
where n is the maximum number of allowable
passes
Software Engineering
Software Testing
Slide 74
Loop Testing: Nested Loops
Nested Loops
Start at the innermost loop. Set all outer loops to their
minimum iteration parameter values.
Test the min+1, typical, max-1 and max for the
innermost loop, while holding the outer loops at their
minimum values.
Move out one loop and set it up as in step 2, holding all
other loops at typical values. Continue this step until
the outermost loop has been tested.
Software Engineering
Software Testing
Slide 75
Loop Testing: Nested Loops (cont.)
Concatenated Loops
If the loops are independent of one another
then treat each as a simple loop
else* treat as nested loops
endif*
Software Engineering
Software Testing
Slide 76
Release testing
• The process of testing a release of a system that will
be distributed to customers.
• Primary goal is to increase the supplier’s confidence
that the system meets its requirements.
• Release testing is usually black-box or functional
testing
– Based on the system specification only;
– Testers do not have knowledge of the system
implementation.
Software Engineering
Software Testing
Slide 77
Performance testing
• Part of release testing may involve testing the
emergent properties of a system, such as
performance and reliability.
• Performance tests usually involve planning a
series of tests where the load is steadily
increased until the system performance
becomes unacceptable.
Software Engineering
Software Testing
Slide 78
Stress testing
• Exercises the system beyond its maximum design
load. Stressing the system often causes defects to
come to light.
• Stressing the system test failure behaviour.. Systems
should not fail catastrophically. Stress testing checks
for unacceptable loss of service or data.
• Stress testing is particularly relevant to distributed
systems that can exhibit severe degradation as a
network becomes overloaded.
Software Engineering
Software Testing
Slide 79
Test automation
• Testing is an expensive process phase. Testing
workbenches provide a range of tools to reduce the
time required and total testing costs.
• Systems such as Junit support the automatic
execution of tests.
• Most testing workbenches are open systems because
testing needs are organisation-specific.
• They are sometimes difficult to integrate with closed
design and analysis workbenches.
Software Engineering
Software Testing
Slide 80
A testing workbench
Software Engineering
Software Testing
Slide 81
Key points
• Testing can show the presence of faults in a system; it
cannot prove there are no remaining faults.
• Component developers are responsible for component
testing; system testing is the responsibility of a
separate team.
• Integration testing is testing increments of the system;
release testing involves testing a system to be
released to a customer.
• Use experience and guidelines to design test cases in
defect testing.
Software Engineering
Software Testing
Slide 82
Key points
• Interface testing is designed to discover defects in the
interfaces of composite components.
• Equivalence partitioning is a way of discovering test
cases - all cases in a partition should behave in the
same way.
• Structural analysis relies on analysing a program and
deriving tests from this analysis.
• Test automation reduces testing costs by supporting
the test process with a range of software tools.
Software Engineering
Software Testing
Slide 83