Lecture 02 Overview of Software Testing
Download
Report
Transcript Lecture 02 Overview of Software Testing
CS4723
Software Validation
and Quality Assurance
Lecture 02
Overview of Software Testing
Approach to remove bugs
Testing
Limitations
Impossible to cover all cases
Test oracles (what is expected)
Static checking
2
Feed input to software and run it to see whether its behavior is
as expected
Identify specific problems (e.g., memory leak) in the software by
scanning the code or all possible paths
Limitations
Limited problem types
False positives
Approach to remove bugs
Formal Proof
3
Formally prove that the program implements the
specification
Limitations
Difficult to have a formal specification
The proof cost a lot of human efforts
Inspection
Manually review the code to detect faults
Limitations:
Hard to evaluate
Sometime hard to get progress
Answer is testing, why?
“50% of my employees are testers, and the rest
spends 50% of their time testing”
---- Bill Gates, 1995
More reliable than inspection, relatively cheap
Actually in the old days, when testing is expensive,
inspection was the major answer
You get what you pay (linear rewards)
Compared to other 3 approaches
4
Inspection, static checking, formal proof
Testing: Concepts
5
Test case
Test oracle
Test suite
Test script
Test driver
Test coverage
Testing: Concepts
Test case
Include:
Input values, sometimes fed in different steps
Expected outputs
Test oracle
6
An execution of the software with a given list of input
values
The expected outputs of software by feeding in a list of
input values
A part of test cases
Hardest problem in auto-testing: test oracle problem
Testing: Concepts: Example
7
Testing: Concepts
Test suite
A collection of test cases
Usually these test cases share similar pre-conditions and
configuration
Usually can be run together in sequence
Different test suites for different purposes
Test Script
8
Smoke test, Certain platforms, Certain feature,
performance, …
A script to run a sequence of test cases or a test suite
automatically
Testing: Concepts
Test Driver
9
A software framework that can load a collection of test
cases or a test suite
It can usually handle the configuration and comparison
between expected outputs and actual outputs
Test Coverage
A measurement to evaluate how well the testing is done
The measure can be based on multiple elements
Code
Input combinations
Specifications
Granularity of Testing: V-model
10
Granularity of testing
Unit / Module Testing
Integration Testing
Test the system as a whole, by developers on test cases
Acceptance Testing
11
Test the interaction between modules
System Testing
Test of a single module
Validate the system against user requirements, by
customers with no formal test cases
Stage of software Testing
Development-time testing
Before-release testing
System testing, Acceptance Testing
User testing
12
Unit testing, Integration Testing
Actual usage -> field bugs & patches
Types of testing by
how they are designed
13
White box testing
The tester knows everything about the implementation
They knows where the bugs are more probably be
They can exercise paths in the code
Black box testing
The tester are just like normal users
They just try to cover input space and corner cases
Black Box Testing: General Guidelines
Divide value range and cover each part
Cover boundary values
Try to reach all error messages
Try to trigger potential exceptions
Feed invalid inputs
14
wrong formats, too long, too short, empty, …
Try combinations of all above
Repeat same and use different inputs for many
times if the input is a sequence
Black Box Testing Techniques
15
Boundary value testing
Boundary value analysis
Robustness testing
Worst case testing
Equivalence class testing
Decision table based testing
Boundary Value Analysis
16
Errors tend to occur near the extreme values of
an input variables
Boundary value analysis focuses on the boundary of
the input space to identity test cases
Boundary value analysis selects input variable
values at their
Minimum
Just above the minimum
A nominal value
Just below the maximum
Maximum
Example of Boundary Value Analysis
Assume a program accepting two inputs y1 and y2,
such that a < y1< b and c < y2 < d
y2
d
c
..
.. ..
.
a
17
..
b
y1
Single Fault Assumption for
Boundary Value Analysis
Boundary value analysis is also augmented by the
single fault assumption principle
“Failures occur rarely as the result of the
simultaneous occurrence of two (or more) faults”
In this respect, boundary value analysis test cases
can be obtained by holding the values of all but one
variable at their nominal values, and letting that
variable assume its extreme values
18
Generalization of Boundary
Value Analysis
The basic boundary value analysis can be
generalized in two ways:
By the number of variables - (4n +1) test cases for
n variables
19
By the kinds of ranges of variables : map to order
Strings
Sequences
Complex Data Structures, e. g., trees
Application Scenario of
Boundary Value Analysis
Several independent variables that represent
bounded physical quantities
No consideration of the function of the program,
nor of the semantic meaning of the variables
Good news: We can distinguish between physical
and other types of variables
20
Robustness Testing
21
A simple extension of boundary value analysis
In addition to the five boundary value analysis
values of variables, we add values slightly greater
that the maximum (max+) and a value slightly less
than the minimum (min-)
The main value of robustness testing is to force
attention on exception handling
Example of Robustness Testing
y2
d
c
..
... ..
..
a
22
…
b
y1
UTSA CS3773
Worst Case Testing
No single fault assumption: error happens when
more than one variable has an extreme value
Considering n inputs in boundary analysis, we take
the Cartesian product of the five values for 1, 2, 3,
… n variables
We can have 5n test cases for n input variables
The more interactions on inputs -> more on worse
case testing
23
Input partitions: Length & Width vs. Length & price
Example of Worst Case Testing
y2
d
c
....
....
..
a
24
..
..
.
....
....
..
b
y1
Equivalence Class Testing
25
Divide the value range of an input to a number of
subsets
Subsets are disjoint
The union of the subset if the value range
Values in one subset does not make difference for
the software concerned
Water temp in a car: <32, 32 – 212, >=212
Normal colors vs. Metallic colors
Example of Equivalence Class Testing
26
Equivalence Class Testing
The use of equivalence class testing has two
motivations:
Sense of complete testing
Avoid redundancy
The disjointness assures a form of non-redundancy
Note
27
The entire set is represented provides a form of
completeness
Also check boundaries
Combinations of inputs also follows the rule: more
interaction -> more combinations
Equivalent Class for non-numeric inputs
Feature extraction
For string and structure inputs
Split the possible value set with a certain feature
Example:
String passwd => {contains space}, {no space}
It is possible to extract multiple features from one input
Example:
String name => {capitalized first letter}, {not}
=> {contains space}, {not}
=> {length >10}, {2-10}, {1}, {0}
One test case may cover multiple features
28
Decision Table
Make it easy to observe that all possible
conditions are accounted for
29
Decision tables can be used for:
Specifying complex program logic
Generating test cases with oracles
Example of Decision Table
Conditions
Printer does not print
Y
Y
Y
Y
N
N N
N
A red light is flashing
Y
Y
N
N
Y
Y N
N
Printer is unrecognized
Y
N
Y
N
Y
N Y
N
X
X
Check the power cable
Actions
Check the printer-computer cable
X
X
Ensure printer software is installed
X
X
Check/replace ink
X
Check for paper jam
30
X
Printer Troubleshooting
X
X
X
X
X
Decision Table Usage
The use of the decision-table model is applicable
when:
Specification is given or can be converted to a
decision table
The order in which the predicates are evaluated
does not affect the interpretation of resulting
action
Note:
31
Decision table needs not cover all combinations
White Box Testing: General Guidelines
Try to cover all branches
Test more on complex modules
32
Study the relationship between input value and
branch logic
Measure complexities of modules by code size,
number of branches and loops, number of calls
and recursions
White Box Testing: Techniques
33
More difficult than black box testing
Seldom done manually
Automatic support
Symbolic execution
Complexity measurement and Defect prediction
Review: Test overview
Test is the practical choice: the best
affordable approach
Concepts: test case, test oracle, test suite,
test driver, test script, test coverage
Granularity: unit, integration, system,
acceptance
Type by design principle: black-box, white-box
34
Black-box-testing: boundary, equivalence, decision
table
White-box-testing: branch coverage, complexity