Transcript PowerPoint Sunusu

Software Testing
Mistake in coding is called error ,
Error found by tester is called defect,
Defect accepted by development team is called bug ,
Product does not meet the requirements then it Is failure.”
The Importance of Test
There are hundreds of stories about failures of
computer systems that have been attributed to errors
in software
There are many reasons why systems fail but the issue
that stands out the most is the lack of adequate
testing.
Example Pepsi - $42 Billion Error Philippines -1992
Due to a software error, 800,000 bottle caps were produced with number 349
instead of one.
It was equivalent to $32 billion in prize money instead of $40,000
Testing Levels
Testing Levels
Unit
Unit
…..
Module
Unit
…..
Build
Each of testing activities is performed
during the process of building an
application
…..
Module
Interface testing
…..
Unit
Unit
…..
Build
Module
Application
on test bed
Application
on final
platform
system testing
usability testing
installation testing
acceptance testing
integration testing
Unit
Regression testing throughout
4
Testing Goals Based on Test Process
Testing Levels (Bezier)
Level 0 There is no difference between testing and debugging
Level 1 The purpose of testing is to show the correctness
(software works).
Level 2 The purpose of testing is to show that the software
doesn’t work.(the purpose is to show failures)
Level 3 Testing shows the presence , not absence or failure. If we
use software, we expose some risk. The risk may be small and
unimportant, or the risk may be great and the consequences
catastrophic, but risk is always there
(The purpose of testing is to reduce the risk of software ; both testers and
developers work together to reduce risk.
Level 4 The purpose of testing is to improve the ability of the
developers to produce high quality software.
(Level 4 testing shows that testers and developers are on the same team)
Debugging and Testing
Debugging is done in the development phase
by the developer
In debugging phase identified bugs are fixed
 Testing is done in the tester phase by the
tester.
In testing, locating and identifying of the bugs are
included.
Software Bug
A software bug is an error, flaw, mistake,
failure, or fault in a computer program or
system
Bug is terminology of Tester
Software bug produces an incorrect or
unexpected result, or causes it to behave in
unintended ways
Software Fault
A static (physical)defect*, imperfection, flaw in
the software.
short between wires
break in transistor
infinite program loop
A fault means that there is a problem within
the code of a program which causes it to
behave incorrectly.
*defect: Mismatch between the requirements
Software Error
Error is an incorrect internal state that is the
demonstration of some fault.
In other words; Error is a deviation from
correctness or accuracy.
Example
 Suppose a line is physically shortened to 0. (there is a
fault).
As long as the value on line is supposed to be 0, there is no
error.
Errors are usually associated with incorrect values in
the system state.
Software Failure
A failure means that the program has not
performed as expected, and the actual
performance has not met the minimum standards
for success.
External, incorrect behavior with respect to the
requirements or other description of the expected
behavior
Example
Suppose a circuit controls a lamp (0 = turn off, 1 = turn
on) and the output is physically shortened to 0 (there
is a fault).
As long as the user wants the lamp off, there is no
failure.
Example: Failure & Fault & Error
Consider a medical doctor making a diagnosis for
a patient.
The patient enters the doctor’s office with a list
of failures (that is, symptoms).
The doctor then must discover the fault, or
root cause of the symptom.
To aid in the diagnosis, a doctor may order tests
that look for anomalous internal conditions.
 In our terminology, these anomalous internal
conditions correspond to errors.
Cause-and-Effect Relationship
Faults can result in errors.
Errors can lead to system failures
Errors are the effect of faults.
Failures are the effect of errors
Bug in a program is a fault.
Possible incorrect values caused by this bug is
an error.
Possible crush of the operating system is a
failure
Program Example
class numZero {
public static int numZero (int[] x)
{
// if x == null throw NullPointerException
// else return the number of occurrences of 0 in x
int count = 0;
for (int i = 1; i < x.length; i++)
{
if (x[i] == 0)
{
count++;
}
}
return
count; } }
The Fault ,Error and Failure
in the Example
The fault is that the program starts looking for zeroes at
index 1 instead of index 0
For example, numZero ([2, 7, 0]) correctly results with 1,
while numZero ([0, 7, 2]) incorrectly results with 0.
In both of these cases the fault is executed.
Both of the cases result in an error (because of next
slide)
But the first case results with correct value.
 Only the second case results in failure.
Error States in the Example
To understand the error states, we need to
identify the state for the program.
The state for numZero consists of values for the
variables x, count , i , program counter (PC).
For the first case: The state at the if statement on the
first iteration ( x = [2, 7, 0], count = 0, i = 1, PC = if)
This state is in error because the value of i should be
zero on the first iteration.
But the value of count is correct, the error state does not
propagate to the output, and the software does not fail.
In other words, a state is in error if it is not the expected
state, even if all of the values in the state are acceptable.
Error States in the Example
In the second case the corresponding (error) state is
(x = [0, 7, 2], count = 0, i = 1, PC = if).
In this case, the error propagates to the variable count
and is present in the return value of the method.
Hence a failure results.
Distinguish Testing from Debugging
The definitions of fault and failure allow us to
distinguish testing from debugging.
The difference is that debugging is conducted
by a programmer and the programmer fix the
errors during debugging phase.
Tester never fixes the errors, but rather find
them and return to programmer.
Testing versus Debugging
Testing activity is carried down by a team of testers, in
order to find the defect* in the software.
 Testers run their tests on the piece of software and if they
encounter any defect,they report it to the development
team.
 Testers also have to report at what point the defect
occurred and what happened due the occurrence of that
defect.
 All this information is used by development team to
DEBUG the defect.
 After finding out the bug, he tries to modify that portion of
code and then he rechecks if the defect has been removed
 After fixing the bug, developers send the software back to
testers.
*actual results don't match expected results
What is Software Quality ?
According to the IEEE Software Quality is:
The degree to which a system, component, or
process meets specified requirements.
The degree to which a system, component, or
process meets customer or user needs or
expectations.
Software Quality Factors
Software Quality Assurance
 Verification
– are
we building the product right ?
– performed at the end of a phase to
ensure that requirements established
during previous phase have been
met
 Validation
– are
we building the right product ?
– performed at the end of the development
process to ensure compliance with product
requirements
Verification & Validation
Verification: The process of determining
whether the products of a given phase of the
software development process fulfill the
requirements established during the previous
phase.
Validation: The process of evaluating software
at the end of software development to ensure
compliance with intended usage.
Difference Between
Verification &Validation- I
Verification is preventing mechanism to detect
possible failures before the testing begin.
It involves reviews, meetings, evaluating
documents, plans, code, inspections,
specifications etc.
Validation occurs after verification and it's the
actual testing to find defects against the
functionality or the specifications
The Difference between
Verification &Validation- II
Verification is usually a more technical activity
that uses knowledge about the individual
software artifacts, requirements, and
specifications.
Validation usually depends on domain
knowledge; that is, knowledge of the application
for which the software is written.
For example, validation of software for an airplane
requires knowledge from aerospace engineers and
pilots.
Software Testing Types
Black box testing : You don't need to know the
internal design in detail or have a good knowledge
about the code for this test.
It's mainly based on functionality and specifications,
requirements.
White box testing : This test is based on detailed
knowledge of the internal design and code.
Tests are performed for specific code statements and
coding styles.
Traditional Testing Levels
main Class P
Class A

Acceptance testing : Is
the software acceptable
to the user?

System testing : Test the
overall functionality of
the system
Class B
method mA1()
method mB1()
method mA2()
method mB2()

Integration testing : Test
how modules interact
with each other

Module testing
(developer testing) : Test
each class, file, module
or component

Unit testing (developer
testing) : Test each unit
(method) individually
26
Object-Oriented Testing Levels

Class A
Inter-class testing : Test
multiple classes
together
Class B
method mA1()
method mB1()
method mA2()
method mB2()

Intra-class testing : Test
an entire class as
sequences of calls

Inter-method testing :
Test pairs of methods in
the same class

Intra-method testing :
Test each method
individually
27
Coverage Criteria
• Even small programs have too many inputs to fully test them all
–
–
–
–
private static computeAverage (int A, int B, int C)
On a 32-bit machine, each variable has over 4 billion possible values
Over 80 octillion possible tests!!
Input space might as well be infinite
• Testers search a huge input space
– Trying to find the fewest inputs that will find the most problems
• Coverage criteria give structured, practical ways to search the input
space
– Search the input space thoroughly
– Not much overlap in the tests
28
Test Requirements and Criteria
Test Criterion : A collection of rules and a
process that define test requirements
̶ Cover every statement
̶ Cover every functional requirement
Test Requirements : Specific things that must
be satisfied or covered during testing
– Each statement is a test requirement
Testing
researchers
haverequirement
defined dozensisof
but they are
– Each
functional
a criteria,
test requirement
all really just a few criteria on four types of structures …
1.
Graphs
2.
Logic expressions
3.
Input domains
4.
Syntax descriptions
29
Coverage Criterion
A coverage criterion is simply a recipe for
generating test requirements in a systematic
way:
Coverage Criterion: A coverage criterion is a rule
or collection of rules that impose test
requirements on a test set.
30
Structural Coverage Criteria -I
Node Coverage (Formal Definition): For each node n ∈ reachG(N0),
TR contains the predicate “visit n.”
Node Coverage (NC): TR contains each reachable node in G.
0
1
2
Node Coverage (NC) (Standard Definition) Test set T
satisfies node coverage on graph G if and only if for
every syntactically reachable node n in N, there is
some path p in path(T) such that p visits n.
TR = { 0, 1, 2 }
path(t1) = [0, 1 , 2] T1={t1}
T1 satisfies node coverage on the graph
31
Structural Coverage Criteria - II
Edge coverage is slightly stronger than node coverage
Edge Coverage (EC) : TR contains each reachable
path of length up to 1, inclusive, in G.
 The “length up to 1” allows for graphs with one node and no edges
Edge Coverage : TR = { (0,1), (0, 2), (1, 2) }
path (t1 ) = [ 0,1,2]
path (t2 ) = [ 0, 2 ]
T2={t1, t2} Test Paths = [ 0, 1, 2 ] [ 0, 2 ]
T2 satisfies edge coverage on the graph
0
1
2
32
Edge-Pair Coverage
TR = { [0,1,2], [0,2,3], [0,2,4], [1,2,3], [1,2,4], [2,3,6],
[2,4,5], [2,4,6], [4,5,4], [5,4,5], [5,4,6] }
0
path (t1 )= [ 0, 1, 2, 3, 6 ]
path (t2 )= [ 0, 1, 2, 4, 6 ]
path (t3 )=[ 0, 2, 3, 6 ]
path (t4 )= [ 0, 2, 4, 5, 4, 5, 4, 6 ]
1
2
3
4
5
Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 1, 2, 4, 6 ] [ 0, 2, 3, 6 ]
[ 0, 2, 4, 5, 4, 5, 4, 6 ]
6
T={t1 , t2 , t3 , t4 }
33
Graph Coverage for Source Code
Graph : Usually the control flow graph (CFG)
Node coverage : Execute every statement
Edge coverage : Execute every branch
Loops : Looping structures such as for loops, while
loops, etc.
Data flow coverage : Augment the (CFG) control flow
graph
– Defs are statements that assign values to variables
– Uses are statements that use variables
34
Control Flow Graphs : The if Statement
if (x < y)
{
y = 0;
x = x + 1;
}
else
{
x = y;
}
1
x<y
y=0
x=x+1
x >= y
2
3
x=y
4
if (x < y)
{
y = 0;
x = x + 1;
}
1
x<y
y=0
x=x+1
2
x >= y
3
35
Control Flow Graphs : while and for Loops
x = 0;
while (x < y)
{
y = f (x, y);
x = x + 1;
}
1
x=0
for (x = 0; x < y; x++)
{
y = f (x, y);
}
2
x<y
x3< y
x >= y
x4>= y
y =f(x,y)
x=x+1
1
2
x<y
y = f (x, y)
3
4
x >= y
5
x=x+1
36
Control Flow Graphs : do Loop, break
and continue 1
x=0
x = 0;
do
{
y = f (x, y);
x = x + 1;
} while (x < y);
println (y)
x=0
1
2
x >= y
3
y = f (x, y)
x = x+1
x<y
x = 0;
while (x < y)
{
y = f (x, y);
if (y == 0)
{
break;
} else if y < 0)
{
y = y*2;
continue;
}
x = x + 1;
}
print (y);
2
3
y =f(x,y)
y == 0
4
break
5
y<0
6
7
y = y*2
continue
x=x+1
8
37
The Application of Graph Criteria:
Source Code
 It is usually defined with the control flow graph
(CFG)
 Node coverage is used to execute every
statement
 Edge coverage is used to execute every branch
 Loops are used to execute looping structures
such as for loops, while loops, etc.
 Data flow coverage is the extended form of
control flow graph (CFG) with defs and uses
 defs are statements that assign values to variables
 uses are statements that use variables
public static void computeStats (int [ ] numbers)
{
int length = numbers.length;
double med, var, sd, mean, sum, varsum;
sum = 0;
for (int i = 0; i < length; i++)
{
sum += numbers [ i ];
}
med = numbers [ length / 2];
mean = sum / (double) length;
1
2
i=0
3
varsum = 0;
for (int i = 0; i < length; i++)
4
{
5
varsum = varsum + ((numbers [ I ] - mean) * (numbers [ I ] - mean));
i=0
}
var = varsum / ( length - 1.0 );
sd = Math.sqrt ( var );
6
System.out.println ("length:
" + length);
i < length
System.out.println ("mean:
" + mean);
i >= length
System.out.println ("median:
" + med);
7
System.out.println ("variance:
" + var);
8
System.out.println ("standard deviation: " + sd); }
i++
39
Post-Unit Testing
Unit
Unit
…..
Each of post-unit testing activities is performed during
The process of building an application
Module
Unit
…..
Build
…..
Module
Interface testing
…..
Unit
Unit
…..
Build
Module
Application
on test bed
Application
on final
platform
system testing
usability testing
installation testing
acceptance testing
integration testing
Unit
Regression testing throughout
40
How to select input values?
Scenario I
-
-
-
Random values
- For each input parameter we randomly select the
values
Tester Experience
- For each input we use our experience to select
relevant values to test
Domain knowledge
- We use requirements information or domain
knowledge information to identify relevant values for
inputs
41
How to select input values?
Scenario II
•
Equivalence classes (for black-box testing)
-
-
We subdivide the input domain into a small number of sub-domains
The equivalence classes are created assuming that they exhibits the same
behavior on all elements
Few values for each classes can be used for our testing
Boundary values (for black-box testing)
–
–
Is a test selection technique that targets faults in applications at the
“boundaries” of equivalence classes
Experience indicates that programmers make mistakes in processing values
at and near the boundaries of equivalence classes
42
Black-Box Testing
Example:
Person a rents « the Matrix» on May5
Person b rents «Gone with the Wind» on May 6
Person a returns «the Matrix» on May 10
 It does not take the manner in which the application
was designed or implemented
 This is analogous to building an automobile and then
testing it by driving it under various conditions.
BUT
If the car is built with a new design for its automatic transmission ,
we would be wise to use this knowledge
White box testing is required
White box tests are based on design and
implementation
Parameter Space of computeFine()
Each element represents a pair: (Days late, Movie name)
new releases
old releases
one-of-a-kind
The penalty also depends -5
on the whether the movie
is a «new» or «old» release or
«one of a kind.»
0
5
10
15
Days late
e.g., (6 days late, “Casablanca”)
44