Transcript CSEB233 Fundamentals of Software Engineering Module 6: Software Verification and Validation Badariah Solemon 2011

CSEB233 Fundamentals
of Software Engineering
Module 6: Software Verification and Validation
Badariah Solemon 2011
Objectives
• To introduce the fundamental concepts of software
verification and validation
• To explain how to conduct software testing and when
to stop
• To discuss several types of testing: unit testing,
integration testing, validation testing, and system
testing
• To introduce the standard for software test
documentation
• To introduce a set of techniques for the creation of
test cases that meet overall testing objectives and the
testing strategies
Badariah Solemon 2011
Verification & Validation
• Is a whole software engineering process
– V & V must be applied at each framework
activity in the software process.
– Verification refers to the set of tasks that ensure
that software correctly implements a specific
function.
– Validation refers to a different set of tasks that
ensure that the software that has been built is
traceable to customer requirements. Boehm
[Boe81] states this another way:
• Verification: "Are we building the product right?"
• Validation: "Are we building the right product?”
Badariah Solemon 2011
Verification & Validation(cnt’d)
• Has two principal objectives:
1. Discover defects in a system;
2. Assess whether or not the system is useful and
useable in an operational situation.
• V&V should establish confidence that the
software is fit for purpose.
– This does NOT mean completely free of defects.
– Rather, it must be good enough for its intended
use and the type of use will determine the degree
of confidence that is needed.
Badariah Solemon 2011
Verification & Validation(cnt’d)
• V & V activities include:
–
–
–
–
–
–
–
–
–
–
–
–
Technical reviews
Quality and configuration audits
Performance monitoring
Simulation
Feasibility study
Documentation review
Database review
Algorithm analysis
Development testing
Qualification testing
Acceptance testing
Installation testing
SQA
activities*
Testing
Badariah Solemon 2011
Software Testing
• The process of exercising a program with the
specific intent of finding errors prior to delivery to
the end user.
• Must be planned carefully to avoid wasting
development time and resources, and conducted
systematically .
• What testing shows?
errors
requirements conformance
performance
an indication
of quality
Badariah Solemon 2011
Who Tests the Software?
developer
Understands the system
but, will test "gently"
and, is driven by "delivery"
independent tester
Must learn about the system,
but, will attempt to break it
and, is driven by quality
Badariah Solemon 2011
Who Tests the Software?(cnt’d)
• Misconceptions :
– The developer should do no testing at all
– Software should be “tossed over the wall” to stranger who
will test it mercilessly
– Testers are not involved with the project until it is time for it
to be tested
• The developer and Independent Test Group (ITG)
must work together throughout the software
project to ensure that thorough tests will be
conducted.
– An ITG does not have the “conflict of interest” that the
software developer might experience.
– While testing is conducted, the developer must be available
to correct errors that are uncovered.
Badariah Solemon 2011
Testing Strategy
• Identifies steps to be undertaken, when
these steps are undertaken, how much
effort, time, and resources will be required.
• Any testing strategy must incorporate:




Test planning
Test case design
Test execution
Resultant data collection and evaluation
• Should provide guidance for the
practitioners and a set of milestones for the
manager.
Badariah Solemon 2011
Testing Strategy (cnt’d)
• Generic characteristics of software testing
strategies that have been proposed in the
literature:
1.
2.
3.
4.
5.
To perform effective testing, you should conduct effective
technical reviews.
• By doing this, many errors will be eliminated before testing
commences.
Testing begins at the component level and works "outward"
toward the integration of the entire computer-based system.
Different testing techniques are appropriate for different software
engineering approaches and at different points in time.
Testing is conducted by the developer of the software and (for
large projects) an independent test group.
Testing and debugging are different activities, but debugging must
be accommodated in any testing strategy.
Badariah Solemon 2011
Overall Software Testing
Strategy
• Maybe viewed in the context of the
spiral.
• Begins by ‘testing-in-the-small’ and
move toward ‘testing-in-the-large’
System engineering
Analysis modeling
Design modeling
Code generation
Unit testing
Integration testing
Validation testing
System testing
Badariah Solemon 2011
Overall Software Testing
Strategy (cnt’d)
• Unit Testing
– focuses on each unit of the software (e.g., component, module,
class ) as implemented in source code
• Integration Testing
– focuses on issues associated with verification and program
construction as components begin interacting with one another
• Validation Testing
– provides assurance that the software validation criteria
(established during requirements analysis) meets all functional,
behavioral, and performance requirements
• System Testing
– verifies that all system elements mesh properly and that overall
system function and performance has been achieved
Badariah Solemon 2011
Unit Testing
• Focuses on assessing:
— internal processing logic and data structures
within the boundaries of a component (module).
— proper information flow of module interfaces.
— local data to ensure that integrity is maintained.
— boundary conditions.
— basis (independent) path.
— all error handling paths.
•
If resources are scarce to do
comprehensive unit testing, select critical
or complex module and unit test only
those.
Badariah Solemon 2011
Unit Testing (cnt’d)
module
to be
tested
software
engineer
results
interface
local data structures
boundary conditions
independent paths
error handling paths
test cases
Badariah Solemon 2011
Integration Testing
• After unit testing of individual modules the
modules are combined together into a
system.
• Question commonly asked once all
modules have been unit tested:
–
“If they work individually, why do you doubt
that they’ll work when we put them
together?”
• The problem is “putting them together” –
interfacing. Examples:
–
–
–
Data can be loss across an interface
Global data structures can present problems
Subfunctions, when combined, may not
produce the desired function
Badariah Solemon 2011
Integration Testing (cnt’d)
• Incremental integration testing strategies:
1.
2.
3.
4.
Bottom-up integration
Top – down integration
Regression testing
Smoke testing
Badariah Solemon 2011
Bottom-up Integration
• An approach where the lowest level modules are tested first,
then used to facilitate the testing of higher level modules.
– The process is repeated until the module at the top of the hierarchy
is tested.
– Top level modules are the most important yet tested last.
• Is helpful only when all or most of the modules of the same
development level are ready.
Badariah Solemon 2011
Bottom-up Integration (cnt’d)
The steps:
A
1.
2.
B
F
G
drivers are replaced one at a
time, "depth first"
C
‘worker modules’ are grouped into
builds and integrated
D
E
cluster
3.
4.
Test D, E individually (using a
dummy program called ‘driver’)
Low-level components are
combined into clusters that
perform a specific software
function.

Test C such that it call D (If an
error occurs we know that the
problem is in C or in the
interface between C and D)

Test C such that it call E (If an
error occurs we know that the
problem is in C or in the
interface between C and E)
The cluster is tested.
Drivers are removed and clusters
are combined moving upward in
the program structure.
Badariah Solemon 2011
Top-down Integration
• The steps:
1.
2.
3.
4.
5.
Main/top module used as a test driver and stubs are substitutes
for modules directly subordinate to it.
Subordinate stubs are replaced one at a time with real modules
(following the depth-first or breadth-first approach).
Tests are conducted as each module is integrated.
On completion of each set of tests and other stub is replaced
with a real module.
Regression testing may be used to ensure that new errors not
introduced.
• The process continues from step 2 until the
entire program structure is built
Badariah Solemon 2011
Top-down Integration (cnt’d)
top module is
tested with stubs
A
Example steps:
1.
2.
B
F
G
stubs are replaced one at
a time, "depth first"
C
as new modules are integrated,
some subset of tests is re-run
D
E
Test A individually (use stubs for
other modules)
Depending on the integration
approach selected, subordinate
stubs are replaced one at a time
with actual components
 In a ‘depth-first’ structure:
a. Test A such that it calls B (use
stub for other modules)

If an error occurs we
know that the problem is
in B or in the interface
between A and B
b. Replace stubs one at a time,
‘depth-first’ and re-run tests
Badariah Solemon 2011
Regression Testing
• Focuses on retesting after changes are made
– Whenever software is corrected, some aspect of the software
configuration (the program, its documentation, or the data that
support it) is changed.
– Regression testing helps to ensure that changes (due to testing or
for other reasons) do not introduce unintended behavior or
additional errors.
• In traditional regression testing, we reuse the same tests.
• In risk-oriented regression testing, we test the same areas
as before, but we use different (increasingly complex)
tests.
• Regression testing may be conducted manually, by reexecuting a subset of all test cases or using automated
capture/playback tools.
Badariah Solemon 2011
Smoke Testing
• A common approach for creating “daily builds”
for product software
• Smoke testing steps:
– Software components that have been translated into code are integrated
into a “build.”
– A build includes all data files, libraries, reusable modules, and engineered
components that are required to implement one or more product
functions.
– A series of tests is designed to expose errors that will keep the build from
properly performing its function.
– The intent should be to uncover “show stopper” errors that have the
highest likelihood of throwing the software project behind schedule.
– The build is integrated with other builds and the entire product (in its
current form) is smoke tested daily.
– The integration approach may be top down or bottom up.
Badariah Solemon 2011
Validation Testing
• Focuses on uncovering errors at the
software requirements level.
• SRS might contain a ‘Validation Criteria’
that forms the basis for a validation-testing
approach
• Validation-Test Criteria:
–
Ensure that:





all functional requirements are satisfied,
all behavior characteristics are achieved,
all content is accurate and properly presented,
all performance requirements are attained,
documentation is correct, and
usability and other requirements are met.
Badariah Solemon 2011
Validation Testing (cnt’d)
• An important element of the validation process is a
configuration review/audit
– Ensure that all elements of the software configuration have been properly
developed, are cataloged, and have the necessary detail to strengthen the
support activities.
• A series of acceptance tests are conducted to enable
the customer to validate all requirements
– To make sure the software works correctly for intended user in
his or her normal work environment.
– Alpha test – version of the complete software is tested by
customer under the supervision of the developer at the
developer’s site
– Beta test – version of the complete software is tested by
customer at his or her own site without the developer being
present
Badariah Solemon 2011
System Testing
• A series of different tests to verify that
system elements have been properly
integrated and perform allocated functions.
• Types of system tests:
1.
2.
3.
4.
5.
Recovery Testing
Security Testing
Stress Testing
Performance Testing
Deployment Testing
Badariah Solemon 2011
System Testing (cnt’d)
1. Recovery Testing

forces the software to fail in a variety of ways and
verifies that recovery is properly performed
2. Security Testing

verifies that protection mechanisms built into a
system will, in fact, protect it from improper
penetration
3. Stress Testing

executes a system in a manner that demands
resources in abnormal quantity, frequency, or volume
Badariah Solemon 2011
System Testing (cnt’d)
4. Performance Testing

test the run-time performance of software within the
context of an integrated system
5. Deployment Testing


examines all installation procedures and specialized
installation software that will be used by customers,
and
all documentation that will be used to introduce the
software to end users.
Badariah Solemon 2011
Software Test Documentation
• IEEE 829 2008 Standard for Software Test
Documentation
–
IEEE standard that specifies the form of a set
of documents for use in eight defined stages
of software testing
• The documents are:
–
–
–
–
–
–
–
–
Test Plan
Test Design Specification
Test Case Specification
Test Procedure Specification
Test Item Transmittal Report
Test Log
Test Incident Report
Test Summary Report
Badariah Solemon 2011
Software Test Documentation
(cnt’d)
•
Test Plan: a management planning document that shows:
–
–
–
–
–
•
How the testing will be done (including SUT (system under test)
configurations).
Who will do it
What will be tested
How long it will take (although this may vary, depending upon
resource availability).
What the test coverage will be, i.e. what quality level is
required
Test Design Specification:
–
detailing test conditions and the expected results as well as test
pass criteria.
Badariah Solemon 2011
Software Test Documentation
(cnt’d)
•
Test Procedure Specification:
–
•
Test Item Transmittal Report:
–
•
reporting on when tested software components have
progressed from one stage of testing to the next
Test Log:
–
•
detailing how to run each test, including any set-up
preconditions and the steps that need to be followed
recording which tests cases were run, who ran them, in what
order, and whether each test passed or failed
Test Incident Report:
–
detailing, for any test that failed, the actual versus expected
result, and other information intended to throw light on why a
test has failed.
Badariah Solemon 2011
Software Test Documentation
(cnt’d)
•
Test Summary Report:
–
–
–
A management report providing any important information
uncovered by the tests accomplished, and including
assessments of the quality of the testing effort, the quality of
the software system under test, and statistics derived from
Incident Reports.
The report also records what testing was done and how long it
took, in order to improve any future test planning.
This final document is used to indicate whether the software
system under test is fit for purpose according to whether or not
it has met acceptance criteria defined by project stakeholders.
Badariah Solemon 2011
Test-case Design
• Focuses on a set of techniques for the
creation of test cases that meet overall
testing objectives and the testing
strategies.
• These techniques provide a systematic
guidance for designing tests that
1. Exercise the internal logic and interfaces of
every software component/module
2. Exercise the input and output domains of the
program to uncover errors in program
function, behaviour, and performance.
Badariah Solemon 2011
Test-case Design (cnt’d)
• For conventional application, software is tested
from two perspectives:
1. ‘White-box’ testing
–
–
–
focus on the program control structure (internal program
logic).
Test cases are derived to ensure that all statements in the
program have been executed at least once during testing
and all logical conditions have been exercised.
Performed early in the testing process
2. ‘Black-box’ testing
–
–
Examines some fundamental aspect of a system with little
regard for the internal logical structure of the software
Performed during later stages of testing
Badariah Solemon 2011
White-box Testing
• Using white-box testing method, you may
derive test-cases that:
1. Guarantee that al independent paths within
a module have been exercised at least once
2. Exercise all logical decisions on their true and
false sides
3. Execute all loops at their boundaries and
within their operational bounds
4. Exercise internal data structures to ensure
their validity
• Example method: basis path testing.
Badariah Solemon 2011
White-box Testing(cnt’d)
Basis path testing:
• Test cases derived to
exercise the basis set
are guaranteed to
execute every
statement in the
program at least one
time during testing
Badariah Solemon 2011
Deriving Test Cases
• Steps to derive the test cases (by applying
the basis path testing method):
1. Using the design or code, draw a
corresponding flow graph.


The flow graph depicts logical control flow using the
notation illustrated in next slide.
Refer Figure 18.2 in page 486 - comparison
between a flowchart and a flow graph
2. Calculate the Cyclometic Complexity V(G) of
the flow graph
3. Determine a basis set of independent paths
4. Prepare test cases that will force execution of
each path in the basis set.
Badariah Solemon 2011
Deriving Test Cases (cnt’d)
Flow graph notation:
UNTIL
I Sequence
F
IF
CASE
WHILE
Badariah Solemon 2011
Drawing Flow Graph: Example 1
void foo (float y, float a *, int n)
{
float x = sin (y) ;
if (x > 0.01) 1
z = tan (x) ; 2
else
z = cos (x) ; 3
1
2
Predicate
for (int i = 0 ; i < x ; + + i) 5 nodes
{
a[i] = a[i] * z ; 6
Cout < < a [i];
7
}
}
Predicate
nodes
R1
3
4
R3
5
R2
6
8
8
7
Badariah Solemon 2011
Drawing Flow Graph: Example 2
void foo (float y, float a *, int n)
{
float x = sin (y) ;
if (x > 0.01) 1
z = tan (x) ; 2
else
z = cos (x) ; 3
1
2
Predicate
for (int i = 0 ; i < x ; + + i) 5 nodes
{
a[i] = a[i] * z ; 6
Cout < < a [i];
7
}
}
Predicate
nodes
R1
3
4
R3
5
R2
6,7
8
8
Badariah Solemon 2011
Drawing Flow Graph: Example 3
See Figure 18.2 in page 486
Badariah Solemon 2011
Deriving Test Cases (cnt’d)
• The arrows on the flow
graph, called edges or
links, represent flow of
control and are analogous
to flowchart arrows.
• Area bounded by edges
and nodes are called
regions.
• When counting regions, we
include the area outside the
graph as region.
Region 4
edges
Region 6
Region 5
Region 3
Region 1
Region 2
Badariah Solemon 2011
Deriving Test Cases: Example 1
Step 1: Draw a flow graph
Predicate
nodes
Predicate
nodes
R1
Predicate
nodes
R2
R3
R4
Badariah Solemon 2011
Deriving Test Cases: Example 1
(cnt’d)
Step 2: Calculate the Cyclomatic complexity, V(G)
• Cyclomatic complexity can be used to count the
minimum number of independent paths.
• A number of industry studies have indicated that the
higher V(G), the higher the probability or errors.
• The SEI provides the following basic risk assessment
based on the value of code:
Cyclomatic Complexity
Risk Evaluation
1 to 10
a simple program, without very much risk
11 to 20
a more complex program, moderate risk
21 to 50
a complex, high risk program
> 50
an un-testable program (very high risk)
Badariah Solemon 2011
Deriving Test Cases: Example 1
(cnt’d)
• Ways to calculate V(G):
1. V(G) = the number of regions of the flow graph.
2. V(G) = E – N + 2 ( Where “E” are edges & “N” are
nodes)
3. V(G) = P + 1 (Where P is the predicate nodes in the
flow graph, each node that contain a condition)
•
Example:
1. V(G) = Number of regions = 4
2. V(G) = E – N + 2 = 16 – 14 + 2 = 4
3. V(G) = P + 1 = 3 + 1 = 4
Badariah Solemon 2011
Deriving Test Cases: Example 1
(cnt’d)
Step 3: Determine a basis set of independent paths
• Path 1: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 14
• Path 2: 1, 2, 3, 4, 5, 14
• Path 3: 1, 2, 3, 4, 5, 6, 7, 11, 12, 5, …
• Path 4: 1, 2, 3, 4, 6, 7, 2, 11, 13, 5, …
Step 4: Prepare test cases
•
•
Test cases should be derived so that all of these paths are executed
A dynamic program analyser may be used to check that paths have
been executed
Badariah Solemon 2011
Deriving Test Cases: Example 2
Refer pages 486 – 488
* Other topics related to testing conventional application
such as control structure testing, black-box testing,
model-based testing , patterns for software testing are
described in pages 489- 508.
Badariah Solemon 2011
When to Stop Testing?
• Testing is potentially endless.
–
We can not test till all the defects are
unearthed and removed -- it is simply
impossible.
• At some point, we have to stop testing and
ship the software. The question is when.
• According to Pan (1999):
– Realistically, testing is a trade-off
between budget, time and quality.
– It is driven by profit models.
Badariah Solemon 2011
When to Stop Testing?(cnt’d)
• The pessimistic, and unfortunately most often
used approach is to stop testing whenever some,
or any of the allocated resources -- time, budget,
or test cases -- are exhausted.
• The optimistic stopping rule is to stop testing
when either reliability meets the requirement,
or the benefit from continuing testing cannot
justify the testing cost.
Badariah Solemon 2011
Summary
•
•
•
•
•
•
•
Software testing plays an extremely important role in V&V,
but many other SQA activities are also necessary.
Testing must be planned carefully to avoid wasting
development time and resources, and conducted
systematically.
The developer and ITG must work together throughout the
software project to ensure that thorough tests will be
conducted.
The software testing strategy is to begins by ‘testing-in-thesmall’ and move toward ‘testing-in-the-large’
The IEEE 829.2009 standard specifies a set of documents for
use in eight defined stages of software testing.
The ‘white-box’ and ‘black-box’ techniques provide a
systematic guidance for designing test cases.
We need to know when is the right time to stop testing.
Badariah Solemon 2011
References and credit
• Contents in the slides are adapted from
the book and the slides that accompanied
the book by R.S. Pressman, Software
Engineering: A Practitioner’s Approach,
7th. Edition, McGraw Hill, 2009.
• Pan, J. (1999). Software Testing. Retrieved
25th January 2011 from
http://www.ece.cmu.edu/~koopman/des_
s99/sw_testing/
Badariah Solemon 2011