Transcript CHAPTER 17 SOFTWARE TESTING STRATEGIES

CHAPTER 18
SOFTWARE TESTING
STRATEGIES
1
TOPICS
 A strategic
approach to software testing
 Unit Testing
 Integration Testing
 Validation Testing
 System Testing
 The ART of Debugging
 Summary
2
December, 2007
STRATEGIC APPROACH TO
SOFTWARE TESTING
Generic characteristics of software testing
strategies: Testing
begins at module level and works outward
towards the of integration entire computer based system.
 Different testing techniques are required at different
points in time.
 Testing is conducted by the s/w developer and ITG
( Independent Test Group ) for large projects.
 Testing and Debugging are different and Debugging is
essential in any testing strategy.
3
December, 2007
Verification and Validation
 Verification
-- Does the product meet its
specifications? Are we building the product
right?
 Validation -- Does the product perform as
desired? Are we building the right product? 4
December, 2007
Software Testing Strategy
A Software Testing Strategy
5
December, 2007
Software Testing Strategy
6
December, 2007
Software Error Model
 f(t) = cumulative remaining
 l0 = initial failure rate
p
errors at time t
= exponential reduction as errors repaired
f(t) = (1/p)ln(l0pt + 1)
7
December, 2007
STRATEGIC APPROACH
 Issues
to be addressed to develop a
successful software testing strategy:
 Specify
product requirements in a quantifiable
manner long before testing commences.
 State testing objectives explicitly.
 Understand the users of the software and
develop a profile for each user category.
 Develop testing plan that emphasizes “rapid
cycle testing.” (test a little, fix problems, deliver
a little, get feedback)
8
December, 2007
STRATEGIC APPROACH
 Issues
to be addressed to develop a
successful software testing strategy:
 Build
robust software that is designed to test
itself.
 Use effective formal technical reviews as a
filter to testing.
 Conduct formal technical reviews to assess
test strategy and test cases.
 Develop continuous improvement approach
9
December, 2007
UNIT TESTING
 Unit
testing -- focuses on the smallest element
of software design viz. the module.
 Corresponds
 Makes
to class testing in the OO context.
heavy use of white-box testing.
10
December, 2007
UNIT TESTING
Unit Test Generation Considerations:
Review Design information - develop unit test cases.
driver
Module to
be tested
stub
stub
interface
local data structures
boundary conditions
independent paths
error handling paths
Test
cases
11
December, 2007
Unit Test Generation
 Interface
considerations
#
of input parameters = # arguments?
 Parameter and argument attributes match?
 Parameter and argument units match?
 Order correct (if important)?
 Number and order of arguments for built-ins?
 References to parms not associated with entry point?
 Attempt to modify input-only arguments?
 Global variable definitions consistent?
 Constraints passed as arguments?
12
December, 2007
Unit Test Generation
 External
I/O considerations
 Files
attributes correct?
 OPEN/CLOSE correct?
Format specification matches I/O statement?
 Buffer size matches record size?
 Files opened before use?
 EOF handled correctly?
I/O errors handled?
 Textual errors in output?
13
December, 2007
Unit Test Generation
 Data
structure considerations
 Improper
or inconsistent typing?
 Erroneous initialization or default values?
 Incorrect variable names?
 Inconsistent data types?
 Underflow, overflow and addressing
exceptions?
14
December, 2007
Unit Test Generation
 Test
cases must cover all execution paths
 Common computational errors to be checked:
 incorrect
arithmetic
 mixed mode operations
 incorrect initialization
 precision inaccuracy
 incorrect symbolic representation of expression
 Other
tests needed
 incompatible
data types in comparisons
 incorrect logical operators or precedence
 comparison problems (e.g., == on floats)
 loop problems
December, 2007
15
Unit Test Generation
 Error
handling tests
 Exception-handling
is incorrect?
 Error description is unintelligible, insufficient
or incorrect?
 Error condition causes system interrupt before
error handling completed?
16
December, 2007
INTEGRATION TESTING
 A systematic
approach for constructing
program structure while conducting tests to
uncover errors associated with interfacing.
 Tendency for Non-Incremental integration..
Big Bang approach …. Chaos !! ( usually ).
 Incremental integration - program is
constructed and tested in small segments.
 Top-Down
Integration testing
 Bottom-Up Integration testing
17
December, 2007
INTEGRATION TESTING
18
December, 2007
INTEGRATION TESTING
Top-Down Approach
 Begin construction and testing with main module.
 Stubs
are substituted for all subordinate modules.
 Subordinate
stubs are replaced one at a time by
actual modules.
 Tests are conducted as each module is integrated.
 On completion of each set of tests, another stub is
replaced with the real module.
 Regression testing may be conducted to ensure
that new errors have not been introduced.
19
December, 2007
Top Down Approach - Use Stubs
20
December, 2007
INTEGRATION TESTING
Top-Down Approach :
 Advantages:
 Verifies
major control or decision points early in the
test process.
 With the use of depth-first integration testing, a
complete function of the software can be
demonstrated. -- Confidence builder for
developer/customer.
 Disadvantages:
 Since
stubs replace lower level modules, no
significant data can flow upwards to the main
module.
December, 2007
21
INTEGRATION TESTING
Bottom Up Approach :
 This
approach begins construction and testing
with modules at the lowest levels in the
program structure.
 Low-level
modules are combined into clusters.
 A driver is written to coordinate test case input and
output.
 The cluster is tested.
 Drivers are removed and clusters are combined
moving upward in the program hierarchy.
22
December, 2007
Bottom Up Approach
23
December, 2007
INTEGRATION TESTING
Bottom Up Approach
 Advantages:
 Easier
test case design and lack of stubs.
 Disadvantages:
 The
program as an entity is does not exist until the
last module is added.
Sandwich Testing:- combined approach
 Top
down strategy for upper levels and Bottom
up strategy for subordinate levels.
24
December, 2007
INTEGRATION TESTING
 Regression Testing
 Re-execution
of some subset of tests already
conducted to ensure that the new changes do
not have unintended side effects.
 The Regression test suite should contain three
different classes of test cases :
 A representative sample of tests that will
exercise all software functions
 Additional tests that focus on functions that are
likely to be affected by the change.
 Tests that focus on software components that
have changed.
25
December, 2007
INTEGRATION TESTING
Integration Test Documentation
1
Scope of
testing
2
Test plan
3
4
5
Test
Procedure n
Actual Test
Results
Ref. &
Appendix
Environment
Test Schedule
Unit
Test
/ Resources
phases
test
case
and
Overhead
data
Test
builds
software
environment
Order of
Integration
Expected
Results
for build
n
26
December, 2007
VALIDATION TESTING
 It
provides final assurance that software
meets all functional, behavioral, and
performance requirements.
-- Exclusive use of Black-box testing techniques.
 After each validation test case either software
conforms to specs or a deviation from specs is
detected and a deficiency list needs to be worked.

Alpha and Beta testing
test -- At developer’s site by customer.
 Beta test -- At customer’s site in a “live”
environment.
27
 Alpha
December, 2007
SYSTEM TESTING
A series
of tests to verify that all system
elements have been properly integrated.
 Recovery Testing:
 Forces
software to fail in a variety of ways and
verifies that recovery is properly performed.
 Security Testing:
 Attempts
to verify the software’s protection
mechanisms. The software designer tries to make
penetration cost more than the value of
information obtained by breaking in.
28
December, 2007
SYSTEM TESTING
 Stress
Testing:
 Executes
the system in a manner that demands
resources in abnormal quantity, frequency or
volume.
 Performance Testing:
 To
test the run time performance of a system
within the context of an integrated system.
29
December, 2007
CLCS Test Approach
Operations Environment
User Acceptance Tests
System S/W
Validation Tests
System Delivery
COTS H/W
on Dock
Acceptance
Test
System Test
User App S/W
Validation Tests
User Eval CSCI Int
Test
Integration Environment
Developers
Early
Unit Integ
User Eval
Test
System Integration and
Test Group
Validation Group
Application S/W IPT
Unit
Design
Test
Development Environment
Users
THE ART OF DEBUGGING
 Debugging
is a consequence of successful
testing -- when a test case uncovers an error,
it is the debugging process that results in the
removal of the error.
 Debugging is an ART. The external
manifestation of the error and the cause of
the error normally do not share an obvious
relationships.
31
December, 2007
THE ART OF DEBUGGING
The Debugging process
Execution of test
cases
Test
cases
Additional
tests
Results
Suspected
causes
Regression tests
Corrections
Debugging
Identified
causes
32
December, 2007
THE ART OF DEBUGGING
 Debugging Approaches
 Brute
force : - Take memory dumps, invoke run
time traces. Least efficient and quite common.
 Backtracking
:- Once an error is uncovered, trace
your way back to the cause of the error.
 Cause
Elimination : - Isolate potential causes,
devise cause hypotheses tests to isolate bug.
 Use
of debugging tools
33
December, 2007
COMMENTS
 Should
the software developer be involved
with testing ?
 Developer’s
have a vested interest in
demonstrating that their software is error-free.
 Developer’s (psychologically) feel that testing
is destructive.
 When
are we done with testing ?
 “You
are never done with testing, the burden
shifts from you to the customer.”
34
December, 2007
SUMMARY
 Software Testing
accounts for the largest
percentage of technical effort in the software
process.
 Objective of Software testing -- To uncover
errors, maintain software quality.
 Steps : Unit, Integration, Validation, System.
 Debugging is often an art and the most
valuable resource is often the counsel of other
software engineers.
35
December, 2007