Transcript Transparency Masters for Software Engineering: A

Software Testing Strategies
1
Software Testing

Strategy

Integration of software test case design methods into
a well-planned series of steps


Provides a road map that describes



Successful software construction
The steps to be conducted as part of testing
How much effort, time and resources will be required
Must incorporate test planning, test case design, test
execution, and resultant data collection and
evaluation
2
Software Testing

Strategy



Should be flexible enough to promote a customized
testing approach
Must be rigid enough to promote reasonable planning
and management tracking as the project progresses
Different test methods are beginning to cluster
themselves into distinct approaches and philosophies
3
Software Testing

What is it?

To uncover errors

How do we conduct the test?





Do we develop a formal plan?
Should we test the entire program as a whole or run tests on a
small part of it?
Should we rerun tests we have already conducted as we add
new components to a large system?
When should we involve customer?
These questions must be answered when we develop a
software testing strategy.
4
Software Testing

Who does it?


The strategy is developed by the project manager,
software engineers, and testing specialists.
Why is it important

Testing often accounts for more project effort than
any other software engineering activity

If conducted haphazardly



time is wasted
Unnecessary effort is expended
Errors sneak through undected
5
Software Testing

What are the steps?

Begins in the small and progresses to the large



Focus on a single component or a small group of related
components
After components are tested they may be integrated until the
complete system is constructed
As errors are uncovered, they must be diagnosed and
corrected using a process that is called debugging
6
Software Testing

What is the work product?

A test specification document



Defining a plan that describes overall strategy
A procedure that defines specific testing steps and the tests
that will be conducted
How do I ensure that I’ve done it right?


By reviewing the Test specification prior to testing
Assess the completeness of test cases and testing
tasks
7
Software Testing

A Strategic Approach

Testing is a set of activities that can be planned in
advance and conducted systematically





Conduct effective formal technical reviews
Begin at the component level and work “outward” toward the
integration of the entire computer-based system
Different testing techniques at different points in time
Usually conducted by the software developer and for large
projects by an independent test group
Testing and debugging are different activities, but debugging
must be accommodated in any testing strategy
8
Software Testing

Verification and Validation

Software testing is often referred to as V&V

Verification refers to the set of activities that ensures that
software correctly implements a specific function


Are we building the product right?
Validation is a different set of activities that ensure that the
software that has been built is traceable to customer
requirements

Are we building the right product?
9
Software Testing

Verification and Validation

Encompass a wide array of Software Quality
Assurance (SQA) activities including









Formal technical reviews
Quality and configuration audits
Performance monitoring
Simulation
Feasibility study
Documentation review
Database review
Algorithm analysis
Development, usability, installation testing
10
Software Testing

Organizing for Software Testing

Psychological point of view

The software engineer analyzes, models, and then creates a
computer program and its documentation


From the software engineers perspective testing is destructive
as it tries to break the thing that SE has built
There are often a number of misconceptions



The developer of the software should no do test
Software should be tossed over the wall to strangers who will
test it mercilessly
Testers get involved with the project only when the testing
steps are about to begin
11
Software Testing

Organizing for Software Testing

The role of the independent test group




Remove the inherent problems associated with letting the
builder test the thing that has been built
Removes the conflict of interest
The software engineer, however, does turn the program over
to the ITG and walk away; both have to work closely
throughout the project
The developer must be available to correct uncovered errors
12
Software Testing

Strategy for conventional software architecture

Unit Testing


Integration Testing


Focus is on design and the construction of the software
architecture
Validation Testing


Concentrates on each unit (i.e. component) of the software
as implemented in the source code
Requirements are analysis are validated against the software
that has been constructed
System Testing

Software and other elements are tested as a whole
13
Software Testing

Strategy for Object Oriented Architecture

Focus on “testing in the small” and work outward
toward “testing in the large”


Testing in the small changes from an individual module to a
class encompassing attributes and operations; implies
communication and collaboration
A series of regression tests are run to uncover errors due to
communication and collaboration between classes
14
Software Testing

Strategic Issues

Specify product requirements in a quantifiable manner long
before testing

A good strategy not only looks for errors, but also asses other
quality characteristics such as




Portability
Maintainability
Usability
State testing objectives explicitly

The specific objectives of testing should be stated in measurable
terms



Test effectiveness
The cost to find and fix defects
Test coverage
15
Software Testing

Strategic Issues

Understand the user of the software and develop a
profile for each


Use-cases that describe the interaction scenario for each
class of user can reduce overall testing effort by focusing
testing on actual use of the project
Build robust software that is designed to test itself

Software should be designed in a manner that uses antibugging techniques. Software should be capable of
diagnosing certain classes of errors. The design should be
accommodated automated testing and regression testing
16
Software Testing

Strategic Issues

Use effective formal technical reviews as a filter prior
to testing


Reviews can reduce the amount of testing effort that is
required to produce high-quality software
Develop a continuous improvement approach

The testing strategy should be measured. The metrics
collected during testing should be used as part of a statistical
process control approach for testing
17
Software Testing

Test Strategies for Conventional Software

There are many strategies for testing software.

A software team could wait until the system is fully
constructed and then conduct tests on the overall system.

At the other extreme software engineer can conduct tests on
a daily basis, whenever any part of the system is constructed

A testing strategies falls between the two extremes


Begin with the testing of individual program units
Moving to tests designed to facilitate the integration of the units
18
Software Testing

Test Strategies for Conventional Software

Unit Testing


Focuses verification effort on the smallest unit of software
design – the software component or module
Integration Testing

“If they all work individually, why do you doubt that they’ll
work when we put them together?”

The problem is “putting them together” – interfacing
 Data may be lost
 One module can have an inadvertent, adverse effect on
another; sub-function when combined
 Global data structure can present problems
19
Software Testing

Test Strategies for Conventional Software

Regression Testing





Each time a new module is added as part of integration
testing, the software changes
New data flow paths are established
New I/O and control logic is invoked
Regression testing is the re-execution of some subset of
tests that have already been conducted to ensure that
changes have not propagated unintended side effects
Smoke testing

An integration testing
20
Software Testing

Test Strategies for Conventional Software

Smoke testing




An integration testing
Software components that have been translated into code
are integrated into a “build” – data files, libraries, reusable
modules, and engineered components that are required to
implement one or more product
The intent of testing here is to uncover show stoppers –
errors that have the highest likelihood of throwing the
software project behind schedule
The build is integrated with other builds and the entire
product is smoke tested daily
21
Software Testing

Test Strategies for OBJECT-ORIENTED
SOFTWARE

Unit Testing



An encapsulated class is the focus of unit testing
Operations within the class are, however, the smallest unit
Integration Testing

Thread based testing


Integrates a set of classes required to respond to one input or
event for the system
Used based testing


Begins the construction of the system by testing independent
classes – use very few if any server classes
Testing dependent classes – which use independent classes
22
Software Testing

Test Strategies for OBJECT-ORIENTED
SOFTWARE

Integration Testing

Cluster Testing

A cluster of collaborating classes (determined by examining the
CRC) to uncover errors in collaboration
23
Software Testing

Validation Testing

At this level the distinction between conventional and objectoriented software disappears


Can be defined in many ways


A simple definition is that validation succeeds when software
functions in a manner that can be reasonably expected by the
customer.
Alpha Testing


Testing focuses on user visible actions and user recognizable
output from the system
Conducted at the developer site by end-users
Beta Testing

Conducted at end user sites – the developer is generally not
present
24
Software Testing

System Testing

Ultimately, software is incorporated with other system elements



Hardware, people, information
A series of system integration and validation tests are conducted
A classic system testing problem – finger pointing

Software engineers should anticipate potential interfacing problems
 Design error handling paths that test all information coming from
other elements of the system
 Conduct series of tests to simulate bad data
 Record the results of testing as evidence
 Participate in planning and design of systems tests to ensure that
software is adequately tested
25
Software Testing

System Testing

Recovery Testing

Is a system test that forces the software to fail in a variety of
ways and verifies that recovery is properly performed


If automatic recovery
 Re-initialization
 Check-pointing mechanisms
 Data recovery
 Restart are evaluated for correctness
If manual
 Mean time to repair (MTTR) is evaluated
26
Software Testing

Security Testing


Stress Testing


It verifies that protection mechanisms built into a
system will, in fact protect it from improper penetration
It executes a system in a manner that demands
resources in abnormal quantity, frequency, or volume
Performance Testing


Are often coupled with stress testing and usually
require both hardware and software instrumentation.
To measure resource utilization
27