Transcript Software Testing

Software Testing
Software Testing
1
Topics
• Principles of Testing
• Testing in the Life Cycle
• Levels of Testing
• Methods of Testing
• Test Techniques (Functional/Structural)
• Test Process
• Testing and Risk
• Reviews
• Defect Management
• Web based testing
• Tool Support for Testing
Software Testing
2
Topics
• Principles of Testing
– Definitions
– Testing Terminology
– Psychology of Testing
– Fundamental Test Process
– Retesting and Regression Testing
– Tester’s Job
Software Testing
3
Software Testing
• Process of executing a program with the intent of finding errors.
(Myers)
• Testing is the process of exercising or evaluating a system or system
component by manual or automated means to verify that it satisfies
specified requirements.
(IEEE 83a)
• Critical element of software quality assurance and represents the
ultimate review of:
– specification
– design
– coding
Software Testing
4
Role of Testing
• Primary
– Determine whether system meets specifications
– Determine whether system meets needs
• Secondary
– Instill confidence
– Provide insight into the software development process
– Continuously improve the testing process
Software Testing
5
Software Testing Terminology
• Error:
– A measure of the difference between the actual and the ideal.
– A human action that produces a Fault
• Fault:
– A condition that causes a system to fail in performing its required function.
– Incorrect Result
– Also Known as Defect or Bug
– May cause a failure, which is a deviation from its expected delivery or service.
• Failure:
– The inability of a system or component to perform a required function according
to its specifications.
• Reliability
– Probability that software will not cause the failure of a system for a specified time
under specified conditions
Software Testing
6
Software Testing Terminology (Cont’d)
• Debugging:
– The activity by which faults are identified and rectified.
• Verification
– All QC activities throughout the life cycle that ensure that interim
deliverables meet their input specification
• Validation
– The “test phase” of the life cycle which assures that the end product meets
the user’s needs.
Software Testing
7
Software Testing Terminology (Cont’d)
• Test Bed
– An environment that contains the integral hardware, software tools, a suite
of test programs and data used for conducting the test of a component or
system.
• Test Case
– An identified set of information about observable states, conditions, events
and data including inputs and expected outputs.
• Test Plan
– A formal or informal plan to be followed to assure the controlled testing of
the product under test.
Software Testing
8
Software Testing Terminology (Cont’d)
• Quality Control
– Testing a product after a given phase to find out if it has any defects
– Done after the product is built
– Usually reactive
– Expensive and sometimes impossible
– Oriented towards defect detection rather than defect prevention
• Quality Assurance
– Meant to be pro-active
– Oriented towards defect prevention rather than defect detection
– Intended to catch the defects as close to the point of injection as possible
rather than let the defects trickle down into the subsequent levels
– Applies to the process rather than to a specific end-product
– Happens across the board and not “in-line”
Software Testing
9
Software Testing Myths
• If we were really good at programming, there would be no bugs to
catch. There are bugs because we are bad at what we do.
• Testing implies an admission of failure.
• Tedium of testing is a punishment for our errors.
• All we need to do is:
– concentrate
– use structured programming
– use OO methods
– use a good programming language
– ...
Software Testing
10
Software Testing Reality
• Human beings make mistakes, especially when asked to create
complex artifacts such as software systems.
• Studies show that even good programs have 1-3 bugs per 100 lines of
code.
• People who claim that they write bug-free software probably haven’t
programmed much.
• Bugs in compilers, languages, DBs, OS
• Certain bugs easier to find in testing
• Post release debugging is expensive
• Good test design is challenging and rewarding
Software Testing
11
Software Testing
The more realistic
your testing ...
… the less shocking reality
will be.
- unknown ancient Zen Software Master
Software Testing
12
Software Applications
Predefined
set of inputs
Predefined set of
procedural steps
Software Testing
Predefined
set of outputs
13
TESTING
False Statement
A tester is given a false
statement that
“The system works”
Software Testing
14
TESTING
From an infinite number of
possibilities,
TESTER
selects an input that
contradicts
the statement
Software Testing
15
Goals of Testing
• Discover and prevent bugs.
• The act of designing tests is one of the best bug preventers known.
(Test, then code philosophy)
• The thinking that must be done to create a useful test can discover and
eliminate bugs in all stages of software development.
• However, bugs will always slip by, as even our test designs will
sometimes be buggy.
Design a series of tests that have a high probability of finding errors prior to
delivery
Software Testing
16
Principles of Testing
• Testing must find the largest number of errors in the least amount of
time.
• Test case development should begin immediately after requirements
definition.
• Pareto’s principle: 80% of errors occur in 20% of the code.
• Testing should begin in the small and progress to the large.
• Exhaustive testing is not possible.
• All tests must be traceable back to customer requirements.
• A good test is not redundant; every test serves a different purpose.
– Tests should be as simple as possible. A test should not attempt to uncover
multiple types of errors.
Software Testing
17
Fundamental Test process
• Plan Tests
• Execute Tests
• Evaluate Tests
Software Testing
18
Limits of software testing
• “Good” testing will find bugs
• “Good” testing is based on requirements,
i.e. testing tries to find differences between the expected and the
observed behavior of systems or their components
• BUT: Testing can only prove the presence of bugs - never their absence
Software Testing
19
Testability =
How easy can a program be tested
• Features that help/influence testing:
• operability - it operates cleanly
• observability
• the results are easy to see
• distinct output is generated for each input
• incorrect output is easily identified
• controllability
• processing can be controlled
• tests can be automated & reproduced
• decomposability - software modules can be tested independently
• simplicity - no complex architecture and logic
• stability - few changes are requested during testing
• understandability - program is easy to understand
Software Testing
20
Who tests the software
developer
Understands the system
but, will test “gently”
and, is driven by “delivery”
independent tester
Must learn about the system,
but, will attempt to break it
and, is driven by quality
Software Testing
21
Phases in a Testers Mental Life
• Phase 0
– No difference between testing and debugging. Test only to support
debugging
• Phase 1
– Purpose is to show that software works
• Phase 2
– Purpose is to show that software DOES NOT work
• Phase 3
– Purpose not to prove anything, but to reduce perceived risk of not working
to an acceptable value
• Phase 4
– Testing is a mental discipline resulting in low risk software without much
testing effort
Software Testing
22
Phases in a Testers Mental Life
• If trying to prove that software works, purpose best served by not
testing at all
• If trying to prove that software does not work, testing has no end!
Software Testing
23
Testing Versus Debugging
Testing
Debugging
• The purpose is to show that a
program has bugs.
• Demonstration of error or apparent
correctness.
• The purpose is to find the error that
led to the program’s failure and to
design and implement the program
changes that correct the error.
• Deductive process
• Proves a programmer’s failure.
• Programmer’s vindication.
• Can be automated to a large extent.
• Automatic debugging is still a
dream.
• Much of testing can be done
without design knowledge (by an
outsider)
• Debugging is impossible without
detailed design knowledge (by an
insider)
Software Testing
24
Function Versus Structure
• Functional Testing:
– Program is treated as a black box.
– Program is subjected to inputs, and its outputs are verified for conformance to
specified behavior.
– Implementation details do not matter.
– Takes a user’s point of view.
– In principle, can detect all bugs in an infinite amount of time.
• Structural Testing:
– Aims at exercising the different control and data structures used in the program.
– Criteria are precise as they are based on program structures (i.e., are quite precise).
– Looks at implementation details.
– Takes a developer’s point of view.
– Is inherently finite but cannot detect all errors.
Software Testing
25
Designer Versus Tester
• The designer and the tester can be completely separated if only
functional testing is performed.
• In structural testing the tester, like the designer, has intimate
knowledge of the structure of the program.
• The more the tester knows about the design, the more likely he will
eliminate useless tests (functional differences handled by the same
code).
• Testers that have design knowledge may have the same misconceptions
as the designer.
• Lack of design knowledge may help the tester to develop test cases that
a designer would never have thought of.
• Lack of design knowledge may result in inefficient testing and
blindness to missing functions Software
and Testing
strange cases.
26
Small Versus Large Systems
• For small systems with one user, quality assurance may not be a major
concern.
• As systems scale up in size and number of users, quality assurance
becomes more of a concern.
• As systems dramatically scale up in size (e.g., millions of lines of
code), our quality criteria may have to change as exhaustive testing
may not be economically possible.
– E.g., a 75% code coverage may be acceptable.
Software Testing
27
Programs and their Environment
• A program’s environment is the hardware and systems software
required to make it run.
• Programmers should learn early in their careers that it is not smart to
blame the environment for bugs.
• Bugs in the environment are rare because most bugs have been found
over a long period of usage by a large number of users.
Software Testing
28
Testing Isn’t Everything
• Other methods for improving software reliability are:
– Inspection methods:
• Walkthroughs, formal inspections, code reading.
– Design style:
• Criteria used by programmers to define what they mean by a “good program”.
– Static analysis:
• Compilers take over mundane tasks such as type checking.
– Good Programming Languages and Tools:
• Can help reduce certain kinds of bugs
Software Testing
29
Aspects that Hinder Effective Testing
• Optimism
• Belief that the system works
• Negative attitude towards effective testing
• Ego
• Don’t want to fail
• Conflict between testers and developers
• Testing is least structured
• Testing is expensive
• Delivery commitments
Software Testing
30
Re-Testing and Regression Testing
• Re-Testing
– Testing to ensure that the original fault has been successfully removed
– Test for similar and related faults
– Performed whenever the fault detected has been fixed
• Regression Testing
– Verify that modifications have not caused unintended adverse side effects
in the unchanged software
– Verify that the modified system still meets its requirements
– Performed whenever the software, or its environment, is changed
– Regression test suites
• Are run many times
• Generally evolve slowly
• Ideal for automation
Software Testing
31
Exercise 1
Software Testing
32
Questions ?
Software Testing
33