Software Testing 101

Key concepts and definitions.

Fundamental Principles and practical aspects

(partly based on material from Giulio Antoniol, LOG3430, Ecole Polytechnique de Montreal ) 1

Software failures: « Fun » Stories

     Hounslow, West London (2009). Tesco cash machine pays

customers DOUBLE after 'operational error’

“a computer glitch played havoc with an ATM at one of the supermarket's stores.” “It makes a change for us to have the joke on the banks for once.”

Bank

: "If the people using the ATM see it as a bit of fun, so be it." Other occurences in 2008, 2011. Europe, North America...

2

Software failures: Horror Stories

 Ariane 5, Flight 501 (1996) 4000 m and 40s  • Loss: 370 millions $ US • One of the root causes: Bad handling of exceptions PRE: -32768 <= x <= +32767 POST: y=int16(x) Radiation machine Therac-25 (1985-87) • Overdoses (up to 100 times) • At least 5 deaths • One of the root causes: No test on non-standard inputs Mars Climate Orbiter (1999) • Crash of the ship on Mars • Losses: 300 millions $US • One of the root causes: mix of imperial and metric system 3

Pervasive Problems

 Software is commonly delivered late, way over budget, and of unsatisfactory quality  Software validation and verification are rarely systematic and are usually not based on sound, well-defined techniques  Software quality is poorly measured, monitored, and controlled.

 NIST study (2002)  software bugs and glitches cost the U.S. economy about $59.5 billion a year even though 50 percent of software development budgets are devoted to testing  better testing could expose and remove bugs at the early development stage, and generate savings of about $22.2 billion.

4

Dealing with software faults: the big picture

Fault Handling Fault Avoidance Fault Detection Design Methodology Inspections Verification Configuration Management Testing Unit Testing Fault Tolerance Atomic Transactions Modular Redundancy Debugging Integration Testing System Testing Debugging Correctness Debugging Performance

5

Test Organization

Unit test Design descriptions Functional specifications Other specifications Customer requirements User environment Unit test .

.

.

Unit test Integration test Function test Performance test Acceptance test Installation test Integrated modules Functioning system Verified, validated software Accepted system

Pfleeger, 1998 SYSTEM IN USE!

Testing Definitions & Objectives

Definitions: software anomalies, testing.

Goals of testing.

Testing Process Overview.

7

IEEE Std 1044-2009 Standard Classification for Software Anomalies



Error:

A human action that produces an incorrect result .



Defect:

An imperfection or deficiency in a work product work product does not meet its requirements or specifications and needs to be either repaired or replaced. where that 

Fault:

A manifestation of an error in software

.



Failure: (A)

required function or its inability to perform within previously specified limits Termination of the ability of a product to perform a . (B) An event in which a system or system component does not perform a required function within specified limits.  Problem: (A) Difficulty or uncertainty experienced by one or more persons, resulting from an unsatisfactory encounter with a system in use. (B) A negative situation to overcome . 8

IEEE Std 1044-2009 Standard Classification for Software Anomalies

Every fault is a defect, but not every defect is a fault.

A defect is a fault if it is encountered during software execution (thus causing a failure).

A defect is not a fault if it is detected by inspection or static analysis and removed prior to executing the software.

9

Basic Testing Definitions

 Testing : Exercise the software with test cases to find faults or gain confidence in the system  Test cases: Set of inputs and a list of expected outputs (sometimes left out)  Test Stub: Partial implementation of a component on which the tested component depends.  Test Driver : Partial implementation of a component that depends on the tested component.  Test stubs and drivers enable components to be isolated from the rest of the system for testing. 10

Summary of Definitions

Test suite * * Test case * exercises * finds * * Failure * * is caused by 1…n Component Test stub Test driver Fault is revised by * * * is caused by Correction repairs * Error

11

Goals of Testing

 Dijkstra, 1972: “Program testing can be used to show the presence of bugs, but never to show their absence.”  No absolute certainty can be gained from testing.

 Testing should be integrated with other verification activities, e.g., inspections.

 Main goal: demonstrate the software can be depended upon, i.e., sufficient dependability .

12

Reality Check

 No matter how rigorous we are, software is going to be faulty  Testing represent a substantial percentage of software development costs and time to market  Impossible to test under all operating conditions – based on incomplete testing, we must gain confidence that the system has the desired behavior  Testing large systems is complex – it requires strategy and technology- and is often done inefficiently in practice 13

Testing Process Overview

SW Representation Tests SW Code Tests Results

Oracle

What makes a good test?

Effective at uncovering faults Helps to locate faults for debugging Repeatable so that a precise under standing of the fault can be gained Automated to reduce costs and time Systematic so as to be predictable Expected Results

Compare

Testing requirements vary

- safety critical applications - mass products 14

Fundamental Principles

Exhaustive testing.

Test Criteria and Test Coverage.

Black Box / White Box Testing.

15

Exhaustive Testing

 Exhaustive testing, i.e., testing a software system using all the possible inputs, is most of the time impossible.

 Examples:   A program that computes the factorial function (n!=n.(n-1).(n-2)…1)  Exhaustive testing = running the program with 0, 1, 2, …, 100, … as an input!

A compiler (e.g., javac)  Exhaustive testing = running the (Java) compiler with any possible (Java) program (i.e., source code)  Technique used to reduce the number of inputs  Testing criteria group input elements into (equivalence) classes  One input in selected in each class (notion of test data coverage) 16

Test Coverage

Software Representation (Model) Associated Criteria

Test cases must cover all the … in the model

Test Data Representation of • the specification



Black-Box Testing • the implementation



White-Box Testing

17

Black-box vs. White-box Testing



Black box

+ Check conformance with specifications + It scales up (different techniques at different granularity levels) – It depends on the specification notation and degree of detail – Do not know how much of the system is being tested – What if the software performed some unspecified, undesirable task?

o White box + It allows you to be confident about test coverage + It is based on control or data flow coverage – It does not scale up (mostly applicable at unit and integration testing levels) – Unlike black-box techniques, it cannot reveal missing functionalities (part of the specification that is not implemented) 18

Black vs. White Box Testing

System Specification Implementation Missing functionality: Cannot be revealed by white-box techniques Unexpected functionality: Cannot be revealed by black-box techniques 19

Complete Coverage: White-Box

if x > y then Max := x;

else

Max :=x ; // fault!

end if; x>y x ≤y

{x=3, y=2; x=2, y=3} can detect the error, more “coverage” {x=3, y=2; x=4, y=3; x=5, y=1} is larger but cannot detect it  Testing criteria group input domain elements into (equivalence) classes defined with regard to the control flow paths  Complete coverage attempts to run test cases from each class 20

Complete Coverage: Black-Box

 Specification of Compute Factorial Number: If the input value n is < 0, then an appropriate error message must be printed. If 0 <= n < 20, then the exact value of n! must be printed. If 20 <= n < 200, then an approximate value of n! must be printed in floating point format, e.g., using some approximate method of numerical calculus. The admissible error is 0.1% of the exact value. Finally, if n>=200, the input can be rejected by printing an appropriate error message.  Because of expected variations in behavior, it is quite natural to divide the input domain into the classes {n<0}, {0<= n <20}, {20 <= n < 200}, {n >= 200}. We can use one or more test cases from each class in each test set. Correct results from one such test set support the assertion that the program will behave correctly for any other value, but there is no guarantee!

21

Empirical Testing Principle

 Exhaustive testing cannot be performed in practice.

 Therefore we need test investigated.

strategies that have been empirically  A significant test case is a test case with high error detection potential – it increases our confidence in the program correctness.

 The goal is to run a sufficient number of significant test cases – that number should be as small as possible.

22

Practical Aspects

Testing activities and development lifecyle 23

Testing Activities Through the Lifecycle

 Testing should not be a post coding activity. Planning is essential.

 Identifying test requirements and test cases early helps shorten the development time. They may help reveal faults.

 To each development phase, its testing  Requirements (possibly wrong or incomplete) => Acceptance testing  Specifications/Analysis => System Testing  Design (possibly faulty) => Integration testing  Class statecharts, method pre- and post-conditions, structure => unit testing 24

Differences among Testing Activities

Pezze and Young, 1998 Unit Testing

From module specifications Visibility of code details Complex scaffolding Behavior of single modules Integration Testing From interface specifications Visibility of integr. Struct.

Some scaffolding Interactions among modules

System Testing

From requirements specs No visibility of code No drivers/stubs System functionalities 25

System vs. Acceptance Testing

 System testing   The software is compared with the requirements specifications (verification) Usually performed by the developers, who know the system  Acceptance testing    The software is compared with the end-user requirements (validation) Usually performed by the customer (buyer), who knows the environment where the system is to be used Sometime distinguished between a purpose products b -testing for general 26

Testing Activities

•

Establish the test objectives

•

Design the test cases

•

Write the test cases

•

Test the test cases

•

Execute the tests

•

Evaluate the test results

•

Understand the cause of failures

•

Do regression testing

27

Testing takes creativity

 Testing often viewed as dirty work (though less and less).

 To develop an effective test, one must have:  Detailed understanding of the system  Knowledge of the testing techniques  Skill to apply these techniques in an effective and efficient manner  Testing is done best by independent testers  Programmer often stick to the data set that makes the program work  A program often does not work when tried by somebody else.

28

SYNTHESIS

     Software testing consists in exercising a software with test cases to find faults or gain confidence in the system.

Software testing consumes up to 50% of software development budgets but in many cases, it is not based on sound and well-defined techniques.

Exhaustive testing is impossible. Testing is a compromise between available resources and sought quality: testing coverage and criteria.

Different software representations (Specification & Implementation)  different testing techniques (Black Box Testing / White Box Texting).

Software Testing is not a post coding activity!

29