What, why and how?

Johnny Bigert, 2012-10-25 [email protected]

My View on Unit Testing

 Part of something bigger – testing on many levels 1.

  Fast and early feedback!

Tight feedback loop, results within seconds Early implies lower cost 2.

3.

 Confidence!

Less fear of refactoring  Better code!

Testability requires decoupling and usable APIs 4.

 Reliability!

Possible to remove all non-determinism

Unit test, functional test, system test – what does it mean?

Definitions and Terminology

 There are no definitive answers  Different dimensions of automated tests:  Level – Unit, component, integration, system  API – Developer test, external test  Characteristic – Functional, non-functional  Acceptance testing, regression testing

What Level Are We Testing On?

 Unit – The smallest unit of program code, normally a single function, class with methods or data structure with functions  Component – A set of units integrated together, normally a module or subsystem comprising an isolated set of classes or files.

 Integration – Several layers of a stack integrated, normally a stand-alone application (e.g. client or server)  System – Several stand-alone applications integrated (e.g. clients and servers), normally the full system

What API Are We Using For Testing?

 Developer tests – Function calls to an internal or external interface, normally test and production code running in the same process  External tests – Invokation of a non-programming interface, normally running in separate processes or machines. For example, UI (simulated user input), network (HTTP etc), IPC.

What Aspect Are We Testing?

 Functional – The behavior of the system, normally expressed as a step-by-step instruction with some alternative paths  Non-functional – The characteristics of the system, normally expressed as measurements and numbers. For example, latency, round-trip time, memory/CPU/bandwidth consumption, scalability, availability.

Functional

Unit Component Integration System

Developer

“unit test” “component test” N/A

External

N/A “functional test” “system test” “Component test” – personal favorite. Provides stability just like a “functional test” (test code less brittle than unit tests). Provides flexibility, reliability and fast feedback just like a “unit test” (quick to write, easy to mock out problematic parts, results within milliseconds).

Non-Functional

Developer

Unit Component e.g. test threading model of a component Integration System N/A

External

N/A e.g. measure CPU consumption e.g. “load test”

Choose How to Test

 When choosing, consider  Cost (of writing, maintaining, execution resources etc.)  Feedback loop (time)  Reliability  Trade-off example: Mocking or using system x?

 Mocking: cost of writing and maintaining, risk of modeling wrong behavior.

 Using: cost/risk of non-determinism, cost of test resources and longer execution times.

Regression And Acceptance Testing

 Definition: “regression” or “regression bug”  Something that was once considered “done” but does not fulfill requirements anymore  Acceptance testing: the use of a test to determine if a feature fulfills the requirements  Acceptance tests becomes part of the regression test suite once a feature is “done”

Necessities of effective unit and component testing

Dependencies

 Assumption: we want to achieve fast feedback and reliable tests  Note, there are other kinds of tests that also have merit  Remove non-determinism: we need to be in control  Randomness, time, file system, network, databases, multi-threading  Remove dependencies to non-deterministic components

Dependencies

 What kind of problematic dependencies to you have in your code base?

Getting Rid of Dependencies, C++

 Example: Randomness  class A { public: }; void foo() { } bool coinflip = rand() % 2 == 0; if(coinflip) …  Problem: dependency to non-deterministic component (randomness)

Getting Rid of Dependencies, C++

 C++ solution: introduce interface and use dependency injection  class IRandomness { virtual bool coinflip() = 0; };

Getting Rid of Dependencies, C++

 class A { public: A( IRandomness &r ) : randomness(r) {} // ctor inject void foo() { if(randomness.coinflip()) … } private: IRandomness &randomness; };

What Is a Mock Object?

 Conforms to interface  Programmable behavior  For example, “if someone calls your send function with argument ‘hello’, return 7”  Can verify function invocations  Mock frameworks available  E.g. gmock (Google)  Much more elegant in reflective languages

Getting Rid of Dependencies, C++

 Test code:  } void testFoo() { RandomnessMock random; // inherits IRandomness A a(random); a.foo(); …

Getting Rid of Dependencies, C++

 Production code:  } void main() { Randomness A a(random); random; // inherits IRandomness …

Bi-Directional Dependencies

Application

(inherits) sendData

INetworkReceiver

onDataReceived

Network INetworkSender

(inherits)

Getting Rid of Dependencies, C

 Function pointers = hard-to-read code  Link-time tricks  Link with other translation unit defining e.g. rand()

Getting Rid of Dependencies, C

 RandMock m; // mixed C/C++ test code RandMock &getRandMock() { return m; // used by test case to access mock } } int rand() { m.rand(); // forward to mock object  Drawbacks: hard to understand test case code, one test executable per component to test = harder to maintain  Or clean C solution

Code Coverage

 Don’t focus on high levels of code coverage per se  Necessary, but not sufficient  Instead, focus on covering the most important user scenarios  After that, use code coverage to determine which code has not been covered  Analyze to understand what user scenarios are missing

Unit Test Frameworks

Test runners/frameworks  GTest  UnitTest++  CppUnit  C Unit Testing Framework Mock frameworks  GMock  Hippo  Mockpp  Amop

Agree on what to aim for

Fast Feedback Example

From a previous job:  Proprietary hardware/software/OS platform  Very expensive hardware = bottleneck in testing  Emulator to test software locally on Linux machine  Ran unit tests in emulator, turn-around time = 15 mins  Unacceptable

Fast Feedback Example

 Main problem: emulator start-up time  Challenge: proprietary types used (e.g. string classes)  Solution: fake proprietary types (string with std::string, many others just empty implementations)  Ran unit tests on local machine, turn-around time < 15 seconds (from 15 mins)

Fast Feedback Example

 Invest to speed up feedback  Invest to remove testability obstacles  Unit testing should be easy, fast and fun

Development Excellence

 Defect prevention  Continuous improvement - feedback from bugs in order to improve software development process  Relentless testing  Regression detection  Continuous integration running all regression tests often  Focus on both functional and non-functional requirements  Always releasable  Keep CI builds green  Keep the product working

Continuous Integration

 Staged  Stage 1 – build, unit tests, code coverage and functional smoke tests – 10 mins  Stage 2 – extensive functional tests, smoke non functional tests – 1 hour  Stage 3 – full functional and non-functional regression tests , static and dynamic analysis – 12 hours  Run stages per  Commit (fine-grained tracability, expensive), or  Regularly (1. commit, 2. hourly, 3. nightly)

Code Analysis Tools

 Static analysis (run on source code)  Lint – free, local analysis only, lots of false positives    Dynamic analysis (need representative data)  Klocwork – commercial, global analysis Coverity – expensive, global analysis, good precision IBM Rational Purify – commercial, memory leaks, double frees, buffer overflow, use of uninitialized memory. Instrumentation.

 Valgrind – free, memory problems, heap, usage, cache usage, call graphs, threading problems. No instrumentation.