Transcript Tutorial Nikolai Tillmann, Peli de Halleux, Wolfram Schulte (Microsoft Research) Tao Xie (North Carolina State University)

Tutorial
Nikolai Tillmann, Peli de Halleux, Wolfram Schulte (Microsoft Research)
Tao Xie (North Carolina State University)






Unit Testing
Parameterized Unit Testing
Input Generation with Pex
Patterns for Parameterized Unit Testing
Parameterized Models
Wrapping up


We will use the tool Pex as part of the exercises
To install it, you need
 Windows XP or Vista
 .NET 2.0

Download Pex from
http://research.microsoft.com/en-us/projects/pex/downloads.aspx

Two variations :
 pex.devlabs.msi:
requires Visual Studio 2008 Team System
 pex.academic.msi:
does not require Visual Studio,
but for non-commercial use only

To run the samples, you need VS Team System
Introduction


A unit test is a small program with assertions
Test a single (small) unit of code
void ReadWrite() {
var list = new List();
list.Add(3);
Assert.AreEqual(1, list.Count);
}





Design and specification By Example
Documentation
Short feedback loop
Code coverage and regression testing
Extensive tool support for test execution and
management


Happy path only
Hidden integration tests
 Touch the database, web services, requires multiple
machines, etc…


New Code with Old Tests
Redundant tests

Coverage: Are all parts of the program exercised?





statements
basic blocks
explicit/implicit branches
…
Assertions: Does the program do the right thing?
 test oracle
Experience:
 Just high coverage of large number of assertions is
no good quality indicator.
 Only both together are!
Goal: Implement and test TrimEnd:
// trims occurences of the ‘suffix’ from ‘target’
string TrimEnd(string target, string suffix);


Open Visual Studio
Create CodeUnderTestProject, and implement TrimEnd

File -> New -> Project -> Visual C# -> Class Library
 Add new StringExtensions class
 Implement TrimEnd

Create TestProject, and test TrimEnd

File -> New -> Project -> C# -> Test -> Test Project
 Delete ManualTest.mht, UnitTest1.cs, …
 Add new StringExtensionsTest class
 Implement TrimEndTest

Execute test

Test -> Windows -> Test View, execute test, setup and inspect Code Coverage
public class StringExtensions {
public static string TrimEnd(string target, string suffix) {
if (target.EndsWith(suffix))
target = target.Substring(0, target.Length-suffix.Length);
return target;
}
}
[TestClass]
public class StringExtensionTest {
[TestMethod]
public void TrimEndTest() {
var target = "Hello World";
var suffix = "World";
var result = StringExtensions.TrimEnd(target, suffix);
Assert.IsTrue(result == "Hello ");
}
}
Introduction

Three essential ingredients:
 Data
 Method Sequence
 Assertions
void Add() {
int item = 3;
var list = new List();
list.Add(item);
var count = list.Count;
Assert.AreEqual(1, count);
}
list.Add(3);

Which value matters?
 Bad choices cause redundant, or worse,
incomplete test suites.
 Hard-coded values get stale when product
changes.
 Why pick a value if it doesn’t matter?


Parameterized Unit Test =
Unit Test with Parameters
Separation of concerns
 Data is generated by a tool
 Developer can focus on functional specification
void Add(List list, int item) {
var count = list.Count;
list.Add(item);
Assert.AreEqual(count + 1, list.Count);
}

A Parameterized Unit Test can be read as a
universally quantified, conditional axiom.
void ReadWrite(string name, string data) {
Assume.IsTrue(name != null && data != null);
Write(name, data);
var readData = Read(name);
Assert.AreEqual(data, readData);
}
 string name, string data:
name ≠ null ⋀ data ≠ null ⇒
equals(
ReadResource(name,WriteResource(name,data)),
data)
Parameterized Unit Tests (PUTs) commonly supported by
various test frameworks
 .NET: Supported by .NET test frameworks
 http://www.mbunit.com/
 http://www.nunit.org/
 …

Java: Supported by JUnit 4.X
 http://www.junit.org/
Generating test inputs for PUTs supported by tools
 .NET: Supported by Microsoft Research Pex
 http://research.microsoft.com/Pex/
 Java: Supported by Agitar AgitarOne
 http://www.agitar.com/
Goal: Create a parameterized unit test for TrimEnd..
 Refactor: Extract Method
[TestMethod]
public void TrimEndTest() {
var target = "Hello World";
var suffix = "World";
var result = ParameterizedTest(target, suffix);
Assert.IsTrue(result == "Hello ");
}
static string ParameterizedTest(string target, string suffix) {
var result = StringExtensions.TrimEnd(target, suffix);
return result;
}
Goal: Given a program with a set of input
parameters, automatically generate a set of input
values that, upon execution, will exercise as many
statements as possible
Observations:
 Reachability not decidable, but approximations
often good enough
 Encoding of functional correctness checks as
assertions that reach an error statement on failure
17
void PexAssume.IsTrue(bool c) {
if (!c)
throw new AssumptionViolationException();
}
void PexAssert.IsTrue(bool c) {
if (!c)
throw new AssertionViolationException();
}


Assumptions and assertions induce branches
Executions which cause assumption violations are
ignored, not reported as errors or test cases

Human
 Expensive, incomplete, …

Brute Force
 Pairwise, predefined data, etc…

Semi - Random: QuickCheck
 Cheap, Fast
 “It passed a thousand tests” feeling

Dynamic Symbolic Execution: Pex, Cute,EXE
 Automated white-box
 Not random – Constraint Solving
Choose next path
Code to generate inputs for:
void CoverMe(int[] a)
{
if (a == null) return;
if (a.Length > 0)
if (a[0] == 1234567890)
throw new Exception("bug");
}
F
F
a.Length>0
a==null
T
Solve
Constraints to solve
Data
Observed constraints
null
a==null
a!=null &&
!(a.Length>0)
a!=null &&
a.Length>0 &&
a[0]!=1234567890
{}
a!=null
a!=null &&
a.Length>0
Execute&Monitor
{0} condition
Negated
a!=null &&
a.Length>0 &&
a[0]==1234567890
{123…}
a!=null &&
a.Length>0 &&
a[0]==1234567890
T
Done: There is no path left.
a[0]==123…
F
T
Open CoverMe project in Visual Studio.
Right-click on CoverMe method, and select “Run
Pex”.
 Inspect results in table.


public static void CoverMe(int[] a)
{
if (a == null) return;
if (a.Length > 0)
if (a[0] == 1234567890)
throw new Exception("bug");
}

How to test this code?
(Actual code from .NET base class libraries)
[PexClass, TestClass]
[PexAllowedException(typeof(ArgumentNullException))]
[PexAllowedException(typeof(ArgumentException))]
[PexAllowedException(typeof(FormatException))]
[PexAllowedException(typeof(BadImageFormatException))]
[PexAllowedException(typeof(IOException))]
[PexAllowedException(typeof(NotSupportedException))]
public partial class ResourceReaderTest {
[PexMethod]
public unsafe void ReadEntries(byte[] data) {
PexAssume.IsTrue(data != null);
fixed (byte* p = data)
using (var stream = new UnmanagedMemoryStream(p, data.Length)) {
var reader = new ResourceReader(stream);
foreach (var entry in reader) { /* reading entries */ }
}
}
}


If the test does not throw an exception, it succeeds.
If the test throws an exception,
 (assumption violations are filtered out),
 assertion violations are failures,
 for all other exception, it depends on further annotations.

Annotations
 Short form of common try-catch-assert test code
 [PexAllowedException(typeof(T))]
 [PexExpectedException(typeof(T))]
A brief overview



Instrumentation framework “Extended Reflection”
Instruments .NET code at runtime
Precise data flow and control flow analysis
Pex
Code
Under
Test
Z3
ExtendedReflection


Pex listens to monitoring callbacks
Symbolic execution along concrete execution
Pex
Code
Under
Test
Z3
ExtendedReflection


Constraint solving with SMT solver Z3
Computes concrete test inputs
Pex
Code
Under
Test
Z3
ExtendedReflection


Execute with test inputs from constraint solver
Emit test as source code if it increases coverage
Pex
Code
Under
Test
Z3
ExtendedReflection



ExtendedReflection: Code Instrumentation
Z3: SMT constraint solver
Pex: Dynamic Symbolic Execution
Pex
Code
Under
Test
Z3
ExtendedReflection
class Point { int x;
static int GetX(Point p) {
if (p != null) return p.X;
else return -1; } }
ldtoken
Point::X
call __Monitor::LDFLD_REFERENCE
ldfld Point::X
call __Monitor::AtDereferenceFallthrough
br
L2
L1:
ldtoken
Point::GetX
Prologue
call __Monitor::AtBranchTarget
call __Monitor::EnterMethod
Record concrete
values
call __Monitor::LDC_I4_M1
brfalseL0
ldarg.0
to have allldc.i4.m1
information
call __Monitor::Argument<Point>
L2:
p ==pnull
L0: .try {
Calls towhen
buildthis method
is called
call
__Monitor::RET
.try {
(The
real
C#with
compiler
path
condition
no proper
stloc.0 context
call __Monitor::LDARG_0
null
Calls
will
perform
ldarg.0
output is actually moreleave L4
call __Monitor::LDNULL
} catch NullReferenceException {
symbolic computation
complicated.)
ldnull
‘
call __Monitor::AtNullReferenceException
call __Monitor::CEQ
rethrow
ceq
call __Monitor::BRTRUE
}
Epilogue L4: leave L5
brtrue
L1
call __Monitor::BranchFallthrough
} finally {
call __Monitor::LDARG_0
call
__Monitor::LeaveMethod
ldarg.0
Calls
to
build
…
endfinally
path condition
}
L5: ldloc.0
ret
Pex’s representation of symbolic values and state is similar to the ones used
to build verification conditions in ESC/Java, Spec#, …
Terms for
 Primitive types (integers, floats, …), constants, expressions
 Struct types by tuples
 Instance fields of classes by mutable ”mapping of references to values"
 Elements of arrays, memory accessed through unsafe pointers
by mutable “mapping of integers to values"
Efficiency by
 Many reduction rules, including reduction of ground terms to constants
 Sharing of syntactically equal sub-terms
 BDDs over if-then-else terms to represent logical operations
 Patricia Trees to represent AC1 operators
(including parallel array updates)

SMT-Solver (“Satisfiability Modulo Theories”)
 Decides logical first order formulas with respect to theories
 SAT solver for Boolean structure
 Decision procedures for relevant theories:
uninterpreted functions with equalities,
linear integer arithmetic, bitvector arithmetic, arrays, tuples

Model generation for satisfiable formulas
 Models used as test inputs

Limitations
 No decision procedure for floating point arithmetic

http://research.microsoft.com/z3
Parameterized Unit Testing
Code-Under-Test
Project
Test Project
Parameterized
Unit Tests
Pex
Generated Tests

Right-click on the code-under-test project
 In the Solution Explorer


Select “Pex”
Select “Create Parameterized Unit Test Stubs”
Generated Test Project
References
Microsoft.Pex.Framework
(and
more)
References
VisualStudio
UnitTestFramework
Contains source files with
generated Parameterized Unit
Test stubs

Right-click on Parameterized Unit Test
 For now, it just calls the implementation,
but the you should edit it, e.g. to add assertions

Select “Run Pex Explorations”
Input/Output table
Current
Exploration
Row = generated test
Issue bar
Parameterized
Column = parameterized testStatus
input
or
Important
messages
Unit
Test
output
here !!!
Test outcome
filtering
Failing
New Test
Fix available
Test
Passing
Test
Exception
Review test
Stack trace
Allow exception
Pex Explorer makes it easy to
digest the results of many
Parameterized Unit Tests, and
many generated test inputs
pex.exe
Rich information, used
byrendered
Pex developers
HTML
from XML report file:
Full event log historyIt is easy to programmatically extract
information!
Parameter
table
 Code
coverage
 etc…




Tell Pex which Type you are testing
 Code coverage
 Exception filtering
 Search prioritization
[PexClass(typeof(Foo))]
public class FooTest {
…

3 Domains
 UserCodeUnderTest
public void CodeUnderTest() { … }
 UserOrTestCode
[PexClass] public class FooTest {
[PexMethod] public void ParamTest(…) { …
 SystemCode
public class String {
public bool Contains(string value) { …

Namespace, type, method filters
 pex.exe test.dll /tf:Foo /mf:Bar

Partial match, case insensitive
 Full match: add “!”: /tf:Foo!
 Prefix: add “*”: /tf:Foo*
 Many other filters
 e.g. to select a “suite”: /sf=checkin
 For more information:
pex.exe help filtering
Goal: Explore input generation techniques with TrimEnd


Write or discuss a random input generator for TrimEnd
Adapt your parameterized unit tests for Pex
 Start over with “Create Parameterized Unit Test Stubs”, or…
▪ Add reference to Microsoft.Pex.Framework
▪ Add using Microsoft.Pex.Framework;
▪ Add [PexClass(typeof(StringExtensions))] on
StringExtensionsTest
▪ Add [PexMethod] on your parameterized tests


Right-click in test and ‘Run Pex Explorations’
Compare test suites
It’s called Parameterized Unit Testing

The yellow event bar notifies about
important events, including certain
Click
on
issue
limitations
kind for more
information

You should act on these events:
 Refactor your code, or
 tell Pex to ignore it in the future,
 let Pex analyze (“instrument”) more code, if
possible.
Understand the
issue,
Clickand
on atake
particular
anevent
action
for more
information

If Pex reports that some code was uninstrumented,
you may tell Pex to instrument and analyze it
(if possible)

Code instrumentation on Demand
 Instrumentation has high performance overhead
 Some parts of the code better ignored

Use PexInstrument… attributes
[assembly: PexInstrumentAssembly(“Foo”)]

Pex will often suggest and insert those
attributes for you

Pex understand managed .NET code only
 Pex does not understand native code.

Problem if branching over values obtained from the
environment
 Pex may not automatically detect all such cases.
File System?
if (!File.Exists(f)) throw ...


Pex analyzes every executed .NET instruction
Some used libraries may be surprisingly expensive
to analyze
 XML parsing
 repeatedly converting data between different
representations
void Sum(string[] A) {
Don’t do this.
var sum = “0”;
foreach(var a in A)
sum = (int.Parse(a) + int.Parse(sum)).ToString();
if(sum == “123”) throw new Exception();
}
There are decision procedures for individual path
conditions, but…
 Number of potential paths grows
exponentially with number of branches
 Reachable code not known initially
 Without guidance, same loop might be
unfolded forever


To deal with the path explosion problem, Pex
uses search heuristics.
Heuristics try to create “diversity”, i.e. they
prefer paths which are so far
underrepresented e.g. by their coverage
vectors


In the presence of loops and recursion,
Pex could search forever, so Pex has many
ways to bound the search
Named arguments of [PexMethod]
[PexMethod(MaxConstraintSolverTime = 4)]

Pex will often suggest and update those
bounds for you.

Configurable bounds include:
 TimeOut
 MaxBranches
 MaxCalls
 MaxConditions
▪ Number of conditions that depend on test inputs
 MaxRuns
 ConstraintSolverTimeOut
 ConstraintSolverMemoryLimit
 Division and modulo are expensive
to reason about
 No decision procedures for floatingpoint arithmetic
 Pex currently uses heuristic search
techniques for floating-point constraints
Initially, choose Arbitrary
Path-constraint
solving is just
hard.
Test
Inputs
Constraint
System
Reachability is
undecidable!
(Loops)
Execution Path
(With external
Known
functions and
Paths
dynamic loading,
summaries
are difficult.)


Pex only explores single-threaded code
Related approach to explore threadschedules (but not input parameters): CHESS
http://research.microsoft.com/en-us/projects/chess/




Write assertions and Pex will try to break
them
Without assertions, Pex can only find
violations of runtime contracts causing
NullReferenceException,
IndexOutOfRangeException, etc.
Assertions leveraged in product and test code
Pex can leverage Code Contracts (discussed
later)



Pex only uses public methods to configure
non-public object fields
Heuristics built-in to deal with common types
User can help if needed
void (Foo foo) {
if (foo.Value == 123) throw …
[PexFactoryMethod]
Foo Create(Bar bar) {
return new Foo(bar);
}
Goal: Understand limitations of DSE.
 PexGoal/PexExpectedGoals
 Apply Pex to
if (File.Exists(fileName))
PexGoal.Reached();
if (DateTime.Parse(s).Year == 2009)
PexGoal.Reached();
class Foo { private Foo() {} }
…
if (foo != null) PexGoal.Reached();
Tips and tricks
Problem:
• Constraint solver determines
test inputs = initial state of test
• Most classes hide their state (private fields)
• State is initialized by constructor, and can be
mutated only by calling methods
• What sequence of method calls reaches a given
target state?
• There may be no such sequence
• In general, undecidable
Two approaches:
• (Guided) exploration of constructor/mutator methods
• Testing with class invariants
Specification:
[PexMethod]
public void ArrayListTest(ArrayList al, object o)
{
PexAssume.IsTrue(al != null);
int len = al.Count;
al.Add(o);
PexAssert.IsTrue(al[len] == o);
}

Factory methods
 Parameterized factories create and configure objects
 Explicit: Public static method annotated with
[PexFactoryMethod]
 Implicit: Pex guesses “best” constructor/setters

Result: Exploration of reachable states
 Reachable using factory method
 Reachable within configured bounds
 Under-approximation of possible states
We will now describe how you can create objects
via the exploration of a class invariant.
 Write class invariant as boolean-valued
parameterless method
 Refers to private fields
 Must be placed in implementation code

Write special constructor for testing only
 May be marked as "debug only"
 Constructor sets fields, assumes invariant

Result: Exploration of feasible states
 May include states that are not reachable
Open project “ArrayListSample”.
 Explore AddTestExplicit
 Explore AddTestImplicit
 Inspect generated test cases
 Notice constructor implicitly used by Pex


Edit code of ArrayList class:
Add “ad-hoc” Invariant method:
private bool Invariant() {
return this._items != null &&
this._size >= 0 &&
this._items.Length >= this._size;
}

Add DEBUG-only constructor that allows to configure object
freely, and then checks invariant:
#if DEBUG
public ArrayList(object[] items, int size) {
this._items = items;
this._size = size;
if (!this.Invariant()) throw new Exception();
}
#endif
Tips and tricks

Assume, Arrange, Act, Assert
[PexMethod]
void Add(List target, T value) {
PexAssume.IsNotNull(target); // assume
var count = target.Count; // arrange
target.Add(value); // act
Assert(target.Count == count + 1)//assert
}

For an API f(x),
f-1(f(x)) = x
for all x
void PropertyRoundtrip(string value) {
var target = new Foo();
target.Name = value;
var roundtripped = target.Name;
Assert.AreEqual(value, roundtripped);
}

For an API f(x),
f-1(f(x)) = x
for all x
void ToStringParseRoundtrip(int value) {
string s = value.ToString();
int parsed = int.Parse(s);
Assert.AreEqual(value, parsed);
}

For an API f(x),
f-1(f(f-1(x)) = f-1(x)
for all x
void ParseToString(string x) {
var normalized = int.Parse(x);
var intermediate = normalized.ToString();
var roundtripped = int.Parse(intermediate);
Assert(normalized == roundtripped);
}

Observe a state change
void ContainedAfterAdd(string value) {
var list = new List<string>();
list.Add(value);
Assert.IsTrue(list.Contains(value));
}

Given two methods f(x) and g(x), and a method b(y)
that observes the result or the exception behavior of a
method, assert that f(x) and g(x) have same observable
behavior under b, i.e. b(f(x)) = b(g(x)) for all x.
public void ConcatsBehaveTheSame(
string left, string right)
{
PexAssert.AreBehaviorsEqual(
() => StringFormatter.ConcatV1(left, right),
() => StringFormatter.ConcatV2(left, right));
}

Given two implementations f and g of the same
function, each possible requiring a different number of
steps, i.e. f(x)=f1(f2(…(fn(x)…)), and g(x)=g1(g2(…
(gm(x)…)), then it should hold that
f1(f2(…(fn(x))…) = g1(g2(…(gm(x)…)) for all x.
string Multiply(string x, string y);
int Multiply(int x, int y);
void CommutativeDiagram1(int x, int y) {
string z1 = Multiply(x, y).ToString();
string z2 = Multiply(x.ToString(), y.ToString());
PexAssert.AreEqual(z1, z2);
}

Allowed exception -> negative test case
[PexAllowedException(typeof(ArgumentException))]
void Test(object item) {
var foo = new Foo(item) // validates item
// generated test (C#)
[ExpectedException(typeof(ArgumentException))]
void Test01() {
Test(null); // argument check
}

Indicate which portions of a PUT should be
reachable.
[PexExpectedGoals]
public void InvariantAfterParsing(string input)
{
ComplexDataStructure x;
bool success = ComplexDataStructure.TryParse(
input, out x);
PexAssume.IsTrue(success);
PexGoal.Reached();
x.AssertInvariant();
}

Generated test asserts any observed value
 Return value, out parameters, PexGoal
int AddTest(int a, int b) {
return a + b; }

When code evolves, breaking changes in observable
will be discovered
void AddTest01() {
var result = AddTest(0, 0);
Assert.AreEqual(0, result);
}
Goal: Identify and apply patterns
 Apply patterns to parameterized unit tests of
TrimEnd.
Unit test: while it is debatable what a ‘unit’ is, a ‘unit’
should be small.
 Integration test: exercises large portions of a system.


Observation: Integration tests are often “sold” as
unit tests
White-box test generation does not scale well to
integration test scenarios.
 Possible solution: Introduce abstraction layers, and
mock components not under test

AppendFormat(null, “{0} {1}!”, “Hello”, “World”);

“Hello World!”
.Net Implementation:
public StringBuilder AppendFormat(
IFormatProvider provider,
char[] chars, params object[] args) {
if (chars == null || args == null)
throw new ArgumentNullException(…);
int pos = 0;
int len = chars.Length;
char ch = '\x0';
ICustomFormatter cf = null;
if (provider != null)
cf = (ICustomFormatter)provider.GetFormat(typeof(ICustomFormatter));
…


Introduce a mock class which implements the interface.
Write assertions over expected inputs, provide concrete outputs
public class MFormatProvider : IFormatProvider {
public object GetFormat(Type formatType) {
Assert.IsTrue(formatType != null);
return new MCustomFormatter();
}
}

Problems:
 Costly to write detailed behavior by example
 How many and which mock objects do we need to write?


Introduce a mock class which implements the interface.
Let an oracle provide the behavior of the mock methods.
public class MFormatProvider : IFormatProvider {
public object GetFormat(Type formatType) {
…
object o = call.ChooseResult<object>();
return o;
}
}

Result: Relevant result values can be generated by white-box
test input generation tool, just as other test inputs can be
generated!
92



Show AppendFormat code
Show AppendFormat test
Run Pex
93

Chosen values can be shaped by assumptions
public class MFormatProvider : IFormatProvider {
public object GetFormat(Type formatType) {
…
object o = call.ChooseResult<object>();
PexAssume.IsTrue(o is ICustomFormatter);
return o;
}
}

(Note: Assertions and assumptions are “reversed” when
compared to parameterized unit tests.)
94

Choices to build parameterized models
class PFileSystem : IFileSystem {
// cached choices
PexChosenIndexedValue<string,string> files;
string ReadFile(string name) {
var content = this.files[name];
if (content == null)
throw new FileNotFoundException();
return content;
}}
Goal: Test a ‘CopyFile’ method with a File System model
interface IFileSystem {
string ReadFile(string fileName);
void WriteFile(string name, string content);
}
Write a CopyFile method using IFileSystem
Implement IFileSystem with System.IO.File
and use it to test Copy.
 Write a parameterized model for IFileSystem and
use it to test Copy.


Design By Contracts
meets
Automated Whitebox Testing

http://research.microsoft.com/en-us/projects/contracts/

Library to state preconditions, postconditions,
invariants
Supported by two tools:

 Static Checker
 Rewriter: turns Code Contracts into runtime checks

Pex analyses the runtime checks
 Contracts act as Test Oracle


Pex may find counter examples for contracts
Missing Contracts may be suggested
If present, Pex leverages [ContractInvariantMethod] to create
objects via reflection and checking if invariant holds:
 Install Code Contracts:
Add reference to Microsoft.Contracts library
 Add empty [ContractInvariantMethod]
 Run Pex, click on “Add Invariant”
 Repeat, edit manually
[ContractInvariantMethod]
protected void Invariant() {
Contract.Invariant(this._items != (object[])null);
Contract.Invariant(
(uint)(this._size) <= (uint)(this._items.Length));
}



Code Contracts allow specification of interface contracts
Pex‘ stubs framework can leverage contracts to check
input and restrict output of mock objects.
Consider the following interfaces annotated with
contracts (using Spec# notation):
interface IFormatProvider {
object GetFormat(Type formatType)
requires
formatType != null;
ensures
result != null &&
}
formatType.IsAssignableFrom(result.GetType());





Create IFormatProvider interface with Code Contracts
in code-under-test project
Create (parameterized) test project
Show stubs
Create parameterized unit test using IFormatProvider
Explore parameterized unit test


Random input generators: many
Combining random testing and constraint solving
 DART, CUTE, EXE, KLEE, CREST (C)
 jCUTE, jFuzz (Java)
 SAGE (X86)
 …

Program model checkers
 JPF, Kiasan/KUnit (Java), XRT (.NET)

Commercial tools
 AgitarOne, …
103
http://research.microsoft.com/pex
http://codeplex.com/Pex
https://sites.google.com/site/asergrp/