Kiran Lakhotia, Mark Harman and Hamilton Gross in AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems

Publication Trend in SBSE

Spread of Activities in SBSE

Yet Another Search Based Testing Tool?

Others?

IGUANA for C (Phil McMinn) eTOC – for JAVA (Paolo Tonella)

Yet Another Search Based Testing Tool?

AUSTIN IGUANA for C (Phil McMinn) Others?

eTOC – for JAVA (Paolo Tonella)

Outline AUSTIN • Introduction to Austin • Overview of ETF ETF Empirical Study • Hypotheses, Test Subjects • Results and Analysis Conclusion • ETF or AUSTIN?

AUgmented Search based TestINg c source files run AUSTIN gcc -P -E preprocessed c source files CIL AST, CFG gcc -o sut.exe

instrumented c source instrument

AUgmented Search based TestINg } void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target

AUgmented Search based TestINg } void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target Inputs: key1 = 87, key2 = 33, p = 0, q = 0

AUgmented Search based TestINg } void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target Inputs: key1 = 88 , key2 = 33, p = 0, q = 0

AUgmented Search based TestINg } void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target Inputs: key1 = 86 , key2 = 33, p = 0, q = 0

AUgmented Search based TestINg } void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target Inputs: key1 = 84 , key2 = 33, p = 0, q = 0

AUgmented Search based TestINg } void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target Inputs: key1 = 80 , key2 = 33, p = 0, q = 0

AUgmented Search based TestINg } void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target Inputs: key1 = 80 , key2 = 33, p = 0, q = 0 Fitness Function: approach_level + norm(branch_distance) Search Method: Alternating Variable Method

AUgmented Search based TestINg } void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target Inputs: key1 = 33 , key2 = 33, p = 0, q = 0

AUgmented Search based TestINg } void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target Inputs: key1 = 33, key2 = 33, p = 0, q = 0 Symbolic Path Condition: key1 == key2 && p == 0

AUgmented Search based TestINg } void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target Inputs: key1 = 33, key2 = 33, p = 0, q = 0 Symbolic Path Condition: key1 == key2 && p == 0 Simplified Path Condition: p == 0

AUgmented Search based TestINg } void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target Inputs: key1 = 33, key2 = 33, p = 0, q = 0 Symbolic Path Condition: key1 == key2 && p == 0 Simplified Path Condition: p == 0 Solve: p ≠ 0

AUgmented Search based TestINg } void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target Inputs: key1 = 33, key2 = 33, p = 0, q = 0 Symbolic Path Condition: key1 == key2 && p == 0 Simplified Path Condition: p == 0 Solve: p ≠ 0 p 0

AUgmented Search based TestINg } void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target Inputs: key1 = 33, key2 = 33, p = 0x...

, q = 0

AUgmented Search based TestINg } void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target Inputs: key1 = 33, key2 = 33, p = 0x...

, q = 0 Symbolic Path Condition: key1 == key2 && p ≠ 0 && p ≠ q Simplified Path Condition: p ≠ 0 && p ≠ q Solve: p ≠ 0 && p == q q

AUgmented Search based TestINg } void testme(key1, key2, p, q) int key1; int key2; int* p; int* q; { if(key1 == key2) if(p != 0) if(p == q) //target Inputs: key1 = 33, key2 = 33, p = 0x..., q = 0x...

rZSlJ3hb Evolutionary Testing Framework (ETF)  ETF was developed as part of the EvoTest project  Integrated into the Eclipse IDE  Supports white & black box testing  Uses Evolutionary Algorithms (through the GUIDE, EO evolutionary library interface) to generate test data ETF Reinsertion Selection Send Fitness Values GUIDE + EO Send Individuals Mutation Recombination

rZSlJ3hb ETF cont.

Screenshots taken from a talk by Peter M. Kruse from Berner & Mattner, April 2009

rZSlJ3hb ETF cont.

Screenshots taken from a talk by Peter M. Kruse from Berner & Mattner, April 2009

Fitness Function and Algorithm  Both ETF and AUSTIN use the standard fitness function for branch coverage in Search Based Testing  apporach_level + norm(branch_distance)  normalization function: 1 - 1.001

-branch_distance  ETF uses a Genetic Algorithm  AUSTIN uses the Alternating Variable Method

Pointers in ETF and AUSTIN   AUSTIN   ETF Construct an equivalence of memory locations based on = and ≠ operators in symbolic path condition Initialize (pointer) input variables based on the graph   Assign new memory location via malloc or use existing memory location On-demand pointer handling   Generate an index and auxiliary variable for each pointer input Assign all input variables of the same type (including globals) to a pool  Optimize indices and values of auxiliary variables along with the inputs. Indices are used to select variables from pools and assign their address to the pointer input

Pointers in ETF struct cell{ int value; struct cell* next; }; int globX; double globDbl; struct cell* pCell; } { void testme(int* p, int* q) ...

globDbl val_p, val_q, globX pCell p = &globX; q = &val_p; pool of input parameters and auxiliary variables

Hypothesis

Null Hypotheses

AUSTIN is as effective as the ETF in achieving branch coverage.

AUSTIN is equally as efficient as the ETF in achieving branch coverage of a function.

Alternate Hypotheses

AUSTIN is more effective than the ETF in achieving branch coverage.

AUSTIN is more efficient than the ETF in achieving branch coverage of a function.

Empirical Study  3 case studies (comprising 8 functions)  Embedded Software Modules from Automotive Industry B

Case Study

C D

Description

Adaptive Headlight Control Software Door-Lock Control Software Electric Windows Control Software

LOC - Tested Total Branches

1,236 598

Nr. Inputs

132 383 339 315 248 71 31

Coverage ETF vs AUSTIN

Efficiency ETF vs AUSTIN

Coverage Random vs AUSTIN

Efficiency Random vs AUSTIN

ETF or AUSTIN?

 Coverage is about the same  AUSTIN is overall a little bit more efficient than the ETF in terms of fitness evaluations. In terms of wall clock time the difference is greater (due to the overhead of the ETF-GUIDE setup)  AUSTIN does not construct random memory graphs  ETF is not able generate inputs such as } struct cell{ int mem; }; void testme(int* p, struct cell* c) { if( p == &c->mem ) //target

Summary

Summary http://www.cs.ucl.ac.uk/staff/K.Lakhotia/software/austin.htm