An Effective Path Selection Strategy for Mutation Testing
Download
Report
Transcript An Effective Path Selection Strategy for Mutation Testing
Automating the Generation of
Mutation Tests
Mike Papadakis and Nicos Malevris
Department of Informatics
Athens University of Economics and Business
1
Test Data Generation Approaches
Symbolic execution
Select set of paths
Produce a system of algebraic constraints
Solve and generate test cases
Challenges (Symbolic execution)
Infeasible paths
Complex-unhandled expressions
Availability of source code
2
Test Data Generation Approaches
Search based approaches
Definition of program input
Dynamic program execution
Fitness function
o (efficiency and effectiveness)
Challenges
Handling of dynamic program inputs
Require a high number of executions
Handling of specific cases (e.g. flag problem)
Effective fitness function
3
Test Data Generation Approaches
Dynamic Symbolic Execution
Simultaneously perform actual and symbolic
execution
o Simplify the process based on concrete program
execution
Produce a system of algebraic constraints
Solve and generate test cases
Challenges
Infeasible paths
Scalability issues
4
Killing Mutants
In order to kill a mutant, tests must
Stage 1: Reach
the mutant
Stage 2: infect
the program state
Stage 3:
propagate the
infected state
5
Killing Mutants
Reach the mutant (Reachability condition)
Infect program state (necessity condition)
Program execution must cause a discrepancy in the
program state
E.g. Orig: a > b, Mut = a ≥ b → a > b ≠ a ≥ b
o a == b where Orig = false, Mut = true
Propagate (sufficiency condition)
Heuristic approaches
o explore path space, measure of mutant’s impact
Joint satisfaction (Reach && Infect && Propagate)
6
Killing Mutants (path selection)
Symbolic Execution
Reach
o Use paths from program input to mutant
Infect
o Use necessity constraints to infect program state
o Original Statement ≠ Mutated Statement
Propagate
o Explore the path space (from mutant to program output)
Path selection Strategy
7
Killing Mutants (path selection)
Enhanced Test Model (Enhanced Control Flow
Graph)
Represent mutants in the model
Guide the selection of paths and the produced
constraints based on the Shortest path strategy
8
Killing Mutants (path selection)
Shortest Path Strategy, the k-value
An efficient path selection strategy. (eliminates
the effects of infeasible paths)
Select shortest paths from program input (S) to
the mutant statement (m)
(S , x) xm
(k )
m
(k )
Heuristic approach to weak mutation
Strong mutation can be tackled incrementally
(k )
m
(S , x) xm mx ( x, F )
(r )
(t )
9
Killing Mutants Dynamic approaches
Dynamic approaches
Gain the required information from actual program
execution
Require high number of program execution
Mutation Testing
Need to unify the runtime information of the original
and mutant programs
Need to include mutant neccesity and sufficiency
conditions into program structure
Need for efficiency (high cost)
10
Mutant Schemata
Mutant schemata-Parameterized mutants
Technique for efficiently producing mutants
Embeds all mutants into one meta-mutant program
Use of global parameter to specify the mutants
Example
original
meta-mutant
a > b + c → RelationalGT(a, ArithmeticPlus(M(b), M(c)))
M(x)
Arithmetic(x, y)
Relational(x, y)
mutates variable x, e.g. abs(x)
mutates arithmetic operator, e.g. x - y
mutates relational operator, e.g. x ≥ y
11
Mutant Schemata (approach I)
Weak mutation
Include the mutant evaluation inside schematic
functions
o Original expression ≠ Mutant expression
o Original statement ≠ Mutant statement
After evaluation continue with the original execution
(schematic function returns the original expression)
Able to execute all mutants with one execution run
Embeds all mutant’s killable condition into
program structure (mutant evaluation → branches)
12
Mutant Schemata
Proposed approach (Mutants to branches)
Mutants M[1]…M[n] on node N.
Node
N-1
Mutant
Evaluation
Branches
Node
N_M[1]
Node
N-1
Node
N_M[2]
Node
N
...
Node
N
Mutant is killed
Mutant is Alive
Node
N_M[n-1]
Node
N+1
Node
N+2
Node
N+1
Node
N+2
Node
N_M[n]
13
Mutant Schemata (approach II)
Strong mutation
In Same lines as Weak
o Include the mutant evaluation inside schematic functions
After evaluation continue with the mutant execution
(schematic function returns the mutant expression)
Incremental approach, from weak to strong
Requires multiple execution runs
Record original and mutant’s execution paths
Evaluate strongly killed mutants
14
Killing Mutants (DSE)
Use the shortest path heuristic
Negate the condition that will lead closer to mutant
node (reach the mutant)
Weakly kill the mutant
Fulfill the mutant necessity condition
Make the mutant evaluation Branch true
Negate the produced conditions after the mutant
Start from the mutant program
Explores the mutant program path space
15
DSE-Example Scenario
Target
Mutant
16
DSE-Example Scenario
Target
Mutant
17
DSE-Example Scenario
Target
Mutant
18
DSE-Example Scenario
Target
Mutant
19
DSE-Example Scenario
Target
Mutant
Infect Mutant
Mutant is
alive
20
DSE-Example Scenario
Target
Mutant
Propagate
Mutant is
alive
21
DSE-Example Scenario
Target
Mutant
Mutant is
alive
22
DSE-Example Scenario
Target
Mutant
Mutant
Killed
23
DSE-Necessity condition
Example: a + b > k → a + abs(b) > k
RelationalGT(ArithmeticPlus(a, Abs(b)), k)
Abs(x) → if (x<0) //mutant necessity condition
a + b > k ≠ a + abs(b) > k //Statement level condition
Test: a=15, b=2, k=0
Conditions: b≥0 && a+b>k && …
Negates: b≥0 → b < 0
Produces: a=15, b=-10, k=0
24
DSE-Necessity condition
Example: a + b > k → a + abs(b) > k
RelationalGT(ArithmeticPlus(a, Abs(b)), k)
Abs(x) → if (x<0) //mutant necessity condition
a + b > k ≠ a + abs(b) > k //Statement level condition
Test: a=15, b=-10, k=0
Conditions: b<0 && a+b>k && a+abs(b)>k && …
Negates: a+abs(b)>k → b<0 && a+b>k && a+abs(b)≤k →
Infeasible
Negates: a+b>k → b<0 && a+b≤k
Produces: a=-15, b=-10, k=0
25
DSE-Necessity condition
Example: a + b > k → a + abs(b) > k
RelationalGT(ArithmeticPlus(a, Abs(b)), k)
Abs(x) → if (x<0) //mutant necessity condition
a + b > k ≠ a + abs(b) > k //Statement level condition
Test: a=-15, b=-10, k=0
Conditions: b<0 && a+b≤k && a+abs(b)≤k && …
Negates: a+abs(b)>k → b<0 && a+b≤k && a+abs(b)>k
Produces: a=5, b=-10, k=0
Mutant infected: Orig: 5-10>0 (false), Mut: 5+10>0 (true)
26
Killing Mutants (Search Based)
Weak-Strong mutation
Measure the closeness of reaching a mutant
Measure branch distance of mutant branches
o Closeness of weakly killing the targeted mutant (mutant
necessity condition)
Use simplified necessity fitness for improved
performance
Sufficiency condition can be approximated by
exploring the path space or based on mutants
impact
27
Fitness function
Approach level
Closeness of executing a targeted mutant
Calculated by the number of control dependent
nodes missed
Branch Distance
Closeness of flipping a specific branch
Mutation Distance
Closeness of weakly killing the targeted mutant
𝑓𝑖𝑡𝑛𝑒𝑠𝑠 = 2 ∗ 𝑎𝑝𝑝𝑟𝑜𝑎𝑐ℎ 𝑙𝑒𝑣𝑒𝑙
+ 𝑛𝑜𝑟𝑚𝑖𝑙𝑖𝑧𝑒𝑑 𝐵𝑟𝑎𝑛𝑐ℎ 𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒
+ 𝑛𝑜𝑟𝑚𝑖𝑙𝑖𝑧𝑒𝑑 𝑀𝑢𝑡𝑎𝑡𝑖𝑜𝑛 𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒
28
Mutant Fitness (example)
Operator Original expression
Relational
a>b
Arithmetic
a+b
Absolute
a
a && b
Logical
a || b
Mutant Fitness
a >= b: abs(a-b)
a < b: k
a <= b: 0
a != b: abs(a-b+k)
a == b: abs(a-b)
true: abs(a-b)
false: abs(a-b+k)
a - b:k
a % b:k
a * b:k
a:k
a / b:k
b:k
abs(a):abs(a+k)
-abs (a):abs(a)
0:abs(a)
a||b:min[Tfit(a)+Ffit(b), b:Ffit(a)+Tfit(b)
Ffit(a)+Tfit(b)]
true:min [Ffit(a), Ffit(b)]
a:Tfit(a)+Ffit(b)
false:Tfit(a)+Tfit(b)
a&&b:min[Tfit(a)+Ffit( b:Tfit(a)+Ffit(b)
b), Ffit(a)+Tfit(b)]
true:Ffit(a)+Ffit(b)
a:Ffit(a)+Tfit(b)
false:min[Tfit(a), Tfit(b)]
29
Path explosion problem
Mutants
Original program
8 program paths
Mutants
Mutant program
8 program paths per mutant
Mutants
Schematic program
Mutant evaluation decisions
8 * 2number of mutants paths
Mutants
Introduction of fake paths
Same of them may be useful
for higher order mutation
30
Handling of path explosion
Path selection strategy, (symbolic execution)
Use weights on the Enhanced Control Flow Graph
Shortest paths include only the original paths
Dynamic Symbolic Execution
Use the mutant parameters as local values
Dynamic production of constraints eliminates the fake
paths. Paths are produced dynamically
Search based approaches
Approach level contains only the control depended
nodes
o Ignores the path space
31
Case Studies
Dynamic Symbolic Execution (Strong mutation)
Program units: Tritype, Remainder
Siemens Programs: Schedule, Tcas, Replace
ABS, AOR and ROR operators
Search based (Weak mutation results)
Hill climbing approach (AVM)
Maximum 10 attempts per mutant
Program units: Tritype, Triangle, Remainder,
Calendar
ABS, AOR, ROR and LCR operators
32
DSE Study-Results
DSE Study-Results
DSE Study-Results
DSE Study-Results
DSE Study-Results
DSE Study-Results
Strong mutation results
Replace 86.5%, Tcas 100%, Schedule 91% (compared with
the accompanied test suite),
Remainder 97.5 and Tritype 96.8% (all killable mutants)
½ Iterations
Program
All Iterations
Killed
No. Mut.
Killed
Solver Calls
Solver Calls
Mutants
Executions Mutants
No. Mut.
Executions
Produced
Tests
Tritype
214
1012
500
216
1588
514
90
Remainder
234
2749
435
235
5498
741
268
Replace
514
14740
19400
520
30238
80471
8927
Tcas
136
1733
3729
137
3705
4720
422
Schedule
94
714
293
94
949
303
301
Search Based Study-Results
39
Search Based Study-Results
40
Search Based Study-Results
41
Search Based Study-Results
42
Search Based Study-Results
Weak mutation results
Test
Objects
No. of Killed Mutants
Mutation Score
Time
(Sec)
Random
Metallaxis
Random
Metallaxis
Tritype
168
245
66.4%
96.8%
42
Triangle
246
275
89.5%
100%
70
Remainder
238
242
98.3%
100%
402
Cal
237
252
93.3%
99.2%
44
43
Conclusion
Mutation based test case generation
Lack of attempts
First steps using dynamic, state of the art
techniques
Dynamic approaches can be adopted to
perform mutation
Search based approaches
o
Mutation distance leads to improved effectiveness
Dynamic Symbolic Execution
o
Quite effective
44
Conclusion
Test data generation approaches and tools
can be effectively extended to mutation
Practical Technique (schemata)
Future directions
Efficient
handling of infeasible paths
Handling of the “flag” problem
Efficient handling of equivalent mutants
Hybrid approaches
45
Thank you for your attention…
Questions ?
Contact
Mike Papadakis
Nicos Malevris
[email protected]
[email protected]
46
References
Mike Papadakis and Nicos Malevris. "Automatic Mutation Test Case
Generation Via Dynamic Symbolic Execution", in 21st International
Symposium on Software Reliability Engineering (ISSRE'10), San Jose,
California, USA, November 2010.
Mike Papadakis and Nicos Malevris. “Metallaxis an Automated Framework
for Weak Mutation", Technical Report,
http://pages.cs.aueb.gr/~mpapad/TR/MetallaxisTR.pdf.
Mike Papadakis, Nicos Malevris and Maria Kallia. "Towards Automating
the Generation of Mutation Tests", in Proceedings of the 5th International
Workshop on Automation of Software Test (AST'10), Cape Town, South
Africa, May 2010, pp. 111-118.
Mike Papadakis and Nicos Malevris. “An Effective Path Selection Strategy
for Mutation Testing", in Proceedings of the 16th Asia-Pacific Software
Engineering Conference (APSEC'09), Penang, Malaysia, December 2009,
pp. 422-429.
47