Transcript Verteilen von JUnit Tests - hs

JUnit test distribution
Bettina Scharpf
ACM 2
University of Applied Science
Furtwangen
Specification
“Develop a Grid Service that partitions JUnit tests,
does a balanced distribution and saves the results.”
Technologies:
• JUnit
• ANT
• Web and Grid Services
• Globus Toolkit 4 and GRAM
Proof of concept:
• partitioning
• efficient distribution
04/07/2005
Bettina Scharpf, ACM 2
2
Agenda
• Introduction to JUnit
• Test partitioning
– atoms
• Efficient distribution
– execution hosts
– algorithms
• Concept
04/07/2005
Bettina Scharpf, ACM 2
3
Introduction to JUnit
• tests visible behavior
– black and white box tests
• “units” of code -> Java classes
• by Erich Gamma and Kent Beck
• “test first”
04/07/2005
Bettina Scharpf, ACM 2
4
Introduction to JUnit
• Distinction between
– errors (e.g. exceptions)
and
– failures (e.g. value not as expected)
• test passes or not
- nothing inbetween
• Test runners
– with graphical user interface
– command line
– ...
04/07/2005
Bettina Scharpf, ACM 2
5
Organisation of JUnit tests
TestSuite
setUp()
TestSuite
TestCase
TestSuite
tearDown()
setUp()
setUp()
setUp()
setUp()
setUp()
TestCase
TestCase
TestCase
TestSetup
TestSuite
TestCase
tearDown()
tearDown()
tearDown()
tearDown()
tearDown()
setUp()
setUp()
setUp()
TestCase
TestCase
tearDown()
tearDown()
setUp()
TestCase
tearDown()
tearDown()
Test.run(testResult)
04/07/2005
Bettina Scharpf, ACM 2
6
Agenda
• Introduction to JUnit
• Test partitioning
– atoms
• Efficient distribution
– execution hosts
– algorithms
• Concept
04/07/2005
Bettina Scharpf, ACM 2
7
Test partitioning
Atoms in JUnit
• Single test cases
setUp()
TestCase
tearDown()
• Single test case wrapped by TestSetup’s
setUp() and tearDown()
setUp()
setUp()
setUp()
TestSetup
tearDown()
TestCase
tearDown()
setUp()
TestCase
tearDown()
tearDown()
=>
setUp()
setUp()
TestCase
tearDown()
tearDown()
+
setUp()
setUp()
TestCase
tearDown()
tearDown()
• but: TestSetup partitioning at any price?
04/07/2005
Bettina Scharpf, ACM 2
8
Agenda
• Introduction to JUnit
• Test partitioning
– atoms
• Efficient distribution
– execution hosts
– algorithms
• Concept
04/07/2005
Bettina Scharpf, ACM 2
9
Efficient distribution
Execution Hosts
•
•
Problem: Hosts perform differently
Solution: Equalize hosts
Run reference test
on each host
Find slowest host
Relate each host to the
slowest one (using factor)
Sum up known
execution times
Calculate execution time
per slowest host
Calculate quota for each
particular host (using factor)
04/07/2005
Bettina Scharpf, ACM 2
10
Efficient distribution
Algorithms
Optimal balance vs. preparation time:
• Check all possibilities (exponential)
– Knapsack:
must be run (n-1) times for n hosts
+ exact
- very high complexity
or
• Iterate: Assign largest test to host with
most capacity left
+ not very complex
+ for most cases exact enough
04/07/2005
Bettina Scharpf, ACM 2
11
Agenda
• Introduction to JUnit
• Test partitioning
– atoms
• Efficient distribution
– execution hosts
– algorithms
• Concept
04/07/2005
Bettina Scharpf, ACM 2
12
Concept
Problems:
1.) Partitioning:
Optimal partitioning
vs. longer cumulative execution time
2.) Efficient distribution:
Optimal balance
vs. test preparation time
Solution:
Must be something inbetween...
04/07/2005
Bettina Scharpf, ACM 2
13
Concept
Solutions
1.) Partitioning of TestSetup objects
– if necessary
• single test case contained in TestSetup object
is too big
• complete TestSetup object is too big
or
– if fixture is not too big, amount of
TestCases not too high (e. g. depending
on the total execution time)
04/07/2005
Bettina Scharpf, ACM 2
14
Concept
Solutions
2.) Balancing the tests
– use knapsack for a small number
of test cases (configurable)
-> optimal “packaging” of tests for
one host (each test has a weight
and a value which are both the same here)
-> iterate:
remove assigned tests and “pack a knapsack”
for the next host
else
– “expensive first”
• assign tests iteratively, starting with the most
expensive test case and the most performant host
04/07/2005
Bettina Scharpf, ACM 2
15
Concept
Solutions
• The problem of atoms which are “too big”
– has longer execution time than the most performant
host (or the most performance left)
– Solution
• put atom on the host with the most performance left
• calculate: how much longer will the test now take?
• re-calculate the execution time per slowest host
• apply “slowest host time” to all execution hosts
(won’t extend the entire test duration)
• Distributing tests with unknown execution
time: Round Robin, beginning with the most
performant host
04/07/2005
Bettina Scharpf, ACM 2
16
Concept
An Example
host
A
61.0
7
s
host
B
host
C
4
11
12s
14s
2.0
4
4
10.5s
91.5
s
2
1
3 s left
14
host
D
12s
14s
2.0
11
6
4
6.5
1.5
2.5 s left
0.5
1
3
2
4
8 s left
11
1
0 s left
test cases
4
1 1
04/07/2005
6
14
1 1
4
4
(39 sec.)
1
1
4
1
1.) obtain factor
2.) calculate tests
and execution
time per host
3.) distribute...
Bettina Scharpf, ACM 2
17
Fragen?
04/07/2005
Bettina Scharpf, ACM 2
18