Transcript IMUnit: Improved Unit Testing

CAPP: Change-Aware
Preemption Prioritization
Vilas Jagannath, Qingzhou Luo, Darko Marinov
Sep 6th 2011
Motivation
• Parallel code required for performance
– Dominant paradigm is shared-memory, multithreaded code
• Difficult to develop (correct) multithreaded code
– Different behavior under different schedules
– Afflicted by bugs like data races, deadlocks, atomicity violations…
• Difficult to test multithreaded code
– Exploration required, time consuming
– Current exploration techniques focus on one code version
– Code evolves, need efficient regression testing
• CAPP: Change-Aware Preemption Prioritization
2
Apache Mina Test for filterWrite()
3
@Test
public void testFilterWriteThreadSafety()
throws InterruptedException {
…
FilterWriteThread fwThread1 = new FilterWriteThread();
FilterWriteThread fwThread2 = new FilterWriteThread();
fwThread1.start(); fwThread2.start();
fwThread1.join(); fwThread2.join();
assertEquals(0, fwThread1.result);
assertEquals(0, fwThread2.result);
}
class FilterWriteThread extends Thread {
int result = 0;
ProtocolCodecFilter pc;
…
public void run() {
try {
pc.filterWrite(nextFilter, session, writeRequest);
…
} catch (Exception e) {
e.printStackTrace(); result = 1;
} } }
Exploration
S0
Transition T1
Preemption
T2
S1
T1
S..
T2
Schedule
S2
T1
S3
4
S..
T2
S..
(S0, T1) (S0, T2)
T1
S..
Multithreaded Testing
Code
Test
Test execution
explores many
schedules
5
Exploration Order
Work Pool
S0
(S0, T1) (S0, T2)
T1
S1
T1
6
S2
(S1, T1) (S1, T2)
Eg.
DFS: Stack
BFS: Queue
…
Exploration Prioritization
Next Work Pool
Current Work Pool
…..
Eg.
CHESS
Gambit
Active Testing
ConTest
AVIO
CTrigger
…
S0
T1
S1
T1
S2
7
(S.., T1) …..
(S0, T1) (S0, T2)
T2
(S1, T1) (S1,
S..T2)
(S.., T2)
T2
S..
Regression Multithreaded Testing
Code
Test
8
Code`
Change-Unaware Prioritization
Code
Test
9
Code`
Change-Unaware Prioritization
Code
Test
10
Code`
CAPP:
Change-Aware Preemption Prioritization
Code
Test
Code`
Use evolution information to prioritize
exploration of schedules that perform
preemptions at change impacted code.
CAPP is a general framework.
We instantiate with 14 heuristics.
11
Apache Mina r912148 to r912149
public void filterWrite(NextFilter nextFilter,
IoSession session, WriteRequest writeRequest)
throws Exception {
…
flushWithoutFuture();
…
}
public void flushWithoutFuture() {
Queue<Object> bufferQueue = getMessageQueue();
for (;;) {
Object encodedMessage = bufferQueue.poll();
encodedMessage == null) {
break;
}
if (// Flush only when the buffer has remaining.
if (!(encodedMessage instanceof IoBuffer)
|| ((IoBuffer)
encodedMessage).hasRemaining()) {
…
} } }
public void filterWrite(NextFilter nextFilter,
IoSession session, WriteRequest writeRequest)
throws Exception {
…
Queue<Object> bufferQueue = getMessageQueue();
while (!bufferQueue.isEmpty()) {
Object encodedMessage = bufferQueue.poll();
// Flush only when the buffer has remaining.
if (!(encodedMessage instanceof IoBuffer)
|| ((IoBuffer)
encodedMessage).hasRemaining()) {
…
} }
…
• DIRMINA-803
• 58 schedules to detect failure using CHESS
• 5 schedules using one of the CAPP heuristics
12
CAPP Parts
Impacted Code Points (ICP) Collection
(Static)
Change-Aware Exploration Prioritization
(Dynamic)
13
Impacted Code Points (ICP) Collection
Code
Code`
• Inspect structure diff to collect impacted:
– Classes, Methods, Lines, Fields
• Focus on changes to thread communication:
– Expand synchronized blocks, methods
– Fields accessed by changed lines
– Fields accessed in methods called by changed lines
14
Change-Aware Exploration Prioritization
Next Work Pool
Current Work Pool
S0
(S0, T1) (S0, T2)
Use ICPs to partition
T2
S1
T1
15
S2
(S1, T1) (S1, T2)
Use ICPs to partition
Family of Heuristics
• Defined by ICP Match Mode and Prioritization Mode
• ICP Match Mode: Is transition impacted by changes?
– Class (on Stack), Method (on Stack), Line (on Stack), Field
• Prioritization Mode: Which work pool to add transitions to?
– All, Some
• 14 heuristics
– CA (Class All)
–…
– MOS (Method on Stack Some)…
–…
16
ICP Match Mode
• ICP Match Mode: Is transition impacted by changes?
– Class (on Stack), Method (on Stack), Line (on Stack), Field
ICPs: {………, class A,………….....}
Class
T1: Executing
class A,class
method
m, linek,55
T2: Executing
B, method
line 32
S..
17
(S.., T1) (S.., T2)
Prioritization Mode
• Prioritization Mode: Which work pool to add transitions to?
– All: Only prioritize preemptions among impacted transitions
– Some: Prioritize preemptions among impacted and not-impacted
Current Work Pool
S..
18
Next Work Pool
(S.., T1) (S.., T2) (S.., T3)
Prioritization Mode
• Prioritization Mode: Which work pool to add transitions to?
– All: Only prioritize preemptions among impacted transitions
– Some: Prioritize preemptions among impacted and not-impacted
Current Work Pool
All
19
S..
Next Work Pool
(S.., T1) (S.., T2) (S.., T3)
Prioritization Mode
• Prioritization Mode: Which work pool to add transitions to?
– All: Only prioritize preemptions among impacted transitions
– Some: Prioritize preemptions among impacted and not-impacted
Current Work Pool
All
20
S..
Next Work Pool
(S.., T1) (S.., T2) (S.., T3)
Prioritization Mode
• Prioritization Mode: Which work pool to add transitions to?
– All: Only prioritize preemptions among impacted transitions
– Some: Prioritize preemptions among impacted and not-impacted
Current Work Pool
Some
21
S..
Next Work Pool
(S.., T1) (S.., T2) (S.., T3)
Prioritization Mode
• Prioritization Mode: Which work pool to add transitions to?
– All: Only prioritize preemptions among impacted transitions
– Some: Prioritize preemptions among impacted and not-impacted
Current Work Pool
Next Work Pool
T1
Some
22
S..
(S.., T1) (S.., T2) (S.., T3)
Implementations
• Two implementations
• Java PathFinder (JPF)
– Stateful exploration using checkpoints
– Depth first, unbounded
– CAPP implemented using a custom SimplePrioritySearch
• ReEx
– Stateless exploration using re-execution
– CHESS (iterative context bound 2)
– CAPP implemented using a custom SchedulingStrategy
23
Evaluation Subjects
Subject
Airline
Allocation
BoundedBuffer
BubbleSort
Deadlock
ReadersWriters
ReplWorkers
RAXextended
Groovy
Lang
Mina
Pool1
Pool2
Pool3
Pool4
SLOC
Threads
ICPs
136
209
110
89
52
154
432
166
54,872
48,369
34,804
10,042
4,473
10,802
10,783
6
3
9
4
3
5
3
6
3
3
3
3
3
2
3
6
5
2
4
3
2
2
2
60
3
36
148
201
29
47
• 15 multithreaded programs:
– Ranging from 52-54,000 loc, with 2-607 classes, running 2-9 threads
– Evolution from non-buggy version to buggy version
24
• 7 deadlock faults, 8 data race and atomicity violation faults
Experiments
• Independent variables:
– 14 heuristics
– Stateful vs. Stateless exploration
– Default exploration order vs. 50 Random exploration orders
• 19,890 explorations, around a month of computing time
• Measured savings in exploration cost compared to changeunaware exploration:
– Transitions for JPF implementation
– Schedules for ReEx implementation
25
Default Non-Randomized Exploration
Average Exploration Reduction
6
5.3x
Statefull
Stateless
5
4
2.7x
3
0.8x
2
1
0
CA
26
CS COA COS MA MS MOA MOS LA
Heuristics
LS LOA LOS FA
FS
CAPP
• Current multithreaded testing techniques focus on one
code version
• Code evolves, need techniques that leverage evolution
info
• CAPP uses evolution info to prioritize exploration:
– Up to 5.3x average reduction in exploration cost
• Ongoing work:
– Regression Schedule Selection [ICST 10, STVR in
revision]
– Schedule Expression [ESEC/FSE 11]
– Multithreaded Tests Generation
27
Possible Project Ideas
• Better impact analysis
• Combine selection and prioritization (RMC)
• Prioritize across tests
• Multiple faults
• Improve ReEx, revisit stateless randomized search (PRSS)
• Combine IMUnit with CAPP
– How to find/generate/simplify buggy IMUnit schedules?
– How to decide whether some IMUnit schedules is valid or not
after code evolves (a.k.a., identify bad concurrency bug fixes)?
28
UPCRC
• Universal Parallel Computing Research Center
– Funded by Intel and Microsoft
– Goal: improve parallelism for client computation
– Two centers: Berkeley and Illinois
• UPCRC Illinois
– About 20 faculty + 40 grad students
– Research on entire computation stack: architecture, runtime
systems, compilers, programming languages, design patterns,
applications… and testing
29
Results Summary
• Does CAPP help?
– Up to 5.3x reduction on average
– Only one heuristic (out of 28) increased cost (0.8x)
• How do heuristics compare?
– Field ICP Match Mode heuristics perform best
– ALL Prioritization Mode heuristics perform best
• Stateful vs. Stateless?
– Greater benefit for stateless exploration
• Default vs. Random?
– Conclusions from random results similar to default results
– Except ICP Match Mode
• Detailed results in the paper
30
Multicore World
Shared Memory
Multithreaded
Performance!
31
Correct
Difficult to Develop Multithreaded Code
Shared Memory
•
•
•
•
•
Multithreaded
32
Scheduling non-determinism
Data races
Deadlocks
Atomicity violations
…
Difficult to Test Multithreaded Code
Code
Test
33
• Exploration required
• Time consuming
• Current techniques focus
on one code version
• Code evolves
• Need efficient regression
testing