Transcript Aditya V. Nori, Sriram K. Rajamani Microsoft Research India  An industrial strength program verifier  Idea: Synergize verification and testing  Synergy [FSE ’06], Dash.

Aditya V. Nori, Sriram K. Rajamani
Microsoft Research India

An industrial strength program verifier

Idea: Synergize verification and testing

Synergy [FSE ’06], Dash [ISSTA ‘08], SMASH [POPL
‘10] algorithms to perform scalable analysis

Engineered a number of optimizations for scalability

Integrated with Microsoft’s Static Driver Verifier (SDV)
toolkit and used internally

Share our experiences in making Yogi robust, scalable and
industrial strength

Several of the implemented optimizations are folklore
 Very difficult to design tools that are bug free ⇒ evaluating
optimizations is hard!
 Our empirical evaluation gives tool builders information about
what gains can be realistically expected from optimizations

Vanilla implementation of algorithms:
 (flpydisk, CancelSpinLock) took 2 hours

Algorithms + engineering + optimizations:
 (flpydisk, CancelSpinLock) took less than 1 second!

Overview of Yogi

Overview of optimizations

Evaluation setup

Empirical Results

Summary
Question
void foo()
{
*p = 4;
*q = 5;
if (condition)
error();
}
Is error() unreachable for all possible inputs?
Verification: can prove the absence of bugs,
but can result in false errors
Testing: finds bugs, but can’t prove their
absence
Input:
Program P
Property ψ
Construct initial abstraction
Construct random tests
Test
succeeded?
yes
Bug!
no
Abstraction
succeeded?
no
τ = error path in abstraction
f = frontier of error path
yes
Can extend
test beyond
frontier?
no
Refine abstraction
yes
Proof!
Input:
Program P
Property ψ
×
y=1
1
Construct initial abstraction
Construct random tests
Test
succeeded?
××
yes
Bug!
no
Abstraction
succeeded?
no
τ = error path in abstraction
f = frontier of error path
yes
0
yes
Proof!
void foo(int y)
{
0: int x, lock = 0;
1: do {
2:
lock = 1;
3:
x = y;
4:
if (*) {
5:
lock = 0;
6:
y = y+1;
}
7: } while (x != y);
8: if (lock != 1)
9:
error();
10:
}
Can extend
test beyond
frontier?
2
××
3
××
4
××
×
×
5
7
××
6
×
8
no
Refine abstraction
9
Symbolic execution +
Theorem proving
×
10
Input:
Program P
Property ψ
×
1
Construct initial abstraction
Construct random tests
Test
succeeded?
××
yes
Bug!
no
Abstraction
succeeded?
no
τ = error path in abstraction
f = frontier of error path
yes
Can extend
test beyond
frontier?
0
yes
Proof!
void foo(int y)
{
0: int x, lock = 0;
1: do {
2:
lock = 1;
3:
x = y;
4:
if (*) {
5:
lock = 0;
6:
y = y+1;
}
7: } while (x != y);
8: if (lock != 1)
9:
error();
10:
}
2
××
3
××
4
××
×
×
5
7
××
6
8:¬ρ
×
8:ρ
×
no
9
Refine abstraction
×
10
Input:
Program P
Property ψ
×
× ×1
Construct initial abstraction
Construct random tests
Test
succeeded?
yes
Bug!
no
Abstraction
succeeded?
no
τ = error path in abstraction
f = frontier of error path
0
yes
Proof!
void foo(int y)
{
0: int x, lock = 0;
1: do {
2:
lock = 1;
3:
x = y;
4:
if (*) {
5:
lock = 0;
6:
y = y+1;
}
7: } while (x != y);
8: if (lock != 1)
9:
error();
10:
}
× ×2
×× 3
4:s
×4:¬s
×
5:s
×5:¬s
6:r
6:¬r
yes
Can extend
test beyond
frontier?
no
Refine abstraction
×
7:q
×7:¬q
×
8:p
8:¬p
×
10
9

Initial abstraction from property predicates

Relevance heuristics for predicate abstraction
 Suitable predicates (SP)
 Control dependence predicates (CD)

Interprocedural analysis
 Global modification analysis
 Summaries for procedures

Thresholds for tests

Fine tuning environment models

Benchmarks:
 30 WDM drivers and 83 properties (2490 runs)
 Anecdotal belief: most bugs in the tools are usually
caught with this test suite

Presentation methodology:
 Group optimizations logically such that related
optimizations are in the same group
 Total time taken, total number of defects found for
every possible choice of enabling/disabling each
optimization in the group
state {
enum {Locked = 0, Unlocked = 1}
state = Unlocked;
}
KeAcquireCancelSpinlock.Entry
{
if (state != Locked) {
state = Locked;
}
else
abort;
}
KeReleaseCancelSpinlock.Entry
{
if (state == Locked) {
state = Unlocked;
}
else
abort;
}
0
𝑇
1
𝑇
0
(𝑠𝑡𝑎𝑡𝑒 = 𝐿𝑜𝑐𝑘𝑒𝑑)
0
(𝑠𝑡𝑎𝑡𝑒 ≠ 𝐿𝑜𝑐𝑘𝑒𝑑)
1
(𝑠𝑡𝑎𝑡𝑒 = 𝐿𝑜𝑐𝑘𝑒𝑑)
1
(𝑠𝑡𝑎𝑡𝑒 ≠ 𝐿𝑜𝑐𝑘𝑒𝑑)
Abstraction
using SLIC
predicates
Total time
(minutes)
#defects
#timeouts
yes
2160
241
77
no
2580
241
86
16%
Irrelevant?
A

𝑇
𝜌 = 𝑊𝑃 𝑎𝑠𝑠𝑢𝑚𝑒 𝜙 , 𝛿 = 𝜙 ∧ 𝛿
B
C
𝑇
𝑇
𝑎𝑠𝑠𝑢𝑚𝑒(𝜙)

Avoid irrelevant conjuncts
D
𝛿
 𝜌 = 𝛿?
A
𝑇
B
𝑇
C
¬𝜌
C
𝜌
𝑎𝑠𝑠𝑢𝑚𝑒(𝜙)
D
𝛿

Abstract assume statements that are not
potentially relevant by skip statements

If Yogi proves that the program satisfies
property, we are done.

Otherwise, validate the error trace and refine
the abstraction by putting back assume
statements, if the error trace is spurious
int x;
void foo() {
bool protect = true;
…
if (x > 0)
protect = false;
…
if (protect)
KeAcquireCancelSpinLock();
for (i = 0; i < 1000; i++) {
a[i] = readByte(i);
}
if (protect)
KeReleaseCancelSpinLock();
A
𝑇
B
C
𝑇
𝑇
𝑎𝑠𝑠𝑢𝑚𝑒(𝑖 > 1000)
D
𝑠𝑡𝑎𝑡𝑒 = 𝑙𝑜𝑐𝑘𝑒𝑑
𝜌 = state = Locked ∧ (𝑖 > 1000)
A
𝑇
B
𝑇
C
¬𝜌
C
𝜌
𝑎𝑠𝑠𝑢𝑚𝑒(𝑖 > 1000)
}
D
𝑠𝑡𝑎𝑡𝑒 = 𝑙𝑜𝑐𝑘𝑒𝑑
int x;
void foo() {
bool protect = true;
…
if (x > 0)
protect = false;
…
if (protect)
KeAcquireCancelSpinLock();
for (i = 0; i < 1000; i++) {
a[i] = readByte(i);
}
if (protect)
KeReleaseCancelSpinLock();
A
𝑇
B
C
𝑇
𝑇
𝑎𝑠𝑠𝑢𝑚𝑒(𝑖 > 1000)
D
𝑠𝑡𝑎𝑡𝑒 = 𝑙𝑜𝑐𝑘𝑒𝑑
𝜌 = state = Locked
A
𝑇
B
𝑇
C
¬𝜌
C
𝜌
𝑎𝑠𝑠𝑢𝑚𝑒(𝑖 > 1000)
}
D
𝑠𝑡𝑎𝑡𝑒 = 𝑙𝑜𝑐𝑘𝑒𝑑
int x;
void foo() {
bool protect = true;
…
if (x > 0)
protect = false;
…
if (protect)
KeAcquireCancelSpinLock();
for (i = 0; i < 1000; i++) {
a[i] = readByte(i);
}
if (protect)
KeReleaseCancelSpinLock();
}
SP
heuristic
CD
heuristic
Total time
(minutes)
#defects
#timeouts
yes
yes
2160
241
77
yes
no
2580
239
91
no
yes
2400
238
87
no
no
2894
235
174
10%
SP
heuristic
CD
heuristic
Total time
(minutes)
#defects
#timeouts
yes
yes
2160
241
77
yes
no
2580
239
91
no
yes
2400
238
87
no
no
2894
235
174
16%
SP
heuristic
CD
heuristic
Total time
(minutes)
#defects
#timeouts
yes
yes
2160
241
77
yes
no
2580
239
91
no
yes
2400
238
87
no
no
2894
235
174
25%

Yogi performs a compositional analysis
 𝜙1 , 𝑃, 𝜙2 : Is it possible to execute 𝑃 starting
from state 𝜙1 and reach state 𝜙2 ?

Global modification analysis

May-Must analysis (SMASH, POPL 2010)
A
𝑇
B
C
𝑇
𝑇
𝑓𝑜𝑜(… )
D
𝑠𝑡𝑎𝑡𝑒 = 𝑙𝑜𝑐𝑘𝑒𝑑
𝜙1 , 𝑓𝑜𝑜(… ), 𝜙2
A
𝑇
B
𝑇
C
¬𝜌
C
𝜌
foo(…)
D
𝑠𝑡𝑎𝑡𝑒 = 𝑙𝑜𝑐𝑘𝑒𝑑
Modification Summaries
analysis
Total time #defects
(minutes)
#timeouts
yes
yes
2160
241
77
yes
no
2760
239
109
no
yes
3180
237
134
no
no
3780
236
165
32%
Modification Summaries
analysis
Total time #defects
(minutes)
#timeouts
yes
yes
2160
241
77
yes
no
2760
239
109
no
yes
3180
237
134
no
no
3780
236
165
28%
Modification Summaries
analysis
Total time #defects
(minutes)
#timeouts
yes
yes
2160
241
77
yes
no
2760
239
109
no
yes
3180
237
134
no
no
3780
236
165
42%

Yogi relies on tests for “cheap” reachability

Long tests ⇒
 avoiding several potential reachability queries
 results in too many states and thus memory
consumption

Test thresholds: time vs. space tradeoff
Test
threshold
Total time
(minutes)
#defects
#timeouts
250
2600
236
92
500
2160
241
77
1000
2359
240
88
1500
2400
239
89
if (DestinationString)
{
DestinationString->Buffer = SourceString;
}
// DestinationString->Length should be set to the
// length of SourceString. The line below is missing
// from the original stub SDV function
DestinationString->Length = strlen(SourceString);
if (SourceString == NULL)
{
DestinationString->Length = 0;
DestinationString->MaximumLength = 0;
}
Issue type
#issues
Integers used as pointers
8
Uninitialized variables
15
Type inconsistencies
9

Described optimizations implemented in Yogi

Evaluated optimizations on the WDM test suite

Empirical data used to decide which optimizations to
include in Yogi

We believe that this detailed empirical study of
optimizations will enable tool builders to decide which
optimizations to include and how to engineer their
tools

http://research.microsoft.com/yogi