Transcript Final

An Evaluation
of BLAST
John Gallagher
CS4117
Overview
BLAST incorporates new, fascinating and
complex technology. The engine and
external components are evolving
monthly.
BLAST can be used successfully on
carefully scoped (systems) programs.
BLAST exists to serve software model
checking research and make it tangible. It
is not a good developer tool.
Criteria
How easy is it to generate the input?
How easy is it to generate the output?
How accurate is the output?
How precise is the output?
How broad are the applications of the tool?
What are the alternatives?
How much time or process does the tool add
or remove from software development?
Opportunity Cost
The Good – BLAST Framework
BLAST internals illustrate interesting
working examples of compiler technology:
int main() {
int i, x, y, ctr;
x = ctr;
ctr = ctr + 1;
y = ctr;
if (x == i) {
assert (y == i + 1);
}
}
:-)
The Good – BLAST Framework
Based on very recent research (earliest
work cited in corresponding paper was
from 2002).
Given good input, correct use of BLAST’s
myriad of runtime options, and good
specification, BLAST performs well.
Able to find real bugs that were fixed
based on counterexample information that
no other tools were providing.
The Good – BLAST Framework
Running time figures
and number of real
counterexample
traces make BLAST
look good. They are
the result of a solid
framework.
Favorable position of
having little
competition in terms
of results achievable.
Program
Predicates
Total Average
Lines
Time
preprocessed
(mins)
kbfiltr
12k
3
72
6.5
floppy
17k
25
240
7.7
diskprf
14k
13
140
10
cdaudio
18k
23
256
7.8
parport
61k
74
753
8.1
parclss
138k
77
382
7.2
Future – BLAST Framework
Using interpolation in predicate discovery
was a step forward. This reduced the
number of abstract predicates to track.
Lazy Interpolation-Based model checking
integrates interpolants further into the engine.
It is being explored by BLAST Kenneth
McMillan and BLASTer Ranjit Jhala.
Expression abstraction may yet be able to
represent more expressive C patterns. Basic
user-mode program specifications cannot be
verified.
The Bad – BLAST Usability
Not easy to install (Specific path
dependencies, purported Cygwin
compatibility a fallacy)
Not easy run


BLAST Query Language pre-processor yields
bad output, contains no usage information
BLAST main executable has 232 lines of
usage information, some of which is
completely broken, some of which has
significant impact on results.
The Bad – BLAST Usability
Output with debugging suppressed contains
about .5% useful information. With
debugging added, that number is significantly
less. (Even a developer would be hard
pressed to decipher over one-hundred
consecutive lines of “: is the zero DD”).
The tool’s specification weaver does not
correctly remember source code lines, so
useful output requires extensive
corresponding grep into the source.
BLAST – Recommendations
Integrate BLAST into development
process early.
Take advantage of BLAST’s typically fast
run times to perform verification as a postcompile operation.
Take note when BLAST output changes
dramatically, and find the corresponding
change in source.
BLAST – Recommendations
BQL (BLAST Query Language) is a good
way to separate specification from
implementation and eliminate BLAST
checks that could turn into useless runtime
costs
Build tools around BQL specification
weaver or improve it (ahh, the gift and the
curse of open source).
Questions?
References
The information shown here is based by
corresponding paper on the BLAST tool,
“Software Verification with BLAST”. A
good list of references can be found there.