Transcript Analysis is necessary – but far from sufficient Jon Pincus Reliability Group (PPRC) Microsoft Research.

Analysis is necessary –
but far from sufficient
Jon Pincus
Reliability Group (PPRC)
Microsoft Research
Why are so few successful real-world
development and testing tools
influenced by program analysis
research?
2
Jon Pincus (Microsoft Research)
Outline
•
•
•
•
•
Provocation
Successful tools
Analysis – in context
Implications for analysis
Conclusion
3
Jon Pincus (Microsoft Research)
Success: a simple view
• A tool is successful if people use it
• Not if people think it’s interesting – but don’t try it
• Not if people try it but don’t use it
• Not if people buy it but don’t use it (“Shelfware”)
4
Jon Pincus (Microsoft Research)
Some examples of success
• Purify
• BoundsChecker
• PREfix (2.X and later)
• Especially interesting because 1.0 was unsuccessful
5
Jon Pincus (Microsoft Research)
Why do people use a tool? If
• it helps them get their work done …
• … more efficiently than they would otherwise
• … without making them look (or feel) bad.
Aside: look at organizational and personal goals.
See Alan Cooper’s books, e.g. About Face
6
Jon Pincus (Microsoft Research)
Value vs. Cost
• Value: the quantified benefit from the tool
• Cost: primarily time investment
• Licensing cost is typically much smaller
• (Value – Cost) must be
• Positive
• Positive fairly quickly
• More positive than any alternatives
• Value and cost are difficult to estimate …
… and others’ estimates are often questionable
7
Jon Pincus (Microsoft Research)
An example
• Purify 1.0:
• Virtually zero initial cost on most code bases
• “trial” license
• easy to integrate
• Immediate value
• Companies then invested to increase the value
• E.g., changing memory allocators to better match
Purify’s
• (and buying lots of licenses)
8
Jon Pincus (Microsoft Research)
Characteristics of successful tools
• Successful tools almost always
•
•
•
•
address significant problems,
on real code bases,
give something for (almost) nothing,
and are easy to use.
9
Jon Pincus (Microsoft Research)
Significant problems
• Nobody fixes all the bugs.
• What are the key ones?
• Often based on most recent scars
• Often based on development or business goals
• Examples:
• Purify: memory leaks
• BoundsChecker: bounds violations
• Lint (back in K&R days): portability issues
10
Jon Pincus (Microsoft Research)
Real code bases
• Large code bases in nasty languages (e.g., C/C++)
• 1M+ LOC is medium-sized; 10M+ LOC is large
• Or, smaller code bases in different nasty languages
• Perl, JScript, VBScript, HTML/DHTML, TCL/Tk, SQL
• 5000+ LOC is medium; 50K+ is large
11
Jon Pincus (Microsoft Research)
More reality …
• Most code bases involve multiple languages
• Extensions and incompatibilities, e.g.
• GCC/G++, MS C++, Sun C++
• ECMAScript/JScript/JavaScript
• HTML versions
• People use all those nasty language features
(e.g., casts between pointers and ints, unions, bit
fields, gotos, …)
12
Jon Pincus (Microsoft Research)
Something for (almost) nothing
• Engineering time is precious
• Engineers are skeptical
… so are unwilling to commit their valuable time
• Don’t even think about requiring significant
up-front investment
• code modifications
• process changes
13
Jon Pincus (Microsoft Research)
Examples: something for
(almost) nothing
• Purify for UNIX: just relink!
• BoundsChecker: you don’t even need to relink!!
• PREfix 2.X: point your web browser to a URL!!!
• A non-technology solution: “we’ll do it for you”
• Commercial variant: an initial benchmark for $X
• Preferably: money back if it isn’t useful
• In many cases, money is cheaper than engineering time …
14
Jon Pincus (Microsoft Research)
“Revolutionary tools”
• People may be willing to do up-front work to
• Enable something previously impossible
• Or provide order-of-magnitude improvements
• BUT!
• Still must be significant problem, real code base
• Need compelling evidence of chance for success
• Any examples?
15
Jon Pincus (Microsoft Research)
Outline
•
•
•
•
•
What makes a tool successful?
Successful tools
Analysis – in context
Implications for analysis
Conclusion
16
Jon Pincus (Microsoft Research)
PREfix
• Analyzes C/C++ source code
• Identifies defects
• GUI to aid understanding and prioritization
• Viewing individual defects
• Sorting/filtering sets of defects
• Integrates smoothly into existing builds
• Stores results in database
17
Jon Pincus (Microsoft Research)
PREfix 2.X ArchitectureWeb Browser
#include <std.h>
int PwrOf2(int a)
{
if (a & (a - 1))
return 0;
else
return 1;
}
C/C++
Parser
Source Code
Simulator
Execution Control
(mod “PwrOf2”
(c “a” init)
(t t1 (& a (-a 1)))
(g t1<0:4> !0
(r 0 success))
(g t1<0:4> 0)
(r 1 success)))
Auto
Modeler
Error
Analysis
Virtual Machine
Defect Database
Model Database
Jon Pincus (Microsoft Research)
Counterintuitively …
Actual analysis is only a small part of any
“program analysis tool”.
In PREfix, < 10% of the “code mass”
19
Jon Pincus (Microsoft Research)
3 key non-analysis issues
• Parsing
• Integration
• Build process
• Defect tracking system
• SCM system
• User interaction
• Information presentation
• Navigation
• Control
20
Jon Pincus (Microsoft Research)
Parsing
• You can’t parse better than anybody else …
… but you can parse worse
• Complexities:
• Incompatibilities and extensions
• Full language complexity
• Language evolution
• Solution: don’t
• Alternatives: GCC, EDG, …
21
Jon Pincus (Microsoft Research)
Integration
• A tool is useless if people can’t use it
• Implied: “use it in their existing environment”
• “Environment” includes
•
•
•
•
Configuration management (SCM)
A build process (makefiles, scripts, …)
Policies
A defect tracking system
• People have invested hugely in their environment
• They probably won’t change it just for one tool
22
Jon Pincus (Microsoft Research)
User interaction
• Engineers must be able to
• Use the analysis results
• Understanding individual defects
• Prioritizing, sorting, and filtering sets of defects
• Interact with other engineers
• Influence the analysis
• Current tools are at best “okay” here
• Improvement is highly leveraged
23
Jon Pincus (Microsoft Research)
Example: Noise
• Noise = “messages people don’t care about”
• Noise can result from
•
•
•
•
•
Incorrect tool requirements
Integration issues
Usability issues (e.g., unclear messages)
Analysis inaccuracies
…
24
Jon Pincus (Microsoft Research)
Dealing with noise
• Improving analysis is usually not sufficient
• May be vital; may not be required
• Successful user interaction techniques:
•
•
•
•
•
Filtering
History
Prioritization
Improving presentation, navigation
Providing more detail
25
Jon Pincus (Microsoft Research)
Outline
•
•
•
•
•
What makes a tool successful?
Characteristics of successful tools
Analysis – in context
Implications for analysis
Conclusion
26
Jon Pincus (Microsoft Research)
Characteristics of useful analyses
• Scalable to “large enough” system
• Typically implies incomplete, unsound, decomposable,
and/or very simple
• “Accurate enough” for the task at hand
• Produce information usable by typical engineer
• E.g., if there’s a defect, where? How? Why?
• Remember: half the engineers are below average
• Handle full language complexity
• (or degrades gracefully for unhandled constructs)
• Handle partial programs
27
Jon Pincus (Microsoft Research)
Analyses are not useful if …
• They don’t apply to the tool’s “reality”
• “For a subset of C, excluding pointers and structs …”
• “We have tested on our approach on programs up to
several thousand lines of Scheme …”
• They assume up-front work for the end user
• “Once the programmer modifies the code to include calls
to the appropriate functions …”
• “The programmer simply inserts the annotations to be
checked as conventional comments …”
28
Jon Pincus (Microsoft Research)
Different tradeoffs from compilers
• Focus on information, not just results
• Compilers don’t have to explain what they did
and why
• Unsoundness is death for optimization – but
may be okay for other purposes
• Intra-procedural analysis often not enough
29
Jon Pincus (Microsoft Research)
Types of analyses
•
•
•
•
•
FCIA: Flow- and context-insensitive
FSA: Flow-sensitive
CSA: Context-sensitive
FCSA: Flow and context sensitive
PSA: Path-sensitive
30
Jon Pincus (Microsoft Research)
Performance vs. “Accuracy”
Speed
Accuracy
FCIA
FSA
CSA FCSA
PSA
Don’t forget “information”!
Speed
Accuracy
Information
FCIA FSA
CSA FCSA PSA
Example analysis tradeoffs
• PREfix: scalable, usable analysis results
•
•
•
•
Path-sensitive
Incomplete (limit # of paths traversed)
Unsound (many approximations)
Major emphasis on summarization (“models”)
• PREfast: fast, usable analysis results
• Local analyses, using PREfix models
• Flow-insensitive and flow-sensitive analyses
• Far less complete than PREfix
33
Jon Pincus (Microsoft Research)
Aside: Techniques for scalability
• Decompose the problem
• Use the existing structure (function, class, etc.)
• Summarization, memoization
• Caveat: make sure you don’t lose key info!
• Give up completeness and soundness
• Use three-valued logic with “don’t know” state
• Track approximations to limit the damage
• Examine and re-examine tradeoffs!!!!
• Optimize for significant special cases
34
Jon Pincus (Microsoft Research)
Outline
•
•
•
•
•
What makes a tool successful?
Characteristics of successful tools
Analysis – in context
Implications for analysis
Conclusion
35
Jon Pincus (Microsoft Research)
Recap: successful tools
• People use tools to accomplish their tasks
• Successful tools must
•
•
•
•
address real problems,
on real code bases,
give something for (almost) nothing,
and be easy to use
• Analysis is only one piece of a tool
• Information is useless if it’s not presented well
36
Jon Pincus (Microsoft Research)
One person’s opinion
Why are so few successful real-world
development and testing tools
influenced by program analysis
research?
Several key areas are outside the traditional scope of
program analysis research
User interaction
Visualization (of programs and analysis results)
Integration
37
Jon Pincus (Microsoft Research)
One person’s opinion (cont.)
Why are there so few successful real-world
programming and testing tools based on academic
research?
Program analysis research in general:
Not directly focused on “key problems”
Not applicable to “real world” code bases
Makes unrealistic assumptions about up-front work
38
Jon Pincus (Microsoft Research)
One tool developer’s mindset
• We have plenty of ideas already.
• We can’t even implement all our pet projects!
• We are interested in new ideas – but skeptical
• The burden is on you to show relevance
• Remember, analysis is only part of our problem
• If we can’t figure out how to present it – forget it
39
Jon Pincus (Microsoft Research)
Making analysis influential
• Show how the analysis addresses a
significant problem
• Synchronization, security, …
• Convince us that it will work in our reality
• Avoid the obvious problems discussed above
• Demonstrate in our reality …
(perhaps by using real-world code bases)
• … or persuade us that it will work
40
Jon Pincus (Microsoft Research)
Some interesting questions …
• Which analyses are right for which problems?
• How to get difficult analyses to scale well?
• Are there soundness/completeness tradeoffs?
• Are there opportunities to combine analyses?
• Can we use a cheap flow-insensitive algorithm to focus a
more expensive algorithm on juicy places?
• Can we use expensive local path-sensitive algorithms to
improve global flow-insensitive algorithms?
41
Jon Pincus (Microsoft Research)
Beyond analysis
• Can visualization and user interaction for
analysis tools become an interesting
research area?
• How can analysis be used to refine
visualization and user interaction?
42
Jon Pincus (Microsoft Research)
Questions?
43
Jon Pincus (Microsoft Research)
Analysis is necessary –
but far from sufficient
Jon Pincus
Reliability Group (PPRC)
Microsoft Research