Transcript Bug Driven Bug Finding

Bug Driven Bug Finding
Chadd Williams
University of Maryland
Motivation


Finding bugs in software is important
Statically checking code has been effective
– finds complex errors
– no need to run the code

Many static checkers available
– some with specific bug patterns to find
– some allow the user to define the patterns
– what kinds of bugs are really out there?

Lots of false positive error reports
– can we rank the errors better?
– can previous bug history help?

Where to start?
– Bug reporting databases
– CVS commit messages
2
University of Maryland
Bug Database

Inspect fixed bugs
–
–
–
–
review bug discussions
tie fixed bug to source code change
classify the type of the bug
look for bugs that can be found statically
Users
Developers
Bug Database
3
University of Maryland
Bug Database: Practical Experience

We inspected the Apache httpd bug
database
– inspected 200 bug reports marked as fixed
– not as helpful as we expected

Only 24% tied directly back to a source
code change
– bug reports include a discussion of the problem
– rarely is a diff or a CVS revision noted

Most are logic errors/feature requests
– not the type found by static checkers
4
University of Maryland
Bug Database Bug Types
NULL pointer check
Return Value check
Logic Errors/Feature Request
Uninitialized Variable Errors
Error Branch Confusion
External Bugs (OS or other
software failed)
System specific pattern

5
Most classified bugs are logic errors
University of Maryland
Bug Database: Practical Experience

Most bug reports originate from users
– 197 out of 200
– does not capture bugs found by developers

Most bug reports came against a release
of the software, not a CVS-HEAD
– 198 out of 200
– does not capture bugs between releases

What about the bugs that don’t make it
into the release?
– they may be in the CVS repository…
6
University of Maryland
CVS Repository

Commits may contain useful data
– any bug fix must show up in a commit
– will commit messages lead us to bug fixes?


Shows bugs fixed between releases
Bugs caught by developers
– bugs that could be found by static checking
1.1
1.2
CVS
Repository
7
University of Maryland
1.3
CVS Repository: Practical Experience

Inspected commit messages
– looked for ‘fix’, ‘bug’ or ‘crash’
– ignored those with bug number listed
– looked at mature source files

Commit messages are useful
– trivially tied to source code change
– less logic errors

Common errors found
– NULL pointer check
– failing to check the return value of a function
before use
8
University of Maryland
CVS Repository Bug Types
NULL pointer check
Return Value Check
Feature Request
Uninitialized Variable Errors
Failure to set value of pointer
parameter
Error caused by if conditional
short circuiting
Loop iterator increment error
System specific pattern

9
NULL pointer bugs and return value
bugs can be found by static analysis
University of Maryland
Return Value Check Bug

Returning error code and valid data
from a function is a common C idiom
– the return value should be
checked before being used
– lint checks for this error

10
Error types
– completely ignored
• foo();
– return value used directly as
an argument
• bar(foo());
– others …
University of Maryland
int foo(){
…
if( error ){
return error_code;
}
….
return data;
}
…
value = foo();
newPosition + = value; // ???
Return Value Checker

Some functions don’t need their return
value checked
– no error value returned
– could lead to many false positives

Naively flagging all unchecked return
values leads to many false positives
– over 7,000 errors reported for the Apache
httpd-2.0 source

Need to determine which are most likely
true errors
– use historical data
– present this data to the user
11
University of Maryland
Which return values need checked?

Infer from historical data
– look for an add of a check of a return value in a
CVS commit
– implies the programmer thinks it’s important
…
value = foo();
newPosition + = value; // ???
…

Commit
Bug Fix
…
value = foo();
if( value != Error) { // Check
newPosition + = value;
}
…
Infer from current usage
– does the return value of a function get checked
in the current version of the software
– how often?
12
University of Maryland
Our Tool

Static checker that looks for return
value check bugs
– built on ROSE by Dan Quinlan, et al.

Classify each error by category
– ignored return value
– return value used as argument, etc.

Produce a ranking of the errors
– group errors by called function
– rank most promising errors higher
• rank functions that most likely need their
return value checked higher
13
University of Maryland
Return Value Checker: Ranking


Rank errors in two ways
Split functions into two groups
– functions flagged with a CVS bug fix commit
• at least one CVS commit adds a check of the
function’s return value
– functions not flagged with CVS bug fix commit

Within each group:
– rank by how often the function’s return value is
checked in the current software distribution
– checked more often means rank higher
14
University of Maryland
Case Study

Apache httpd-2.0 on Linux
– core system
– modules
– Apache Runtime Library

Checked all the CVS commits for a
return value check bug fix
– 6100 commits checked
– 2600 commits failed to go through our tool
• wrong (too new) version of autoconf
• parser problems
• compile bugs in the CVS commits
15
University of Maryland
Case Study: Results

Our checker marked over 7,000 errors
– individual call site for non-void function where
the return value is not checked

Too many too look at!
– expect many are false positives

Rank errors
value = foo(); // ERROR
newPosition + = value;
…
result = foo(); // ERROR
zoo(result);
– inspect CVS bug fix commit flagged functions
– inspect functions with return value checked
more than 50% of the time in the current
source tree
16
University of Maryland
Case Study: Error Breakdown

Inspected 453 errors (of 7,000)
– found 98 that may be bugs!

231 errors associated with a CVS bug
fix flagged function
– 61 of the 98 bugs found here
– false positive rate of 74%

222 errors associated with a function
that has its return value checked > 50%
of the time
– 37 of the 98 bugs found here
– false positive rate of 83%
17
University of Maryland
Case Study: A Bug

We investigated an error and found it
did crash httpd
– error reported near the top of the ranking

The called function builds a filename
– arguments represent file and pathname
– a char array is returned and directly used as
an argument to strcmp()
– strcmp(foo())
– NULL return value will cause a seg fault
– return value is NULL if the path is too long!
18
University of Maryland
Analysis

False positive rate too high!
– overall false positive rate: 78% (1-(98/453))

A false positive rate closer to 50%
would be acceptable
– the user is likely as not to find a true error
– cluster them near the top of the ranking

We did cull 7,000 errors down to 453
– lint would have flagged only the ‘ignored’ errors
and not ranked them
19
University of Maryland
Conclusion

Bug databases are not useful in
understanding much about low-level bugs
– good for logic errors
– good for misunderstood specifications

CVS commit messages give a better
picture of low-level bugs
– especially bugs that don’t enter a release

20
CVS commits can give useful data to
help classify error reports
University of Maryland
Future Work




What other types of bugs are common?
What other checkers can benefit from
CVS data?
How can we cut the false positive rate?
Can we dynamically gather data on
functions called via function pointers?
– many of the error messages involved calls
through function pointers
– Dyninst will allow us to instrument function
pointer call sites and gather data
21
University of Maryland