Test suite for automatic bug finding

StackOverflow https://stackoverflow.com/questions/11312924

Question

For many kinds of hard computational problems e.g. propositional satisfiability, theorem proving, traveling salesman etc., there exists an extensive test suite that can be used for evaluating the performance of a program that attempts to solve the problem.

Does any such thing exist for automatic bug finding, i.e. a collection of programs or code fragments containing known bugs of a kind that could be automatically detected? I assume e.g. Coverity must have such a thing for internal use, but a Google search doesn't seem to show anything publicly available.

Was it helpful?

Solution

I don't think such a suite exists. The most glaring question for a "standard" suite is, what language would it be written in?

For C, you can look at SAFECode's test suite, here. SAFECode is meant to detect these bugs at runtime, but a "perfect" static analysis tool should probably be able to detect them statically.

A more comprehensive, and more importantly, a static analysis tool, is Frama-C. See the paper entitled "Test-driving static analysis tools in search of C code vulnerabilities". The paper describes a need for the very test suite that you ask for. It also links to a test suite and the effectiveness among common analyzers. Frama-C might have its own benchmarks, you can download the source and check.

See also:

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top