What mutation-testing frameworks exist for C/C++?

Question

Mutation testing has been out there for a while now, and it seems there are at least one or two commercial mutation testing frameworks for C/C++. Have you used them? What are your experiences? Are there any open source alternatives?

Answer 1

A brief search resulted in:

• PlexTest: http://www.itregister.com.au/products/plextest_detail.htm
• Insure++: http://www.parasoft.com/jsp/products/insure.jsp;jsessionid=baacpvbaDywLID?itemId=63
• MILU (may be only for C): http://www.dcs.kcl.ac.uk/pg/jiayue/milu/

With that said, you need to realize that mutation testing isn't particularly useful (at least from some stuff I've previously read). It's an interesting tool when faced with hard (metaphorically speaking) asserts and for making sure that data requirements are heeded to (when dealing with if and only if situations).

In my opinion, there are much more established ways of analyzing the robustness of code.

Answer 2

Notice that Parasoft's tool only generate equivalent mutations. That echoes the problem described on Wikipedia article about Mutation Testing - it is hard to distinguish between equivalent and non-equivalent mutations so they decided to stick with equivalent.

I tried another interesting tool that can automatically discover invariants in instrumented C and C++ code - it is called "Daikon". Essentially it is doing same thing as tool that generates equivalent mutations, but instead of identifying problematic code it gives you a set of invariants such as "A == B + 1". I think invariants are more useful because when you look at discovered invariant it gives you assurance that your code is correct if invariant make sense, and then you can convert invariants into asserts and that gives you more confidence when you change code.

Answer 3

A straight forward python script for mutating c programs is available at:

https://github.com/parunbabu/mutate.py

the author says it works better if the code under test is de-commented and indented.

and it is also free and opensource ... i think this is what you are looking for.

Answer 4

I am currently using:

• Dextool Mutate: https://github.com/joakim-brannstrom/dextool/tree/master/plugin/mutate

The tool helped me expose some test cases that did not kill any mutant. In one of them, the assertion had been commented out.

I also discovered some test cases that kill the same mutants. This was especially common when using parameterized testing.

Answer 5

The existing frameworks where way too time-consuming to set up and use so I did my own implementation, a quick and easy solution that should work on any machine. There is binaries available for MacOSX, Windows and RaspberryPi (Linux):

https://github.com/RagnarDa/dumbmutate

Hope it helps anyone!