What is the advantage of instruction (byte code) coverage over line and branch coverage?

Question

Our project uses Jacoco for code coverage metrics. Previously, we have checked the line and branch coverage against percentage values. However, the documentation also describes an "Instructions" level, which counts Java byte code instructions. From their documentation:

The smallest unit JaCoCo counts are single Java byte code instructions. Instruction coverage provides information about the amount of code that has been executed or missed. This metric is completely independent from source formatting and always available, even in absence of debug information in the class files.

If we have been using line and branch coverage previously, what is the advantage of switching to instruction coverage? Is it just counting the same things in a different way? Or does it provide more accurate metrics? Conversely, is there a situation where we would want to use line and branch coverage metrics instead of byte code?

Answer 1

There is no advantage of byte-code or line coverage over branch coverage.

If one element (line or instruction) of a branch is covered, then all elements of that branch are covered as well.

If a single line of code contains more than one branch (e.g., in C: (a==0) ? do_b() : do_c()), branch coverage takes it into account.

Moreover, branch coverage is completely independent from source code formatting.

Answer 2

The advantage the docs mention is that line coverage counts

x = 1; y = 2; z = 4;

as one line, but

x = 1;
y = 2;
z = 4;

as three. On the plus side, bytecode-based measures would not vary between the two, because the two versions compile to the same bytecode sequence. Line-based metrics would report different numbers.

Bytecode coverage as a reporting metric seems gimmicky. Having a metric independent of code formatting is certainly a virtue in theory. It might be a plausible defense against developers gaming coverage numbers, say by putting many logical statements on one line. But it depends on language implementation details, not the source code. It wouldn't provide stable numbers across different implementations of Java, some of which don't even have bytecodes, and would even vary over time as the JVM and compilers evolve. Most if not all testing engines already discount the most obvious formatting deltas, such as comment and blank lines. Code reviews and coding style standards also already naturally discourage gratuitous line-packing and other coverage metric-gaming.

Bytecode coverage is not a standard or widely-accepted metric / unit of reporting. Unlike branch coverage, counting bytecodes is not a "subtle, but meaningful" semantic distinction in the coverage being measured. It is just an alternate, less-formatting-sensitive way to measure / report the same thing line coverage already measures. It has some slight advantages, but they don't seem especially large when balanced against them being a nonstandard measure, tied to a specific (if popular) language implementation, and not directly correlated to visible program source.

Answer 3

Java byte code interpreters are considered an implementation detail. You shouldn't have to test or assess metrics at the byte code level, since you'd essentially be testing for bugs in the byte code interpreter (which you really shouldn't have to worry about). I read the paragraph you linked as "if all you have is the byte code binary, you can still get metrics this way."