What are the differences between the three code coverage analysis methods?

Question

This sonar page mainly lists the various methods used by various code analysis tools:

• Source Code Toolkit (used by Clover )
• Standalone Byte Code Toolkit (Used by Cobertura )
• Byte Code Toolkit on the fly (used by Jacoco )

What are these three methods, and which one is most effective and why? If the answer to the question about effectiveness is “dependent”, then explain why?

Answer 1

The source code toolkit consists of adding instructions to the source code before compiling it. These instructions are used to track which parts of the codes have been executed.

Offline bytecode consists of adding the same instructions, but after compilation directly into bytecode.

The on-the-fly byte code toolkit is to add the same instructions to the bytecode, but dynamically, at runtime, when the bytecode is loaded by the JVM.

This page contains a comparison between the methods. It may be biased as it is part of the Clover documentation.

Depending on your definition of effective, choose the one you like best. I do not think that you will get huge differences. All of them do this work, and the big picture will be the same as in the method used.

Answer 2

In general, the effect on the coating is the same.

Source code tools can give excellent reporting results , simply because byte code tools cannot distinguish any structure in the source lines, because the granularity of the code block is written only in terms of the source lines.

Suppose I have two nested if statements (or equivalently , if (a & b) ... * ) on the same line. The source code analyzer tool can see them and provide coverage information for several arms within if, within the source line; it can report blocks based on rows and columns. The bytecode tool sees only one line wrapped around conditions. Does the line in the line mean “covered” if condition a is satisfied, but is false?

You can argue that this is a rare circumstance (and it probably is), and therefore not very useful. When you get fictitious coverage and then the field fails, you can change your mind about the utility.

There is a good example and explanation of how byte code coverage makes getting information about switch statements correct , extremely difficult.

The source code analyzer tool can also speed up the execution of tests because it has a compiler that helps optimize the tool code. In particular, a probe inserted inside the loop by a binary can be compiled inside the loop by the JIT compiler. A good Java compiler will see that the hardware produces a loop-dependent result and pulls the measuring hardware out of the loop. (The JIT compiler can do this too, the question is whether they really do this).