How important is the data to assess code coverage?

Question

per the title how important is the data in assessing code coverage?

As background, let's say that you're given 20% of the entire dataset to help trace the different pathways each row of data goes through the code. My understanding of this problem is that the limitation of the data can mean that not all possible pathways have been reached.

Now, the big question is does one need the other 80% of the data to assess code coverage, or do you just need the one or two extra lines to hit the remaining pathways? To put it differently, if I hit all of the remaining pathways, is that sufficient to assess code coverage?

As an example, let's say you have the following:

> df.columns
[numerator, denominator]
> df.head(3).values
array([2, 4, 0],
      [4, 0, 0])


# Code
for idx, row in enumerate(df):
    try:
        if row[0] / row[1] == 0.5:
            return row[0]/row[1]
    except DivideByZeroException as e:
        return 'DivideByZeroException'

So in this case, there's only two possible outcomes: 0.5, DivideByZeroException

As such, would we require additional values from the dataset, such as numerator:1, denominator:2 if its corresponding pathway has already been identified with numerator:2, denominator:4.

My background is in statistics and data science, so my apologies if my interpretation might have come off as niave.

Answer 1

The actual data that gets used when testing is largely irrelevant and very often made-up bogus values.

What is important is that the data set contains all the characteristics to trigger the relevant code paths. Real production data will often have those characteristics multiple times, but that is not needed for triggering the relevant code paths.

So, it doesn't matter if you get 20%, 80% or 100% of the production data, as long as the dataset you get has all the needed characteristics.

Answer 2

Your tests are only as good as the data you feed in to them, and if you feed incomplete or biased data into them, they'll give you incomplete or biased results. For example, if you wanted to test a method which divided two numbers, but only wrote one test which operated on the numbers 1 and 2, you'd never discover a potential exception when dividing by 0.

That said, it's usually impossible to test every possible scenario. In our division example, there are an infinite number of numbers, so you can't test your method against all of them. You have to use your knowledge of the domain to inform the tests that you write. In our division example, you know from how division works that something bad happens when you try to divide by 0 - so you should write a test for that scenario. Also from your knowledge, you know that the numerator doesn't matter when dividing by 0 (except maybe if it's 0), so you can pick a random number or a specific number. You then might wind up with a few test cases:

numerator: 2, denominator: 4, expected: 0.5
numerator: 4, denominator: 0, expected: DivideByZeroException
numerator: 0, denominator: 0, expected: DivideBYZeroException
numerator: 0, denominator: 2.7, expected: 0

These likely satisfy all possible routes the method can take. You can verify that with code coverage tools, and add additional test cases where appropriate.

You may also wish to test some of the mathematical properties of division. So you would write additional tests, even though these probably don't traverse any new pathways in the code:

numerator: 4, denominator: 1, expected: 1
numerator: positive, denominator: positive, expected: positive
numerator: positive, denominator: negative, expected: negative
etc

In this simple example, you probably don't need to generate a dataset of any kind, as there are few enough tests to hard code them as test cases in your unit test framework. But, nothing stops you from putting these all into a table of some kind and writing a single test that checks the whole table, row by row.