How to guarantee FsCheck reproducibility

Question

We want to use FsCheck as part of our unit testing in continuous integration. As such deterministic and reproducible behaviour is very important for us.

FsCheck, being a random testing framework, can generate test cases that potentially sometimes break. The key is, we do not only use properties that would have to hold for necessarily every input, like say List.rev >> List.rev === id. But rather, we do some numerics and some test cases can cause the test to break because of being badly conditioned.

The question is: how can we guarantee, that once the test succeeds it will always succeed?

So far I see the following options:

• hard code the seed, e.g. 0. This would be the easiest solution.
• make very specific custom generators which avoid bad examples. Certainly possible, but could turn out pretty hard, especially if there are many objects to generate.
• live with it, that in some cases the build might be red due to pathological cases and simply re-run.

What is the idiomatic way of using FsCheck in such a setting?

Answer 1

(I started writing a comment but it got so long I guess it deserved its own answer).

It's very common to test properties with FsCheck that don't hold for every input. For example, FsCheck will trivially refute your List.rev example if you run it for list<float>.

Numerical stability is a tricky problem in itself - there isn't any non-determinism in FsCheck to blame here(FsCheck is totally deterministic, it's just an input generator...). The "non-determinism" you're referring to may be things like bugs in floating point operations in certain processors and so on. But even in that case, wouldn't you like to know about them? And if you algorithm is numerically unstable for a class of inputs, wouldn't you like to know about it? If you don't it seems to me like you're setting yourself up for some real non-determinism...in production.

The idiomatic way to write properties that don't hold for all inputs of a given type in FsCheck is to write a generator & shrinker. You can use ==> as a step up to that, but it doesn't scale up well to complex preconditions. You say this could turn out pretty hard - that's true in the sense that I guarantee you'll learn something about your code. A good thing!

Fixing the seed is a bad idea, except for reproducing a previously discovered bug. I mean, in practice what would you do: keep re-running the test until it passes, then fix the seed and declare "job done"?

Answer 2

some test cases can cause the test to break because of being badly conditioned.

That sounds like you need a Conditional Property:

let isOk x =
    match x with
    | 42 -> false
    | _ -> true

let MyProperty (x:int) = isOk x ==> // check x here...

(assuming that you don't like the number 42.)