Testing for non-deterministic behavior of python function

Question

We've got a large and complex function which needs to be deterministic. It is one of the workhorses at our company and covers a large amount of code. This code often becomes non-deterministic due to python's dict iterator. This has happened many times, and it is very difficult to track down, and often not noticed immediately. We would like to write an automated test to detect non-determinism, but I'm not sure how to do it.

We have tried running the function in a loop, and testing the results are always the same, but sometimes, even though the function is non-deterministic, the function will pass this test due to the arbitrary but somewhat consistent ordering of the dict iterator.

Is there a way to write an automated test to catch this kind of bug?

Perhaps there is a way to hack python's dict so that the iterators are random rather than arbitrary during this test? That way repeated calls to function would be more likely to diverge? This seems like a fairly involved method, but I can't think of any other way.

EDIT:

We are currently using Python 2.7.

We have unit tests of the various sub-modules, however they often don't expose the non-determinism due the arbitrary, but consistent nature of the dict order.

Also, perhaps non-deterministic is not the right way to describe this problem. This function takes {id : data}, however the value of the ids should not affect the result of the code, however due to python dict ordering, it sometimes does. Perhaps the best way to test this would be to replace the ids with random values and check to see if the output is the same after many runs with different ids.

Answer 1

If you want to randomize the hash seed, you can specify the -R flag to python:

-R     : use a pseudo-random salt to make hash() values of various types be
         unpredictable between separate invocations of the interpreter, as
         a defense against denial-of-service attacks

a la

~$ python -c "print {y:x for x,y in enumerate('foobar')}"
{'a': 4, 'r': 5, 'b': 3, 'o': 2, 'f': 0} #it will always be this
~$ python -R -c "print {y:x for x,y in enumerate('foobar')}"
{'a': 4, 'b': 3, 'r': 5, 'f': 0, 'o': 2}
~$ python -R -c "print {y:x for x,y in enumerate('foobar')}"
{'a': 4, 'b': 3, 'r': 5, 'o': 2, 'f': 0}
~$ python -R -c "print {y:x for x,y in enumerate('foobar')}"
{'f': 0, 'o': 2, 'b': 3, 'r': 5, 'a': 4}
~$ python -R -c "print {y:x for x,y in enumerate('foobar')}"
{'r': 5, 'f': 0, 'o': 2, 'a': 4, 'b': 3}

Note that this behavior is the default in python 3.3.

Answer 2

You can use OrderedDict to force two similar dicts to have a different "order".

Using those as inputs to your code instead of vanilla dicts, you can reliably check the code behavior with regards to dict order issues.

For example, this test fails from times to times (relatively rarely):

d1 = {'a':1, 'b': 2}
d2 = dict(d1)

j1 = json.dumps(d1)
j2 = json.dumps(d2)

assert j1 == j2:

and this test fails predictably:

import json
from collections import OrderedDict

d1 = OrderedDict([('a', 1), ('b', 2)])
d2 = OrderedDict([('b', 2), ('a', 1)])

j1 = json.dumps(d1)
j2 = json.dumps(d2)
assert j1 == j2

However, this might be more suited for unit-testing small functions. If you're testing a "large and complex function" all at once, it is likely that the dicts are generated inside the function so acting on the inputs won't be enough.