How to create a corrupt pkl file python

Question

I'm creating a library that caches function return values to pkl files. However, sometimes when I terminate the program while writing to the pkl files, I wind up with corrupt pkl files (not always). I'm setting up the library to deal with these corrupt files (that lead mostly to an EOFError, but may also lead to an IOError). However, I need to create files that I know are corrupt to test this, and the method of terminating the program is not consistent. Is there some other way to write to a pkl file and be guaranteed an EOFError or IOError when I subsequently read from it?

Answer 1

Short answer: You don't need them.

Long answer: There's a better way to handle this, take a look below.

Ok, let's start by understanding each of these exception separately:

1. The EOFError happens whenever the parser reaches the end of file without a complete representation of an object and, therefore, is unable to rebuild the object.

2. An IOError represents a reading error, the file could be deleted or have it's permissions revoked during the process.

Now, let's develop a strategy for testing it.

One common idiom is to encapsulate the offending method, pickle.Pickler for example, with a method that may randomly throw these exceptions. Here is an example:

import pickle
from random import random

def chaos_pickle(obj, file, io_error_chance=0, eof_error_chance=0):
    if random < io_error_chance:
        raise IOError("Chaotic IOError")

    if random < eof_error_chance:
        raise EOFError("Chaotic EOFError")

    return pickle.Pickler(obj, file)

Using this instead of the traditional pickle.Pickler ensures that your code randomly throws both of the exceptions (notice that there's a caveat, though, if you set io_error_chance to 1, it will never raise a EOFError.

This trick is quite useful when used along the mock library (unittest.mock) to create faulty objects for testing purposes.

Enjoy!

Answer 2

Take a bunch of your old, corrupted pickles and use those. If you don't have any, take a bunch of working pickles, truncate them quasi-randomly, and see which ones give errors when you try to load them. Alternatively, if the "corrupt" files don't need to even resemble valid pickles, you could just unpickle random crap you wouldn't expect to work. For example, mash the keyboard and try to unpickle the result.

Note that the docs say

The pickle module is not intended to be secure against erroneous or maliciously constructed data. Never unpickle data received from an untrusted or unauthenticated source.