Why doesn't pickle.dump write the file?

Question

I currently have a project that runs for several days. As errors might occur in some steps of execution, I pickle critical steps to be able to restart the script at the correct step (so I don't have to do the work of eventually over 24h of execution again).

One thing I store with pickle is a list steps. This list contains every step that successfully finished. It is used to skip steps when I start the script again.

The problem is that pickle seems not to update this after I switch the module.

Code

mainscript.py

import subscript

def set_status(mysql_script_instance_id, status, state=None):
    # [...]
    # update status in database (works as expected)
    # [...]

        if state is not None:
            with open("state.pickle", "wb") as f:
                pickle.dump(state, f)
            logging.debug("Dumped pickle. steps_done: %s" % state['steps_done'])
    logging.info(status)

subscript.py

import mainscript
[...]
logging.info("%s finished." % (step.__name__))
self.state['steps_done'].append(step.__name__)
[...]
logging.debug("self.state['steps_done'] = %s" % self.state['steps_done'])
mainscript.set_status(self.mysql_script_instance_id, "step xy done", self.state)

pickleviewer

#!/usr/bin/env python
import pickle
import pprint
state = pickle.load(open("state.pickle", "rb"))
pp = pprint.PrettyPrinter(indent=4)
pp.pprint(state)

What I've tried

I get all log messages I've expected:

2014-04-03 08:23:07,727 INFO: step1 finished.
2014-04-03 08:23:07,728 DEBUG: self.state['steps_done'] = ['Fetch recordings', 'preparation', 'step1']
2014-04-03 08:23:07,927 DEBUG: Dumped pickle. steps_done: ['Fetch recordings', 'preparation', 'step1']

but when I look at the pickle file, I get:

{   [...]
    'steps_done': ['Fetch recordings', 'preparation'],
    [...]}

What could be the error? What can I do to find the error?

(If open would not work, I would get an exception, right?)

Answer 1

Use absolute file paths to open your pickle files; by using relative paths you now are writing a pickle file in the current working directory.

You can write in the same location as the script by basing the path of the __file__ global:

import os

here = os.path.dirname(os.path.abspath(__file__))

then use

with open(os.path.join(here, "state.pickle"), "wb") as f:

to create an absolute path to a pickle file in the same directory.