I currently have a project that runs for several days. As errors might occur in some steps of execution, I pickle critical steps to be able to restart the script at the correct step (so I don't have to do the work of eventually over 24h of execution again).
One thing I store with pickle is a list steps
. This list contains every step that successfully finished. It is used to skip steps when I start the script again.
The problem is that pickle seems not to update this after I switch the module.
Code
mainscript.py
import subscript
def set_status(mysql_script_instance_id, status, state=None):
# [...]
# update status in database (works as expected)
# [...]
if state is not None:
with open("state.pickle", "wb") as f:
pickle.dump(state, f)
logging.debug("Dumped pickle. steps_done: %s" % state['steps_done'])
logging.info(status)
subscript.py
import mainscript
[...]
logging.info("%s finished." % (step.__name__))
self.state['steps_done'].append(step.__name__)
[...]
logging.debug("self.state['steps_done'] = %s" % self.state['steps_done'])
mainscript.set_status(self.mysql_script_instance_id, "step xy done", self.state)
pickleviewer
#!/usr/bin/env python
import pickle
import pprint
state = pickle.load(open("state.pickle", "rb"))
pp = pprint.PrettyPrinter(indent=4)
pp.pprint(state)
What I've tried
I get all log messages I've expected:
2014-04-03 08:23:07,727 INFO: step1 finished.
2014-04-03 08:23:07,728 DEBUG: self.state['steps_done'] = ['Fetch recordings', 'preparation', 'step1']
2014-04-03 08:23:07,927 DEBUG: Dumped pickle. steps_done: ['Fetch recordings', 'preparation', 'step1']
but when I look at the pickle file, I get:
{ [...]
'steps_done': ['Fetch recordings', 'preparation'],
[...]}
What could be the error? What can I do to find the error?
(If open
would not work, I would get an exception, right?)