Ensure release of resources in interactive session

Question

I have a class whose objects use system resources (creating temporary files, for example) that need to be cleaned up. At the very least, this needs to happen when the program ends. So far I've been using contextmanager to do this, for example

@contextlib.contextmanager
def tempdir(prefix='tmp'):
    tmpdir = tempfile.mkdtemp(prefix=prefix)
    try:
        yield tmpdir
    finally:
        shutil.rmtree(tmpdir)

with tempdir() as tmp:
   do_something(tmp)

This ensures that the resources are released as soon as I'm done with them, and this works well for most of my purposes. However, I find that this model breaks down when using python interactively, for example in ipython. In this case, the entire session would need to be inside a with block, which isn't very practical (in fact, ipython will only evaluate blocks all at once, so the whole interactivity would disappear).

Is there a way to ensure cleanup of resources while still staying usable in interactive sessions? In C++ this could be achieved by putting the cleanup in the destructor. The python __del__ statement is similar, but I have been told that __del__ is unreliable and should be avoided. In particular, it is it is not guaranteed that __del__ will be called for objects that still exist when the interpreter exits.. Is this one of the cases where __del__ is still the best solution? And if the contextmanager approach is incompatible with interactive sessions, why is it recommended as the most "pythonic" approach?

Answer 1

I don't think there is a silver bullet for your problem, but you can somehow solve it by:

• explicitly calling method to dispose resources (your close method). The only drawback is, well, the explicitness.

• creating a thin wrapper for interactive interpreter, that registers instance's close method at exit using atexit module. Drawback is that all of your resources will be released usually later than you would like.

• creating helper functions (it's not hard to create them dynamically), that wraps usage of resources. It's not feasible if you need to call more functions with one resource. For example:

  def do_something()
      with tempdir() as tmp:
          return original.do_something(tmp)
  

• creating a solution that would hide resource handling. For example I don't care about TCP sockets, ssl, 301/302 redirection, opening certificate file etc., I just need to send a GET request over https using one specific certificate. Naturally it depends on problem you would like to solve.

Answer 2

I ended up following prokopst's atexit suggestion, and defined this class decorator:

import atexit

_toclean_ = set()
def call_exit_for_objects(objs):
        """Calls __exit__ for all objects in objs, leaving objs empty."""
        while len(objs) > 0:
                obj = objs.pop()
                obj.__exit__(None,None,None)
atexit.register(call_exit_for_objects, _toclean_)

def autoclean(cls):
        global _toclean_
        # Fail on purpose if __init__ and __exit__ don't exist.
        oldinit  = cls.__init__
        oldexit  = cls.__exit__
        def newinit(self, *args, **kwargs):
                oldinit(self, *args, **kwargs)
                _toclean_.add(self)
        def newexit(self, type, value, traceback):
                try:    _toclean_.remove(self)
                except KeyError: pass
                oldexit(self, type, value, traceback)
        cls.__init__ = newinit
        cls.__exit__ = newexit
        return cls

With this, I can have a class that supports both with syntax and interactive. For example, for the tmpdir class above, I would redefine it as:

@autoclean
class tempdir
    def __init__(self, prefix='tmp'):
        self.dir = tempfile.mkdtemp(prefix=prefix)
    def close(self): shutil.rmtree(self.dir)
    def __enter__(self): return self
    def __exit__(self, type, value, traceback): self.close()
    def __str__(self): return self.dir

And then use it as either:

with tempdir() as tmp:
    do_something(tmp)

or

tmp = tempdir()
do_something(tmp)
tmp.close() # If this is skipped, @autoclean ensures it
              still happens when python exits