How to do cleanup reliably in python?

Question

I have some ctypes bindings, and for each body.New I should call body.Free. The library I'm binding doesn't have allocation routines insulated out from the rest of the code (they can be called about anywhere there), and to use couple of useful features I need to make cyclic references.

I think It'd solve if I'd find a reliable way to hook destructor to an object. (weakrefs would help if they'd give me the callback just before the data is dropped.

So obviously this code megafails when I put in velocity_func:

class Body(object):
    def __init__(self, mass, inertia):
        self._body = body.New(mass, inertia)

    def __del__(self):
        print '__del__ %r' % self
        if body:
            body.Free(self._body)

    ...        

    def set_velocity_func(self, func):
        self._body.contents.velocity_func = ctypes_wrapping(func)

I also tried to solve it through weakrefs, with those the things seem getting just worse, just only largely more unpredictable.

Even if I don't put in the velocity_func, there will appear cycles at least then when I do this:

class Toy(object):
    def __init__(self, body):
        self.body.owner = self

...

def collision(a, b, contacts):
    whatever(a.body.owner)

So how to make sure Structures will get garbage collected, even if they are allocated/freed by the shared library?

There's repository if you are interested about more details: http://bitbucket.org/cheery/ctypes-chipmunk/

Answer 1

If weakrefs aren't broken, I guess this may work:

from weakref import ref

pointers = set()

class Pointer(object):
    def __init__(self, cfun, ptr):
        pointers.add(self)
        self.ref = ref(ptr, self.cleanup)
        self.data = cast(ptr, c_void_p).value # python cast it so smart, but it can't be smarter than this.
        self.cfun = cfun

    def cleanup(self, obj):
        print 'cleanup 0x%x' % self.data
        self.cfun(self.data)
        pointers.remove(self)

def cleanup(cfun, ptr):
    Pointer(cfun, ptr)

I yet try it. The important piece is that the Pointer doesn't have any strong references to the foreign pointer, except an integer. This should work if ctypes doesn't free memory that I should free with the bindings. Yeah, it's basicly a hack, but I think it may work better than the earlier things I've been trying.

Edit: Tried it, and it seem to work after small finetuning my code. A surprising thing is that even if I got del out from all of my structures, it seem to still fail. Interesting but frustrating.

Neither works, from some weird chance I've been able to drop away cyclic references in places, but things stay broke.

Edit: Well.. weakrefs WERE broken after all! so there's likely no solution for reliable cleanup in python, except than forcing it being explicit.

Answer 2

What you want to do, that is create an object that allocates things and then deallocates automatically when the object is no longer in use, is almost impossible in Python, unfortunately. The del statement is not guaranteed to be called, so you can't rely on that.

The standard way in Python is simply:

try:
    allocate()
    dostuff()
finally:
    cleanup()

Or since 2.5 you can also create context-managers and use the with statement, which is a neater way of doing that.

But both of these are primarily for when you allocate/lock in the beginning of a code snippet. If you want to have things allocated for the whole run of the program, you need to allocate the resource at startup, before the main code of the program runs, and deallocate afterwards. There is one situation which isn't covered here, and that is when you want to allocate and deallocate many resources dynamically and use them in many places in the code. For example of you want a pool of memory buffers or similar. But most of those cases are for memory, which Python will handle for you, so you don't have to bother about those. There are of course cases where you want to have dynamic pool allocation of things that are NOT memory, and then you would want the type of deallocation you try in your example, and that is tricky to do with Python.

Answer 3

In CPython, __del__ is a reliable destructor of an object, because it will always be called when the reference count reaches zero (note: there may be cases - like circular references of items with __del__ method defined - where the reference count will never reaches zero, but that is another issue).

Update From the comments, I understand the problem is related to the order of destruction of objects: body is a global object, and it is being destroyed before all other objects, thus it is no longer available to them.
Actually, using global objects is not good; not only because of issues like this one, but also because of maintenance.

I would then change your class with something like this

class Body(object):
    def __init__(self, mass, inertia):
        self._bodyref = body
        self._body = body.New(mass, inertia)

    def __del__(self):
        print '__del__ %r' % self
        if body:
            body.Free(self._body)

...        

def set_velocity_func(self, func):
    self._body.contents.velocity_func = ctypes_wrapping(func)

A couple of notes:

1. The change is only adding a reference to the global body object, that thus will live at least as much as all the objects derived from that class.
2. Still, using a global object is not good because of unit testing and maintenance; better would be to have a factory for the object, that will set the correct "body" to the class, and in case of unit test will easily put a mock object. But that's really up to you and how much effort you think makes sense in this project.