The problem you are having is caused by a reference cycle and an implicit __del__
defined on your generator (it's so implicit, CPython doesn't actually show __del__
when you introspect, because only the C level tp_del
exists, no Python-visible __del__
is created). Basically, when a generator has a yield
inside:
- A
try
block, or equivalently - A
with
block
it has an implicit __del__
-like implementation. On Python 3.3 and earlier, if a reference cycle contains an object whose class implements __del__
(technically, has tp_del
in CPython), unless the cycle is manually broken, the cyclic garbage collector cannot clean it up, and just sticks it in gc.garbage
(import gc
to gain access), because it doesn't know which objects (if any) must be collected first to clean up "nicely".
Because your class's __acquire_resources__(self)
contains a reference to the instance's self
, you form a reference cycle:
self
-> self.__res_mgr__
(generator object) -> generator frame (referencing locals which includes) -> self
Because of this reference cycle, and the fact that the generator has a try
/finally
in it (creating tp_del
equivalent to __del__
), the cycle is uncollectable, and your finally
block never gets executed unless you manually advance self.__res_mgr__
(which defeats the whole purpose).
You experiment happens to display this problem automatically because the reference cycle is implicit/automatic, but any accidental reference cycle where an object in the cycle has a class with __del__
will trigger the same problem, so even if you just did:
class Foo():
def __init__(self):
# Acquire some resources here
print "Initialize"
self.f = 1
def __del__(self):
# Release the resources here
print "Releasing Resources"
self.f = 0
if the "resources" involved could conceivably lead to a reference cycle with an instance of Foo
, you'd have the same problem.
The solution here is one or both of:
- Make your class a context manager so users provide the information necessary for deterministic finalization (by using
with
blocks) as well as providing an explicit cleanup method (e.g.close
) for whenwith
blocks aren't feasible (part of another object's state that is cleaned up through its own resource management). This is also the only way to provide deterministic cleanup on most non-CPython interpreters where reference counting semantics have never been used (so all finalizers are called non-deterministically, if at all) - Move to Python 3.4 or higher, where PEP 442 resolves the issue with uncollectable cyclic garbage (it's technically still possible to produce such cycles on CPython, but only via third party extensions that continue to use
tp_del
instead of updating to use thetp_finalize
slot that allows cyclic garbage to be cleaned properly). It's still non-deterministic cleanup (if a reference cycle exists, you're waiting on the cyclic gc to run, sometime), but it's possible, where pre-3.4, cyclic garbage of this sort could not be cleaned up at all.