Python - how to moderate list of weakproxy objects

Question

So weakref.proxy objects don't at all seem to work like weakref.ref objects when it comes to checking if the reference is live, or 'dereferencing' them, or really in just about any way at all, actually. :P

They do still seem to have their place - such as maintaining a list of objects which respond to events. Since one doesn't need to deref it to call methods on the objects they are weakly referencing (unlike weakref.ref objects which need to be called first), one does gain some time back from the use of proxy objects over thousands of iterations. However, they seem to be harder to actually identify as 'dead' and summarily clean up once the object they are referencing disappears. Par example -

>>> mylist    #note the object at mylist[1] is already dead...
[<weakproxy at 0x10ccfe050 to A at 0x10ccf9f50>, <weakproxy at 0x10ccfe1b0 to NoneType at 0x10cb53538>]
>>> for i in mylist[:]:
...     try:
...             getattr(i, 'stuff')   #the second arg could be any sentinel;
...                                   #I just want to hit a ReferenceError
...     except AttributeError:
...             pass
...     except ReferenceError:
...             mylist.remove(i)
... 
Traceback (most recent call last):
  File "<stdin>", line 7, in <module>
ReferenceError: weakly-referenced object no longer exists

So it lives there, it just can't really be referenced directly or passed around like a variable, because that apparently causes Python to 'dereference' it and work with the object it's weakly pointing at. (Or something.)

The only thing I've found so far that seems to be somewhat reliable is to route it through a try/except, but it feels a bit like a runaround.

for i, obj in enumerate(mylist):
    try:
        obj.__hash__
    except ReferenceError:
        del mylist[i]

It works, but Python tends to be a "one right answer for all the things" language, and this feels like a pretty hefty solution - I might be wrong on this, but if the list is sufficiently large, doesn't copying the list in this way lend itself to issues?

Unfortunately this is all I can seem to really think of as a potential solve that doesn't involve type-checking or other garbage, but I'm guessing I just missed something in the weakref documentation about how to handle weakref.proxy objects appropriately. It 'feels like' looking to ensure a weakref.proxy isn't exceptional circumstances, so using a try/except is a one-off utility.

So if I am right in my assumption about the use of a try/except here, is there a better method for identifying dead weakref.proxy objects?

EDIT: I've accepted an answer, so thank you for that - try/except seems like the only acceptable out.

As to why I'm not using WeakSets - allow me to post a simple demonstration which ultimately led me to the use of proxy objects.

>>> from weakref import WeakSet, proxy
>>> import cProfile
>>> class A(object):
...     def __init__(self):
...             self.x = 0
...     def foo(self, v=1):
...             self.x += v
... 
>>> Stick = A()
>>> Dave = A()
>>> Jupiter = A()
>>> ##just a list of objects, no fancy
>>> one_group = [Stick, Dave, Jupiter]
>>> cProfile.run("for i in xrange(0, 2**16): [x.foo(i) for x in one_group]")
         196610 function calls in 0.136 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   196608    0.049    0.000    0.049    0.000 <stdin>:4(foo)
        1    0.087    0.087    0.136    0.136 <string>:1(<module>)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}


>>> Stick.x
2147450880
>>> Dave.x
2147450880
>>> Jupiter.x
2147450880

So we know it works, and we know that it is fairly fast for 65k iterations. Seems decent; let's take a look at WeakSets.

>>> ##now a WeakSet of objects. should be ideal; but...
>>> two_group = WeakSet((Stick, Dave, Jupiter))
>>> cProfile.run("for i in xrange(0, 2**16): [x.foo(i) for x in two_group]")
         851970 function calls in 0.545 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   196608    0.055    0.000    0.055    0.000 <stdin>:4(foo)
        1    0.158    0.158    0.545    0.545 <string>:1(<module>)
    65536    0.026    0.000    0.026    0.000 _weakrefset.py:16(__init__)
    65536    0.043    0.000    0.051    0.000 _weakrefset.py:20(__enter__)
    65536    0.063    0.000    0.095    0.000 _weakrefset.py:26(__exit__)
    65536    0.024    0.000    0.024    0.000 _weakrefset.py:52(_commit_removals)
   262144    0.159    0.000    0.331    0.000 _weakrefset.py:58(__iter__)
    65536    0.009    0.000    0.009    0.000 {method 'add' of 'set' objects}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
    65536    0.008    0.000    0.008    0.000 {method 'remove' of 'set' objects}

Premature optimization? Nah. :) That's about 4x slower. Blimey.

>>> ##now finally, a list of proxy objects
>>> three_group = [proxy(x) for x in one_group]
>>> cProfile.run("for i in xrange(0, 2**16): [x.foo(i) for x in three_group]")
         196610 function calls in 0.139 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   196608    0.050    0.000    0.050    0.000 <stdin>:4(foo)
        1    0.089    0.089    0.139    0.139 <string>:1(<module>)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

Based on these numbers, I came to the idea that the simplest and fastest method for maintaining a list of objects which must correctly be noted as deallocated is to use proxy objects. I think that strategic use of the try/except as a 'list cleaner' will ensure that dead proxies don't cause errors. For their convenience, I may yet return to an approach which employs weakref.ref objects, but the use of proxy objects is intriguing for their seemingly 'direct' access to the object.

Answer 1

Your weak reference object raises the ReferenceError exception again on the mylist.remove(i) line, not when looping or accessing, most likely because Python tries to use the __eq__ method on the proxy.

Your second method uses an index instead and doesn't trigger that exception, use that in your first loop:

remove = set()
for index, proxy in enumerate(mylist):
    try:
        getattr(proxy, 'stuff')
    except AttributeError:
        pass
    except ReferenceError:
        remove.add(index)

mylist = [p for i, p in enumerate(mylist) if i not in remove]

Demo:

>>> import weakref
>>> class Foo(object): pass
... 
>>> remove = set()
>>> mylist = [weakref.proxy(Foo())]  # instant dead proxy
>>> for index, proxy in enumerate(mylist):
...     try:
...         getattr(proxy, 'stuff')
...     except AttributeError:
...         pass
...     except ReferenceError:
...         remove.add(index)
... 
>>> remove
set([0])

If you specifically wanted a function that tests for the object being live still, you could use:

def proxy_live(p)
    try:
        bool(p)
    except ReferenceError:
        return False
    return True

but take into account that the boolean test itself could trigger deletion of the object, if the proxied object hooks into attribute access, the __nonzero__ or __len__ methods and triggers a delete. That, and threads, can then lead to race conditions where the above function would return True and you'll still have to take into account that actions on the object can raise a ReferenceError.

If order isn't important, I'd use a weakref.WeakSet() object here; when looping over these you get the live objects; already dereferenced with dead references pruned for you, without risk of race conditions.