Generic object "watcher" proxy to wrap collection objects to log state-changes (add/del)

Question

I want to add "watchers" to some variables used by a library. They are of type collections.deque, dict, weakref.WeakSet, and list.

Anytime that an item is added/appended or popped to/from them I'd like to generate a log message.

I'd like to minimize the changes to the original code:

• I don't want to add log messages in all of the places where the variables are accessed and modified.

• I don't want to create my own classes that inherit from the original classes (e.g. class InstrumentedDeque(collections.deque): ...)

Instead, and this is the question, is it possible to create a single generic class (or wrapper/decorator) that works for all of the collection objects above, so that the only place where changes are needed is where the objects are originally created. If this is the original code, with 2 vars to "watch": self.scheduled and self.ready...

def __init__(self):
    self.scheduled = []
    self.ready = collections.deque()

then the only changes required would be something like...

def __init__(self):
    self.scheduled = MyLogger([], var_name='scheduled')
    self.ready = MyLogger(collections.deque(), var_name='ready')

without instrumentation

test = Test()

test.add(1)
test.add(2)
test.pop()

after instrumentation changes

test = Test()

test.add(1)
***instrumentation output. scheduled: [1]
***instrumentation output. ready: deque([1])

test.add(2)
***instrumentation output. scheduled: [1, 2]
***instrumentation output. ready = deque([1, 2])

test.pop()
***instrumentation output. scheduled: [2]
***instrumentation output. ready: deque([2])

where the example add() and pop() would look something like this...

    def add(self, val):
        heapq.heappush(self.scheduled, val)
        self.ready.append(val)

    def pop(self):
        heapq.heappop(self.scheduled)
        self.ready.popleft()

I've tried creating a "wrapper" class and played with __new__, __init__, __getattr__ but haven't been able to get this to work. Something like this...

class MyLooger:
    def __new__(cls):
        # what to do?

    def __init__(self):
        # what to do?

    def __getattr__(self, name):
        # what to do?

Any help is greatly appreciated.

Answer 1

Note: the following abstraction does not work with C extension code that goes directly into the low-level internals of the wrapped object (such as heapq.heappush, both on CPython as well as PyPy); there's nothing that can be done to alleviate that at the Python level. You might see if you can "patch the leak" at the C level, but then you'll have to get your hands dirty with writing C and a Python extension.

Solution: You don't need to go as far as __new__. The following will work generically on all objects. It will also make isinstance work on the wrapper as if it were called on the wrapped object.

from functools import wraps

class Logged(object):
    def __init__(self, obj, obj_name):
        self.obj = obj
        self.obj_name = obj_name

    def __getattribute__(self, attr_name):
        obj = object.__getattribute__(self, 'obj')
        obj_name = object.__getattribute__(self, 'obj_name')

        attr = getattr(obj, attr_name)
        # this is not 100% generic, mathematically speaking,
        # but covers all methods and the `__class__` attribute:
        if not callable(attr) or isinstance(attr, type):
            return attr

        @wraps(attr)
        def fn(*args, **kwargs):
            print "%s called on %s with: %s and %s" % (attr_name, obj_name, args, kwargs)
            return attr(*args, **kwargs)

        return fn

    def __repr__(self):
        return repr(object.__getattribute__(self, 'obj'))

And then just:

>>> scheduled = Logged([], obj_name="scheduled")

>>> scheduled.append
<function append>

>>> scheduled.append(3)
append called on scheduled with: (3,) and {}

>>> scheduled.extend([1,2])
extend called on scheduled with: ([1, 2],) and {}

>>> isinstance(scheduled, list)
True

>>> scheduled
[3, 1, 2]