Python decorator, self is mixed up [duplicate]

Question

This question already has an answer here:

• Decorating Python class methods - how do I pass the instance to the decorator? 4 answers

I am new to Python decorators (wow, great feature!), and I have trouble getting the following to work because the self argument gets sort of mixed up.

#this is the decorator
class cacher(object):

    def __init__(self, f):
        self.f = f
        self.cache = {}

    def __call__(self, *args):
        fname = self.f.__name__
        if (fname not in self.cache):
            self.cache[fname] = self.f(self,*args)
        else:
            print "using cache"
        return self.cache[fname]

class Session(p.Session):

    def __init__(self, user, passw):
        self.pl = p.Session(user, passw)

    @cacher
    def get_something(self):
        print "get_something called with self = %s "% self
        return self.pl.get_something()

s = Session(u,p)
s.get_something()

When I run this, I get:

get_something called with self = <__main__.cacher object at 0x020870F0> 
Traceback:
...
AttributeError: 'cacher' object has no attribute 'pl'

for the line where I do self.cache[fname] = self.f(self,*args)

The problem - Obviously, the problem is that self is the cacher object instead of a Session instance, which indeed doesn't have a pl attribute. However I can't find how to solve this.

Solutions I've considered, but can't use - I thought of making the decorator class return a function instead of a value (like in section 2.1 of this article) so that self is evaluated in the right context, but that isn't possible since my decorator is implemented as a class and uses the build-in __call__ method. Then I thought to not use a class for my decorator, so that I don't need the __call__method, but I can't do that because I need to keep state between decorator calls (i.e. for keeping track of what is in the self.cache attribute).

Question - So, apart from using a global cache dictionary variable (which I didn't try, but assume will work), is there any other way to make this decorator work?

Edit: this SO question seems similar Decorating python class methods, how do I pass the instance to the decorator?

Answer 1

Use the descriptor protocol like this:

import functools

class cacher(object):

    def __init__(self, f):
        self.f = f
        self.cache = {}

    def __call__(self, *args):
        fname = self.f.__name__
        if (fname not in self.cache):
            self.cache[fname] = self.f(self,*args)
        else:
            print "using cache"
        return self.cache[fname]

    def __get__(self, instance, instancetype):
        """Implement the descriptor protocol to make decorating instance 
        method possible.

        """

        # Return a partial function with the first argument is the instance 
        #   of the class decorated.
        return functools.partial(self.__call__, instance)

Edit :

How it's work ?

Using the descriptor protocol in the decorator will allow us to access the method decorated with the correct instance as self, maybe some code can help better:

Now when we will do:

class Session(p.Session):
    ...

    @cacher
    def get_something(self):
        print "get_something called with self = %s "% self
        return self.pl.get_something()

equivalent to:

class Session(p.Session):
    ...

    def get_something(self):
        print "get_something called with self = %s "% self
        return self.pl.get_something()

    get_something = cacher(get_something)

So now get_something is an instance of cacher . so when we will call the method get_something it will be translated to this (because of the descriptor protocol):

session = Session()
session.get_something  
#  <==> 
session.get_something.__get__(get_something, session, <type ..>)
# N.B: get_something is an instance of cacher class.

and because :

session.get_something.__get__(get_something, session, <type ..>)
# return
get_something.__call__(session, ...) # the partial function.

so

session.get_something(*args)
# <==>
get_something.__call__(session, *args)

Hopefully this will explain how it work :)

Answer 2

Closures are often a better way to go, since you don't have to muck about with the descriptor protocol. Saving mutable state across calls is even easier than with a class, since you just stick the mutable object in the containing scope (references to immutable objects can be handled either via the nonlocal keyword, or by stashing them in a mutable object like a single-entry list).

#this is the decorator
from functools import wraps
def cacher(f):
    # No point using a dict, since we only ever cache one value
    # If you meant to create cache entries for different arguments
    # check the memoise decorator linked in other answers
    print("cacher called")
    cache = []
    @wraps(f)
    def wrapped(*args, **kwds):
        print ("wrapped called")
        if not cache:
            print("calculating and caching result")
            cache.append(f(*args, **kwds))
        return cache[0]
    return wrapped

class C:
    @cacher
    def get_something(self):
        print "get_something called with self = %s "% self

C().get_something()
C().get_something()

If you aren't completely familiar with the way closures work, adding more print statements (as I have above) can be illustrative. You will see that cacher is only called as the function is defined, but wrapped is called each time the method is called.

This does highlight how you need to be careful with memoisation techniques and instance methods though - if you aren't careful to account for changes in the value of self, you will end up sharing cached answers across instances, which may not be what you want.

Answer 3

First, you explicitly pass cacher object as first argument in the following line:

self.cache[fname] = self.f(self,*args)

Python automatically adds self to the list of arguments for methods only. It converts functions (but not other callables as your cacher object!) defined in class namespace to methods. To get such behavior I see two ways:

1. Change your decorator to return function by using closures.
2. Implement descriptor protocol to pass self argument yourself as it's done in memoize decorator recipe.