De-uglify memcache boilerplate in Python

Question

I'm just getting started with Python. I'm making heavy use of caching in my app and my code is increasingly littered with this same pattern, which is the standard caching pattern I've seen used all over the shop. Are there some sexy syntactic tricks in Python that can DRY out some of this boilerplate?

(btw, this is not actual code)

# Determine if we are allowed to use cache
cacheable = settings.cache.lifetime is not None

# Generate unique cache key
cache_key = 'something_unique_{some_arg}'.format(some_arg=*args[0])

# Return cached version if allowed and available
if cacheable:
    cached = memcache.get(cache_key)
    if cached:
        return cached

# Generate output
output = do_something_fooey(args[0])

# Cache output if allowed
if cacheable:
    memcache.set(cache_key, output, settings.cache.lifetime)

return output

I'm going to have a stab at this too, probably writing a caching wrapper function and passing the output generation to it as a "delegate" (dunno if that's Python lingo), but it'd be great to get some advice from Python experts.

Answer 1

You want a decorator:

def cached(func):
    def _cached(*args):
        # Determine if we are allowed to use cache
        cacheable = settings.cache.lifetime is not None

        # Generate unique cache key

        cache_key = '{0}-{1}-{2}'.format(func.__module__, func.__name__, args[0])

        # Return cached version if allowed and available

        if cacheable:
            result = memcache.get(cache_key)
            if result is not None:
                return result

        # Generate output
        result = func(args[0])

        # Cache output if allowed
        if cacheable and result is not None:
            memcache.set(cache_key, result, settings.cache.lifetime)

        return result

    return _cached

@cached
def do_something_fooey(*args):
    return something

You may want to use functools.wraps (http://docs.python.org/2/library/functools.html#functools.wraps) for a well-behaved decorator.

Answer 2

I've found a couple of alternate pre-rolled solutions:

https://github.com/jayferd/python-cache and https://gist.github.com/abahgat/1395810

In the end I created the below, which is a fleshed-out version of @bruno's example. The nice thing about this one is that you can pass an extra_key to the decorator, which forms part of the caching key and can be either a string or a delegate function. (lifetime can also be a delegate function or an integer). This allows you to add stuff at runtime such as caching uniquely by user id.

def cached(lifetime=settings.cache.default_lifetime, extra_key=None):
    def _cached(func):
        @functools.wraps(func)
        def wrapper(*args, **kwargs):

            # Resolve lifetime if it's a function
            resolved_lifetime = lifetime(*args) if hasattr(lifetime, '__call__') else lifetime

            if resolved_lifetime is not None:

                # Hash function args
                items = kwargs.items()
                items.sort()
                hashable_args = (args, tuple(items))
                args_key = hashlib.md5(pickle.dumps(hashable_args)).hexdigest()

                # Generate unique cache key
                cache_key = '{0}-{1}-{2}-{3}'.format(
                    func.__module__,
                    func.__name__,
                    args_key,
                    extra_key() if hasattr(extra_key, '__call__') else extra_key
                )

                # Return cached version if allowed and available
                result = memcache.get(cache_key)
                if result is not None:
                    return result

            # Generate output
            result = func(*args, **kwargs)

            # Cache output if allowed
            if resolved_lifetime is not None and result is not None:
                memcache.set(cache_key, result, resolved_lifetime)

            return result

        return wrapper

    return _cached