Django Cache: Use Memcached and fallback to FileSystem

Question

Simple question:

My server has 1G RAM and 10GB disk-space

I'm using per-site cache, and I want to use as much as possible Memcached, but when its out of space that the cache will be saved in hard disk.

(All site's pages together are about 2GB)

Is there a simple configuration to achive this?

Is it a smart thing to do at all?

Thanks

Answer 1

It sounds like you want Memcached to behave like normal RAM and page to disk when it is full. It doesn't by default, but you can imitate it by writing your own cache backend as @AlexanderAfanasiev mentioned. The implementation would be something along these lines:

First, define three caches:

CACHES = {
    'default': {
        'BACKEND': 'myapp.cache.DoubleCache',
    },
    'memcached': {
        'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
        'LOCATION': '127.0.0.1:11211',
    },
    'filecache': {
        'BACKEND': 'django.core.cache.backends.filebased.FileBasedCache',
        'LOCATION': '/foo/bar',
    }
}

Then, in myapp/cache.py:

from django.core.cache.backends.base import BaseCache
from django.core.cache import get_cache, cache
mem_cache = get_cache('memcached')
file_cache = get_cache('filecache')

class DoubleCache(BaseCache):

    def get(self, key, default=None, version=None):
        result = mem_cache.get(key, default=default, version=version)
        if result:
            return result
        else:
            return file_cache.get(key, default=default, version=version)

    def set(self, key, value, timeout=None, version=None, client=None, _add_only=False):
        memcache_result = mem_cache.set(key, value, timeout=timeout, version=version, client=client, _add_only=_add_only)
        file_result = file_cache.set(key, value, timeout=timeout, version=version, client=client, _add_only=_add_only)
        return memcache_result

This will always store values in both caches. It will retrieve values from Memcached, and if it misses, will try the file_cache. That means that Memcached can manage its own roll-off and only the oldest hits will have to fall back to file_cache. This is what you'd want.

Of course, you'll also have to implement the rest of the cache functions like delete(), get_many(), etc. Hopefully this gets you on the right path.

Answer 2

Exploit TCP/IP. With minor effort this logic can be extended to create nice multicache backend with fallbacks.

import socket
socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

try:
    socket.connect(('127.0.0.1', 11211))
    socket.close()

    CACHES = {
        'default': {
            'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
            'LOCATION': '127.0.0.1:11211',
        }
    }
except:
    CACHES = { 
        'default': {
            'BACKEND': 'django.core.cache.backends.filebased.FileBasedCache',
            'LOCATION': '/srv/django_cache',
        }
    }