Maximum size of object that can be saved in memcached with memcache.py

StackOverflow https://stackoverflow.com/questions/1440722

  •  10-07-2019
  •  | 
  •  

Question

I need to return a rather big file (11MB) to the user. For certain reasons, I can't just provide a direct url to the file (http://www.sample.com/mybigfile.exe); instead it must be accessed through code.

Instead of having to read it from disk over and over, I thought of saving it in memcached (if this is not a good idea, let me know). Everything seems to work, fine (no errors), but when I try to retrieve the file from memcached I always get None, as if the file wasn't cached.

Is there a size limit to what can be saved?

Here's the code:

def download_demo():
    """
    Returns the demo file
    """
    KEY = "xyz"
    TIME = 86400 #24 hours

    buff = memc.get(KEY)
    if not buff: 
        file = open(FILENAME, 'r')
        buff = file.read()
        memc.set(KEY, buff, TIME)

    print "Content-Type:application/x-download\nContent-Disposition:attachment;filename=%s\nContent-Length:%s\n\n%s" %    (os.path.split(FILENAME)[-1], len(buff), buff)
Was it helpful?

Solution

There are two entries about that in the memcached FAQ :

The answer to the first one is (quoting, emphasis mine) :

The maximum size of a value you can store in memcached is 1 megabyte. If your data is larger, consider clientside compression or splitting the value up into multiple keys.

So I'm guessing your 11MB file is quite too big to fit in one memcached entry.

Increasing the size of an object is possible, as per other answers.

OTHER TIPS

As of memcache 1.4.2, this is a user-configurable parameter: ReleaseNotes142 * memcached @ Github

Configurable maximum item size

Many people have asked for memcached to be able to store items larger than 1MB, while it's generally recommended that one not do this, it is now supported on the commandline.

A few enlightened folk have also asked for memcached to reduce the maximum item size. That is also an option.

The new -I parameter allows you to specify the maximum item size at runtime. It supports a unit postfix to allow for natural expression of item size.

Examples:

memcached -I 128k # Refuse items larger than 128k. 
memcached -I 10m  # Allow objects up to 10MB

To summarise the necessary steps:

1) Update Memcache to version 1.4.2 or later.

2) Add the flag -I 15M (or however many megabytes) to your memcache run command.

That's either command line, or in Ubuntu, add the line

-I 15M

to anywhere in /etc/memcached.conf and restart the service.

3) Add the necessary flag to the client in memcache.

import memcache
memc = memcache.Client(['localhost'], server_max_value_length=1024*1024*15)
memc.set(KEY, buff, TIME)

If you don't have direct access to the memcache client (i.e. working through a framework), then just hack the memcache code directly.

On Ubuntu, it's /usr/local/lib/python2.7/dist-packages/memcache.py. Change the line:

SERVER_MAX_ITEM_LENGTH = 1024 * 1024

to 

SERVER_MAX_ITEM_LENGTH = 1024 * 1024 * 15

Obviously, you'll need to do this hack over again if you update memcache, but it's a very simple and quick fix.

The maximum data size is 1mb per item

http://code.google.com/p/memcached/wiki/FAQ#What_is_the_maximum_data_size_you_can_store?_%281_megabyte%29

Update: This is a 2009 answer. And in that date the info was accurate and the source was official. Now in 2014 memcached can store 128mb instead of 1mb (but the developers didn't bother to update the official FAQ). The only reference someone could find id an obscure page that probably will be dead in one year.

Following article explained why it limit max size of a single file to 1M.

http://www.mikeperham.com/2009/06/22/slabs-pages-chunks-and-memcached/

Basicly, there are small pieces of memory pages in memcache, inside which objects reside.

And the default page size seems to be 1M, so the max object a page can contain is limited to 1M.

Even though you can config it to increase the size, but I think this might have some kind of performance trade-off.

If you are using python memcached client, please also make sure to increase the limit in client memcache.py file along with the memcached server.

~/anaconda/lib/python2.7/site-packages/memcache.py

SERVER_MAX_KEY_LENGTH = 250 
SERVER_MAX_VALUE_LENGTH = 1024 * 1024 * 15

you can see in memcached console ( telnet localhost 11211 )

stats slabs STAT 48:chunk_size 3677344

Nobody seems to address this part of the question:

Instead of having to read it from disk over and over, I thought of saving it in memcached (if this is not a good idea, let me know).

This is not a good idea. All modern file systems have efficient caching mechanisms. Reading a file from disk that was read a few seconds earlier will typically still be in memory cache. This is much faster than accessing memcached over a network.

If you are running memcached on 127.0.0.1, the memory it claimed will only be used for that single file, wereas if you simply rely on the caching of your operating system the memory can be used for various purposes depending on the jobs at hand.

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top