Redis-python setting multiple key/values in one operation

Question

Currently I use the basic mset feature to store a key/value;

from common.redis_client import get_redis_client
cache = get_redis_client()
for k,v in some_dict.items():
   kw = {'key': value}
   cache.mset(kw) 

#later:
   cache.get('key')

I store each key/value separatly (not in one json for example) Since storing the whole dict would turn it into a string and would require me to serialize/deserialize on storing and retrieving and I really need access to seperate key/values.

My question:: is there a way I can mset multiple key/values at once? Instead of multiple writes to the redis db? And vice-versa can I have multiple reads (get) in one access? (and Yes - I have a lot of redis activity going on and with heavy load. I do care about this)

Answer 1

updated after Agis comment

If you use Redis-py, which is currently the recommended Redis client for python, you can use the pipelining which does exactly what you want. Here is a simple example:

>>> r = redis.Redis(...)
>>> r.set('bing', 'baz')
>>> # Use the pipeline() method to create a pipeline instance
>>> pipe = r.pipeline()
>>> # The following SET commands are buffered
>>> pipe.set('foo', 'bar')
>>> pipe.get('bing')
>>> # the EXECUTE call sends all buffered commands to the server, returning
>>> # a list of responses, one for each command.
>>> pipe.execute()
[True, 'baz']

I don't know which redis client you are using, but either it supports the pipelining, or you should consider switching to redis-py.

Have a look at redis documentation about pipelining; it explains that you can expect a x5 performance boost -but also that you must not perform too important bulk operations (10 000 operations at each execution is OK).

Answer 2

Since you seem to be looking for storing a dictionary, I see two more possible solutions:

1. Use HMGET/HMSET commands. This does allow to get and set multiple individual keys from a dictionary. The disadvantage is that it's all on one key, so if you plan for key expiration of individual dict entries - as is common with caching - this will not work for you.

2. I decided to use MGET and MSET commands. This allows to set multiple keys at once and also retrieve them at once. You already used the MSET (multi set) command in your example, but only set ONE value...

Your example might work this way:

cache.mset(some_dict)

You will want to add a common prefix to your dictionary entries:

cache.mset({'prefix:' + k:v for k,v in some_dict.items()})

This way you will be able to retrieve your dict entries with MGET by key. If you don't know the keys upfront you can run a KEYS command with a "prefix:*" wildcard before.