Redis and premissions

Question

I want to use redis, but i want to be able to limit the clients that will be allowed to get a value of a key...

The idea is a main machine who transfer 'tasks' in the shape of a pythonic dict throgh redis and only one client/worker is allowed to get it, it doesn't matter which worker, only that one worker will get it.

I tried looking for a way to communicate privatly so the main machine will be able to give workers the key, which will be a uuid, and that way only one worker will get it... Couldn't find a way though.

Thanks a lot!

Answer 1

You do not need specific permission management to distribute a given workload to a set of workers so that one item is processed by only one worker: you just need a queuing system.

You can easily build one on top of Redis by using the list data type, and primitives such as LPUSH/RPUSH to queue, and BRPOP/BLPOP to dequeue in blocking mode.

You can also reuse one of the numerous existing Python modules/examples:

• http://celeryproject.org/
• http://python-rq.org/
• https://github.com/josiahcarlson/rpqueue
• http://peter-hoffmann.com/2012/python-simple-queue-redis-queue.html
• http://www.saltycrane.com/blog/2011/11/unique-python-redis-based-queue-delay/
• http://richardhenry.github.io/hotqueue/
• http://itybits.com/pyres/index.html

Update:

If you need a query-reply mechanism, with load balancing on several workers (something like Gearman), this can still be implemented with Redis (even if it is probably not the most efficient way to do it).

The client process can generate a UUID on startup to identify itelf. Then it can push jobs to queue:main (main queue) with RPUSH/LPUSH. The items should contain the payload itself, plus the identity (UUID) of their sender. Then, the client just has to wait the answers on queue:UUID (BLPOP/BRPOP).

If you want to support pipelining (i.e. sending multiple queries and waiting for multiple replies), you need to way to identify the items themselves (on top of the sender), so that the client process can safely associate queries and replies.

The workers just dequeue items from queue:main (BLPOP/BRPOP), process the payload, retrieve the UUID of the sender, and returns the result by pushing them to queue:UUID (with LPUSH/RPUSH). If the items are identified, be sure to return the identity of the items with the results.

Note: you do not need to explicitly create or delete queues. A Redis list is automatically created when the first item is pushed, and removed when the last item is popped.