Hundreds of Redis instances on the same server

Question

Is this doable or will this degrade CPU performance?

The reason I want to do this is presharding. I want to shard my data across many (128, or possibly 256) Redis instances, and then run them all on the same server. Then when I need to scale out I can just fire up more servers and easily move the instances to other servers.

I know Redis itself has a small memory footprint so memory overhead would presumably not be a problem. But if I have just 4 cores, would they be handle hundreds of Redis instances? Or should I maintain a max of one instance per core?

Answer 1

Redis works like single-threaded application, it forks additional threads for e.g. I/O. So I would say that it could work one core per one redis instance just fine. As you still share some resources on multi-core computer among cores. So you can barely achieve same performance, but I don't see problem here.

The problem could appear once you would switch "durability" on to allow persistence to you hard drive. Even if redis itself uses new forked thread to provide persistence, I don't believe that it would not hurt performance a lot for loaded system.

As I've already said, it could work pretty well w/o persistence.