Redis Read-Replicas On Web Servers

Question

I am currently developing a system that makes heavy use of redis for a series of web services.

One of the key criteria of this system is fast responses.

At present the layout (ignoring load balancers etc) is as follows:

• 2 x Front End Play Framework 2.x Servers
• 2 x Job Handling/Persistence Play Framework 2.x Servers
• 1 x MySQL Server
• 2 x Redis Servers, 1 master, 1 slave

In this setup, redis serves 2 tasks - as a shared cache and also as a message bus.

Currently the front end servers host a service which interacts in its entirety with Redis.

The front end servers try to balance reads across the pool of read servers (currently the master and 1 slave), but being Redis they need to make their writes to the master server. They handle cache updates etc by sending messages on the queues, which are picked up by the job handling servers.

The job handling servers do blocking listens (BLPOP) to the Redis write server and process tasks when necessary. They have the only connection to MySQL.

At present the read replica server is a dedicated server - more there to be able to switch it to write master if the current master fails.

I was thinking of putting a read replica slave of redis on each of the front end servers which means that read latency would be even less, and writes (messages for queues) get pushed to the write server on a separate connection.

If I need to scale, I could just add more front end servers with read slaves.

It sounds like a win/win to me as even if the write server temporarily drops out, the front end servers can still read data at least from their local slave and act accordingly.

Can anyone think of reasons why this might not be such a good idea?

Answer 1

I understand the advantages of this approach... but consider this: what happens when you need to scale just one component (i.e. FE server or Redis) but not the other? For example, more traffic could mean you'll need more app servers to handle it while the Redises will be considerably less loaded. On the other hand, if your dataset grows and/or more load is put on the Redises - you'll need to scale these and not the app.

The design should fit your requirements, and the simplicity of your suggested setup has a definite appeal (i.e. to scale, just add another identical lego block) but from my meager experience - anything that sounds too good to be true usually is. In the longer run, even if this works for you now, you may find yourself in a jam down the road. My advice - separate your Redis(es) from you app servers, deal with and/or work around the network and make sure each layer is available and scalable on its own right.