Node.js + Socket.io and Redis Store issues when redis restarts

Question

I'm hoping someone else has encountered this issue before:

Issue Description

I have Socket.IO configured to use the redis-store module, which is working great for keeping our node instances in sync when broadcasting to a room / emitting events. However I've been testing for failure points and it seems that when the redis instance is restarted the node servers stop receiving messages from other nodes.

Here is how I am setting up the socket.io client (redis connections are created before this bit of code)

// Setup the socket server for web clients
self.ioServer = socketIo.listen(9000);
self.ioServer.enable('browser client minification');    // send minified client
self.ioServer.set('log level', 3);

self.ioServer.set( 'store', new socketIo.RedisStore ({
        redisPub: self.redisPub, 
        redisSub: self.redisSub, 
        redisClient: self.redisStore
    })
);

In essence, the socket.io redis-store fails to continue sending events through redis to the other node servers whenever redis crashes / restarts.

Are there any special steps that I need to take for reconnecting to the redis server? I've spent quite a bit of time searching around to hopefully find a solution to this issue, but haven't had any luck.

Regards,

-Ryan

Answer 1

Incase anyone is interested here is the fix I've implemented for the above issue:

Since the system is using cluster for the socket.io nodes, I'm catching any 'end' events on the redis connections (reidsPub,redisSub, etc.) and killing the forked process. On the master process I catch the exit of the child process and re-fork.

On a side note, I've also discovered another annoying issue with redis and socket.io's redis-store: The redis connections will eventually timeout if no data is exchanged with redis. To solve this I've had to implement a simple keep alive function that runs every 15 seconds that exchanges data with redis for all the above redis connections.

Normally I would be ok using the default redis client reconnection logic, but since I have no way of knowing if the redis server actually restarted, or if it was just a timeout.

Answer 2

I have not gone over redis-store's implementation in much detail. I do not believe pub-sub with redis is backed to disk by default and in case the server is restarted, the messages are lost. Unless redis-store is doing the same already for you, you could use either lists or perhaps sorted sets to store the messages in addition to using pub-sub.

In case not provided by redis-store by default, you will need to ensure that your list/sorted set does not grow too large ti negatively impact performance. You could look into using MULTI/EXEC (and look into ACK messages) transactions for ACID protection and to ensure that messages are not removed unless processed/consumed by your subscribers.

You could always look into other messaging servers such as ActiveMQ etc which provide durable (disk backed) messaging queues/topics.

Hope it helps.