Handling transactions/data lost in ZeroMQ?

Question

I couldn't tell from looking at the FAQ and documentation. What happens if the power goes out? Does zeromq forget everything? Are all messages stored on the harddrive as transactions? Lets say I have a site with source code. Lets say 5 people clicked on download zip on 5 different large projects and it will take me a while to zip it. Lets say I forward them to a page saying we'll email you when it's ready. There is a power surge and the machine reboots at this point.

How do I recover and process these messages? In my head I think 1) I have to send the message 2) ZeroMQ must store it somehow 3) A client must take a message and mark it as pending 4) Another client should take a different message to work on 5) The client either finishes sending data/a reply or dies which zeromq changes pending to not-processed.

How do I handle data that shouldn't be lost? I don't think I'm suppose to store it in an ACID db?

Answer 1

ZeroMQ doesn't persist messages to a storage device, partly because that would be very inefficient, so you must handle this yourself.

If a downstream connection dies (loss of power, network failure, etc) then ZeroMQ will buffer the data until the connection is restored. However, buffer space is finite, and depending on your throughput, may not be at all sufficient. The company I work for needed to solve this problem, so we used a memory mapped file for buffering data, which works very well.

Your best option to avoid any kind of data loss is probably to take a similar route to what we did. Using request / reply is the best way to guarantee that a sent message was received downstream.

If data loss is mission critical then there are a bevy of UPS solutions that can help mitigate power outages, and as long as the program and the memory are kept alive, ZeroMQ is generally very good at handling error scenarios.