ZeroMQ - Backwards Compatability between 4.x & 2.2 with LogStash

Question

I have a logstash server setup with the ZeroMQ input plugin to collect data from a simple application I've written. When I run the application on a system with the same ZeroMQ install (Ubuntu, apt-get install libzmq1) it works beautifully.

However, on our real systems we use ZeroMQ 4.0.1, and logstash reports that the data payload is empty. I've added as much debug as I can think of, and I'm starting to think it's a simple version incompatability (despite ZeroMQ saying this should work). The application I'm using to test is trivial:

//  Prepare our context and socket
zmq::context_t context (1);
zmq::socket_t socket (context, ZMQ_PUB);

socket.connect ("tcp://logstash:2120");

for (int request_nbr = 0; request_nbr != 10; request_nbr++) {
    dataList["number"] = boost::lexical_cast<std::string>(request_nbr);

    std::string log = map2json(dataList);
    zmq::message_t request (log.size());
    memcpy ((void *) request.data (), log.c_str(), log.size());
    std::cout << "Sending msg " << request_nbr << "…" << std::endl;
    socket.send (request);
    sleep(1);
}

This works perfectly when the app is linked again ZMQ2.2 (from libzmq1), but when I link the app against my ZeroMQ 4.0.1 install (built from source), logstash fails with a json parsing error (stating that the payload is empty).

Update

I installed ZeroMQ 3.2.4 from source on the Logstash server, and now it works.

Answer 1

I think this is a bug in the LogStash utilization of ZeroMQ 2.2, or at least an incompatability between 2.2 and 4.0.

I installed ZeroMQ 3.2.4 (built from source) on the server, and now my 4.x client is able to Pub logs to my Logstash server.

Answer 2

Pubsub filtering is reversed from 2.x to 3.x, and does not work if you are trying to use topic subs.

I posted a question, see Jordan's response on the below thread.

https://groups.google.com/forum/m/#!msg/logstash-users/Tx_J7eCFUNI/pS3qkTRCFDIJ