How can I handle IOException when Kafka is down?

Question

I am trying to publish message, and Apache Kafka is down. How should I handle an emergency situation like this?

KafkaProducer::send() method doesn't throw any exceptions that could be handled. Producer swallows them and logs errors instead, so I get flooded with messages like this, and everything hangs until Kafka gets connected again.

2014-03-31 09:38:23.752 ERROR o.a.kafka.common.network.Selector - Error in I/O: 
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.7.0_51]
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) ~[na:1.7.0_51]
    at org.apache.kafka.common.network.Selector.poll(Selector.java:205) ~[kafka-clients-0.8.1.jar:na]
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:212) [kafka-clients-0.8.1.jar:na]
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:150) [kafka-clients-0.8.1.jar:na]
    at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]

Answer 1

Call get() on the future returned by Producer.send(...), or if you don't want to block your code, pass a callback.

try { producer.send(new ProducerRecord("mytopic", key, value)) .get(); // block until acknowledged } catch(Exception e) { // handle message wasn't acknowledged }

Answer 2

I think the one thing you could do is to use a try/catch to log all Throwable in your producer code.