Monitoring Kafka Consumer Lag

Question

I want to write a shell script to monitor consumer lag in my cluster using

bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zkconnect localhost:2181 group test

mentioned at http://kafka.apache.org/documentation.html#basic_ops_consumer_lag. I have 3 servers that are being used as zookeeper's do I need provide the address to all 3 as a parameter to get a report for a consumer groups running.

Answer 1

Generally, no. Zookeepers will communicate among themselves and then respond with an answer that is in agreement with all of the servers. This is one of the assumptions that Zookeeper runs on, unless they are experiencing failures.

If you query one, if it has enough information, it will respond, and, if not, it will ask the other zookeepers for the information and then reply.

Answer 2

Well, in general you should because if you specify a minority of servers in your zkconnect and that minority is not available your call will fail even if the zookeeper ensemble is up and running.