Kafka, is it possible to get the message offset after producing?

Question

While using Apache Kafka, is it possible to get the offset of the message that has been produced?

From the Kafka documentation page it says:

each message in the partition are assigned a unique sequential id/number called Offset

Now suppose when I am producing any message I want to retrieve the offset that has been assigned to it.

Answer 1

You shouldn't do that from a class that acts as a producer, like in most queue systems, as its responsibility is to just fire and forget the messages. The broker will do the appropriate meta data handling with id's, offsets, whatever it needs to handle the messages in a correct manner.

You can get the offsets as a consumer of messages from a Kafka broker. If you look in the SimpleConsumer class, you'll notice it fetches MultiFetchResponse objects that contains offsets as a list. Furthermore when you iterate the Kafka messages you'll end up with a MessageAndOffset objects that contains both the message sent and it's offset.

Answer 2

The producer response protocol provides the offset of the first message sent in the messageset. Provided the API/Implementation you're using exposes this information, you should be able to retrieve it from the response to producing your message. More here: https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-ProduceResponse