Kafka and python - I dont get how to consume

Question

From the docs..[1]: https://github.com/mumrah/kafka-python

# To send messages asynchronously
producer = SimpleProducer(kafka, async=True)
producer.send_messages("my-topic", "async message")

# To consume messages
consumer = SimpleConsumer(kafka, "my-group", "my-topic")
for message in consumer:
    print(message)

Where did the "my-group" come from? How do I set "my-group"?

Answer 1

According to the docstring of SimpleConsumer, group is "a name for this consumer, used for offset storage and must be unique".

So it's just a unique name you give to a consumer.

Answer 2

groups are used by kafka to coordinate multiple consumers reading from the same topic. if multiple consumers use the same group name, it tells kafka that they want to share the consumer load and each message will only be delivered to one of the group members.

if you don't already know you want groups, you probably don't. just make sure the name is unique so that kafka sends your consumer all available messages for the topic.

Answer 3

As described before, every message should be delivered only once to every consumer group. For more information, check http://kafka.apache.org/documentation.html#introduction.

Until last version of kafka-python (0.9.3 when writing this), the consumer groups don't work properly using SimpleConsumer.