Is Kafka useful if we have less messages to process

Question

Is Kafka useful if we have less messages to process. If I have 1000 messages per second to process, is Kafka feasible?

Answer 1

As any experienced software engineer will say, it depends ;-). There are many factors to consider. Here is just a sample:

1. Do you need to have these messages persisted? If not, then probably Kafka is not what you're looking for.

2. Even if you require persistence, it doesn't mean that Kafka can handle your throughput requirements (although my gut feeling says it can cope with your volume). The only way to determine that is to run performance tests with your message volumes against Kafka and see how it copes. It's also quite possible that other brokers like ActiveMQ can handle your volumes as well. Then it falls down to how appropriate is the broker for your use case (e.g., event sourcing?) Checkout out Kafka's docs to see how Kafka is used in the industry.

3. You have to keep in mind that Kafka is currently not as popular as other brokers such as ActiveMQ. So even if Kafka is useful to your scenario, you could have a hard time finding help on Kafka questions/issues you'll have along the way.