How to achieve multi-tenancy in the context of Kafka and storm?

Question

What are the best practices, for building a multi-tenant app in the context of Kafka and storm?

For example: creating topic for each tenant and consume multi-topics spout (using wildcard).

Answer 1

I think that topic for each tenant is the right choice.

The naming convention might be something like this: topic_base_name_tenant_id.

The reasons are:

1. It allows flexible configuration for each tenant (like @Sebastian mentioned earlier).
2. Clearer logical separation.

Now let's say that we will use different approach. For example, partition for each tenant. It might be problematic, since:

1. You are limiting the parallelism level to the number of tenants.
2. Adding new tenants, results adding new partition → republish old messages (The default partitioning algorithm is: message_key % partition_size).

Answer 2

Don't consider my answer as the "best practice" because I'm not Kafka expert, but in case it helps: we faced a similar design scenario (one Kafka endpoint available to multiple teams that acts as a central point for event creation and routing). After pondering different alternatives, we decided what you suggest: to have one topic per application (you could say tenant), customizing the topic (partition, limits, etc.) according to the application needs.

Hope this helps.

Answer 3

Sometimes you may need multiple topics per application. In that case you could follow the following topic naming convention: topic_base_name_tenant_id. So for a given base topic, you will have as many topics as the number of tenants