How many queue for multiple instances of a azure worker role?

Question

I have a question.

I thought that Microsoft's Azure Queue was design to support a lot of messages as suggested by this post and I read you are able to store 100TB of messages (capacity max of an account). So you can use only one queue for all a worker role.

But I found an article which shows us only a queue by worker role instance.

I think the rest of the article is serious so I don't know what think about this.

Are multiple queues better in some cases?

Answer 1

If you're looking at storage it doesn't matter if you have 1 queue or 100, the limits are the same. But you have to know the performance implications. In a storage account performance is based on partitions (the Fabric Controller is able to assign more resources to a hot partition for example).

When you're working with Windows Azure Storage Queues, 1 queue is 1 partition. And this comes with some scalablity targets. For queues this means that 1 queue should be able to handle 500 messages / second. Read more about it here: Windows Azure Storage Abstractions and their Scalability Targets

Now this is an important factor for when you're defining your architecture and it depends on what you're going to use the queues for. If you never expect to reach 500 messages / second you should be fine with one queue for all your workers. But if you think to exceed this limit (even if it won't happen right away) you'll need to set up a good architecture to handle this. And this doesn't simply mean 1 queue for 1 worker. Think about different queues for different processes, splitting up processes in sub processes and use queues per subprocess, have your workers poll different queues, think of a round robin system, ...

Also consider looking at an existing implementation like this one: Partitioned CloudQueue - Azure Storage Queue without scalability limits. I also suggest you look at the website of the Customer Advisory Team, they have a number of interesting articles regarding queue scalablity (the first article you referenced comes from the CAT website).