How can I get around the Azure Queue maximum time to live?

Question

Azure Message Queue messages have a maximum time to live of 7 days. Ugh why? I'd like my message to have an infinite time to live and wait in the queue until I get around to processing it. If not for this bizarre 7 day limit Azure queues would be a perfect solution for me. I have a whole 100TB storage account backing my Queue, why can't I use it?

I'm hoping someone has an idea for a workaround or solution to this problem. Any ideas?

Answer 1

You can now opt-in to infinite TTL by specifying an expiry of -1 seconds when you queue the message. This is new with api-version 2017-07-29:

For the Queue service, the Put Message API now allows a time-to-live value in the messagettl parameter of over seven days. You may also specify -1 for this parameter to indicate that the message should remain in the queue until dequeued and deleted. The default value for this parameter is still seven days.

If you're stuck with an earlier version of the Azure Storage libraries, I solved this problem by requeuing old messages - once they were 6 days old, I added a duplicate to the queue and deleted the original.

Answer 2

Service Bus Queues give you unlimited TTL as Brian points out.

You can get over the 5GB limit by sharding across multiple SB Queues. What we find, though, is that people who hit that limit will commonly find and approach more useful and cost effective whereby whatever large data items they have go into Storage (i.e. Blobs) and only the jobs associated with those go into the Queue.

That will allow you to leverage storage's ability to do block-wise uploads (with block blobs) and then communicate the presence of that blob through an SB queue. Once you've processed the job you can then remove the data.

There aren't a lot of common use-cases where a Queue will exceed the 5GB cap and that sort of pattern isn't the overall better choice.

Answer 3

You might look at Service Bus Queues. I'm not super familiar with them but in some cases they are more flexible than Storage Queues. You could also look at other queueing systems (MSMQ, RabbitMQ, etc) but they are probably not worth the effort to set up in Azure.

Another option is to use a Storage Table as a Queue. You can then support any TTL you like. There's some discussion about doing that here. I did actually implement that system and it works fine. If I get a chance I'll try to post the code.

Answer 4

I think the 7 day limit you are referring to is the visibility delay (i.e. time that the message will be invisible)

For time-to-live (specifies how long a message will remain in the queue) there is no documented limit that I aware of ( please point me to a link if this is not the case)

I am basing my comment on the following page http://msdn.microsoft.com/en-us/library/windowsazure/hh563575.aspx

Hope this helps