How can I get around the maximum life time in Azure Queue?

Question

Azure Message Queuing messages have a maximum lifetime of 7 days. Why? I want my message to have endless time to live and wait in line until I get it processed. If not for this weird 7-day Azure limit, this would be the perfect solution for me. I have a whole 100 TB account supporting my queue, why can't I use it?

I hope someone has the idea of ​​a workaround or a solution to this problem. Any ideas?

Answer 1

Now you can opt out of an infinite TTL by specifying the expiration of -1 seconds when sending a message to the queue. This is the new version of the api version 2017-07-29:

For the Queue service, the Post Message API now allows a lifetime value in the messagettl parameter of more than seven days. You can also specify -1 for this parameter to indicate that the message should remain in the queue until deleted and deleted. The default value for this parameter is still seven days.

If you are stuck with an earlier version of Azure Storage libraries, I solved this problem by overwriting the old messages - as soon as they were 6 days old, I added a duplicate to the queue and deleted the original.

Answer 2

Service bus queues give you unlimited TTL, as Brian points out.

You can overcome the 5 GB limit by going through several SB queues. However, we find that people who fall within this limit will usually find and approach more useful and economical ones, so any large data items that they send to the repository (i.e. Blobs), and only related jobs enter the queue.

This will allow you to use the storage capabilities for block loading (with block blobs), and then report the presence of this blob through the SB queue. After processing the job, you can delete the data.

There are not many common use cases where the queue will exceed 5 GB and this type of template is not the best choice.

Answer 3

You can see the service bus queues. I am not very familiar with them, but in some cases they are more flexible than storage queues. You can also look at other queue systems (MSMQ, RabbitMQ, etc.), but they are probably not worth the effort to configure in Azure.

Another option is to use the storage table as a queue. Then you can support any TTL that you like. There is a discussion of this issue . I really implemented this system and it works great. If I get a chance, I will try to publish the code.

Answer 4

I think the 7-day limit you are referring to is a delay in visibility (i.e. the time that the message will be invisible)

For the lifetime (it is indicated how long the message will remain in the queue), I do not know the document limit (provide me a link if this is not so)

I base my comment on the next page http://msdn.microsoft.com/en-us/library/windowsazure/hh563575.aspx

Hope this helps