Lock Azure Queue Processing in an Azure Function Application

Question

I created an Azure Function application with an Azure Storage queue trigger that processes a queue in which each queue item is a URL. The function simply loads the contents of the URL. I have another function that downloads and parses an XML site file and adds all the page URLs to the queue. The problem is that the Functions application starts too fast and it clogs the website to return Server Errors. Is there a way to limit / decrease the speed at which the functions application works?

I could, of course, write a simple web task that processed them in series (or with some asynchronous, but limited the number of simultaneous requests), but I really liked the simplicity of Azure Functions and wanted to try "serverless" computing.

Answer 1

There are several options you can consider.

-, , host.json, ( ). queues.batchSize - . 1, , , . .

NextVisibleTime , , - , , .

, URL- , , , , URL- parallelism .

Answer 2

NextVisibleTime , . , : , "throttled-items", . , , NextVisibleTime.

    [FunctionName("ThrottleQueueItems")]
    public static async Task Run([TimerTrigger("0 * * * * *")] TimerInfo timer, ILogger logger)
    {
        var originalQueue = // get original queue here;
        var throttledQueue = // get throttled queue here;
        var itemsPerMinute = 60; // get from app settings
        var individualDelay = 60.0 / itemsPerMinute;
        var totalRetrieved = 0;
        var maxItemsInBatch = 32; // change if you modify the default queue config
        do
        {
            var pending = (await originalQueue.GetMessagesAsync(Math.Min(maxItemsInBatch, htmlPerMinute - totalRetrieved))).ToArray();
            if (!pending.Any())
                break;
            foreach (var message in pending)
            {
                await throttledQueue.AddMessageAsync(new CloudQueueMessage(message.AsString), null,
                                                                                        TimeSpan.FromSeconds(individualDelay * ++totalRetrieved), null, null);
                await originalQueue.DeleteMessageAsync(message);
            }
        } while (htmlPerMinute > totalRetrieved);
    }