Azure webjob showing no respect for MaxDequeueCount property

Question

I have an Azure website with several queued features. SDK documentation at https://docs.microsoft.com/en-us/azure/app-service-web/websites-dotnet-webjobs-sdk-storage-queues-how-to#config defines the MaxDequeueCount property as:

The maximum number of attempts before a queue message is sent to the queue poison (default is 5).

but I do not see such behavior. In my weblog I have:

 JobHostConfiguration config = new JobHostConfiguration(); config.Queues.MaxDequeueCount = 1; JobHost host = new JobHost(config); host.RunAndBlock(); 

and then I have a function with a running queue in which I throw an exception:

 public void ProcessQueueMessage([QueueTrigger("azurewejobtestingqueue")] string item, TextWriter logger) { if ( item == "exception" ) { throw new Exception(); } } 

Looking at the webjobs toolbar, I see that the SDK makes 5 attempts (the default is 5):

and after the 5th attempt, the message moves to the poison queue. I expect to see 1 repeat (or no attempts?) Not 5.

UPDATE: Detailed logging is enabled for the web application and the option to save these logs to the Azure blob container is selected. Found several logs related to my problem located in the azure-jobs-host-archive container. Here's an example showing an element with a decompression number of 96:

 { "Type": "FunctionCompleted", "EndTime": "2017-02-22T00:07:40.8133081+00:00", "Failure": { "ExceptionType": "Microsoft.Azure.WebJobs.Host.FunctionInvocationException", "ExceptionDetails": "Microsoft.Azure.WebJobs.Host.FunctionInvocationException: Exception while executing function: ItemProcessor.ProcessQueueMessage ---> MyApp.Exceptions.MySpecialAppExceptionType: Exception of type 'MyApp.Exceptions.MySpecialAppExceptionType' was thrown. }, "ParameterLogs": {}, "FunctionInstanceId": "1ffac7b0-1290-4343-8ee1-2af0d39ae2c9", "Function": { "Id": "MyApp.Processors.ItemProcessor.ProcessQueueMessage", "FullName": "MyApp.Processors.ItemProcessor.ProcessQueueMessage", "ShortName": "ItemProcessor.ProcessQueueMessage", "Parameters": [ { "Type": "QueueTrigger", "AccountName": "MyStorageAccount", "QueueName": "stuff-processor", "Name": "sourceFeedItemQueueItem" }, { "Type": "BindingData", "Name": "dequeueCount" }, { "Type": "ParameterDescriptor", "Name": "logger" } ] }, "Arguments": { "sourceFeedItemQueueItem": "{\"SourceFeedUpdateID\":437530,\"PodcastFeedID\":\"2d48D2sf2\"}", "dequeueCount": "96", "logger": null }, "Reason": "AutomaticTrigger", "ReasonDetails": "New queue message detected on 'stuff-processor'.", "StartTime": "2017-02-22T00:07:40.6017341+00:00", "OutputBlob": { "ContainerName": "azure-webjobs-hosts", "BlobName": "output-logs/1ffd3c7b012c043438ed12af0d39ae2c9.txt" }, "ParameterLogBlob": { "ContainerName": "azure-webjobs-hosts", "BlobName": "output-logs/1cf2c1b012sa0d3438ee12daf0d39ae2c9.params.txt" }, "LogLevel": "Info", "HostInstanceId": "d1825bdb-d92a-4657-81a4-36253e01ea5e", "HostDisplayName": "ItemProcessor", "SharedQueueName": "azure-webjobs-host-490daea03c70316f8aa2509438afe8ef", "InstanceQueueName": "azure-webjobs-host-d18252sdbd92a4657d1a436253e01ea5e", "Heartbeat": { "SharedContainerName": "azure-webjobs-hosts", "SharedDirectoryName": "heartbeats/490baea03cfdfd0416f8aa25aqr438afe8ef", "InstanceBlobName": "zd1825bdbdsdgga465781a436q53e01ea5e", "ExpirationInSeconds": 45 }, "WebJobRunIdentifier": { "WebSiteName": "myappengine", "JobType": "Continuous", "JobName": "ItemProcessor", "RunId": "" } } 

What I'm still looking for is logs that show me the details for a particular queue element, where processing succeeds (and therefore is removed from the queue) or fails due to an exception and is put in the poison queue. I still have not found any magazines showing this detail. Log files referenced by the above output do not contain this type of data.

UPDATE 2: Looked at the state of my poisonous lineup, and it looks like it might be a smoking gun, but I'm too tight to put 2 and 2 together. Looking at the screenshot of the queue below, you can see the message with the identifier (left column) 431210 there many times. The fact that it appears several times tells me that the message in the original queue does not work correctly.

Answer 1

As Rob W mentioned, this problem exists when using WindowsAzure.Storage> 7.1.2. The issue was apparently fixed in issue # 1141 , but this has not yet reached the release.

Contributor asifferman shared a snippet of code in the comments on question No. 985 . which seems to solve the problem (it worked fine for me).

In the case of the rot link and to comply with SO rules, here's the message along with a code snippet:

For those (like me) who can't wait for the next version to get the WebJobs SDK to work with the latest versions of Azure Storage and based on @brettsam's explanation, you can just write the CustomQueueProcessorFactory custom to create a new CloudQueueMessage in CopyMessageToPoisonQueueAsync.

 namespace ConsoleApplication1 { using Microsoft.Azure.WebJobs.Host.Queues; using Microsoft.WindowsAzure.Storage.Queue; using System.Threading; using System.Threading.Tasks; public class CustomQueueProcessorFactory : IQueueProcessorFactory { public QueueProcessor Create(QueueProcessorFactoryContext context) { return new CustomQueueProcessor(context); } private class CustomQueueProcessor : QueueProcessor { public CustomQueueProcessor(QueueProcessorFactoryContext context) : base(context) { } protected override Task CopyMessageToPoisonQueueAsync(CloudQueueMessage message, CloudQueue poisonQueue, CancellationToken cancellationToken) { var newMessage = new CloudQueueMessage(message.Id, message.PopReceipt); newMessage.SetMessageContent(message.AsBytes); return base.CopyMessageToPoisonQueueAsync(newMessage, poisonQueue, cancellationToken); } } } } 

Then in the main folder, you just need to configure your own factory queue processor in the job host configuration:

 var config = new JobHostConfiguration(); config.Queues.QueueProcessorFactory = new CustomQueueProcessorFactory(); 

I could get it to work with WindowsAzure.Storage 8.1.1 and Microsoft.Azure.WebJobs 2.0.0. Hope this helps!

Answer 2

If you are still looking for an answer, we tried some of the answers given without success. It turns out that it was a problem with the version of Storage sdk (WindowsAzure.Storage) and Webjob sdk (Microsoft.Azure.WebJobs). To fix this, we had to redefine our version of Storage sdk to 7.2.1 (we recently upgraded to 8.1.1). Based on the article below, engineers are now aware of the problems and hopefully fix this soon:

https://github.com/Azure/azure-webjobs-sdk/issues/1045

Answer 3

The MaxDequeueCount property works correctly for me if I configure it.

So it is very strange that it does not work for you. When I set config.Queues.MaxDequeueCount = 2; then I get the expected result, please refer to the screenshot.

And we could also use dequeueCount to control repeat time. Below is a demo code without any attempt.

 public void ProcessQueueMessage([QueueTrigger("queue")] string item, int dequeueCount, TextWriter logger) { if (dequeueCount == 1) { if (item == "exception") { throw new Exception(); } logger.WriteLine($"NewMsge: {item}"); Console.WriteLine($"NewMsge: {item}"); } } 

See the screenshot for log information.

Answer 4

I suspect this because you do not actually manage the binaries that you think are in Azure. This one also threw me on a noose.

When you start running WebJobs on Azure, publishing a new version of WebJob does not immediately load the old running WebJob and start a new one. If you look at your WebJob logs, I suspect you won't see a restart when you republish.

This is because Kudu, by default, copies all of your WebJob files to the temp directory and executes them. From Kudu WebJob Docs :

WebJob is copied to the temporary directory in the% TEMP% \ jobs {job type} {job name} {random name} section and will be launched from there. This option prevents WebJob source binaries from being blocked, which may cause WebJob to redeploy. For example, updating the .exe file that is currently running.

The only success I came across to make sure the recently published launched WebJob actually works is this:

• Log in to the Kudu console. This is https://yourappname.scm.azurewebsites.net . You will use the same credentials as when logging into Azure Portal.

• After logging in, click the "Explorer" button at the top of the window. Find the current WebJob process and kill it.

• FTP to your web application. Go to the directory containing your WebJob code and delete it. It should be under / app_data / jobs / triggered / [your website name].

• Then I go to the portal, look at the web application management blade that hosts the web application, click on the WebJobs menu item and confirm that the old website is no longer there.

• Publish the new WebJob from Visual Studio.

This should ensure that you use the code that you publish. Hope this helps.

Answer 5

I see the same thing that messages go past the maximum number of dequeue. I will tell you more details a little, but I also see that a very large number seems to fall into the poisonous queue. Therefore, I suspect that he adds to the poison queue after 5, but tries more than it ends in the poison queue (hundreds).