Tasks targeting a dynamic backend often fail silently

Question

I converted some tasks to work on a dynamic backend.

Tasks fail [without errors in the journal, without repetition, nothing] ~ 20% of the time (min: 10%, maximum: 60%, sample: large, long-term). Disconnecting a task from the backend restores retries and returns a failure rate of up to ~ 0%.

Any ideas?

Answer 1

Converting it to the backend exacerbated the problem, but it is not a problem.

I specified task_retry_limit and the queue was a point queue. The database indicates the number of instances. (I believe that you can repeat this problem in the interface, quickly increasing the number of requests, to a large number).

Tasks failed 503: Instance Unavailable until they reached task_retry_limit . This is temporarily displayed in the Task Queue, but not displayed in the logs.

I have to use exhaust queues. Even if my use case was stupid, I might have +1 lost the task due to multiple 503: Instance Unavailable logging so that it doesn't look like a phantom task.

Answer 2

What working environment do you use on the backend? Try to run the backend for bits without dynamic dialing to true and execute the failed component.

In my project, I saw tasks that target a static backend, sometimes disappear, but no where next to the one you see.