How to limit the maximum number of celery tasks performed by name

Question

How do you limit the number of instances of a specific Celery task that can run simultaneously?

I have a task processing large files. I ran into a problem where a user can run several tasks, which is why the server runs out of processor and memory, because it is trying to process too many files at once. I want only N instances of this one type of task to be executed at any given time, and the rest of the tasks will be queued in the scheduler until the rest are complete.

I see the rate_limit option there in the task decorator, but I don’t think it does what I want. If I understand the documents correctly, this will simply limit the speed of launching tasks, but it will not limit the total number of running tasks, so it will make my server fail more slowly ... but it will still be, but nonetheless.

Answer 1

What you can do is move these tasks to a specific queue and have X the number of workers processing them. The presence of two workers in a queue with 100 elements ensures that only two tasks will be performed at a time.

Answer 2

, Celery, , , , , , ( , , .

, .

:

app = Celery(...)
i = app.control.inspect()
i.active()