How to set up different worker pools using celery?

Question

I need to deploy a queue service in a queue with (usually) the following three worker classes:

• Periodic, low priority work class that takes a lot of time and can be processed sequentially; these jobs should only use 0.2 workers in the system.

• A deadline-sensitive period class that takes a short or medium time (for example, up to 5 minutes)

• The ad-hoc class, which is a higher priority than C # 2, but can alternate with C # 2. Any workers from class 2 who are inactive when this type of work arrives should cope with this without even starving the workers pool for # 2

All three job classes are the same task, the only difference between them is how they are requested; they will take the same input and generate the same output, but each has different performance guarantees.

How can I realize this with celery?

Answer 1

You can use Celery Routing to achieve your goal. What you do is set up three different routing keys and have several celery workers (with customizable settings) listing only specific routing keys.