How to make celery try to use the same worker?

Question

I am just starting with celery in a Django project, and I am a bit stuck with this particular problem: Basically, I need to distribute a long-term task for different workers. The task is actually divided into several stages, each of which requires considerable time to complete. Therefore, if any step failed, I would like celery to repeat this task using the same employee to reuse the results from the completed steps. I understand that celery uses routing to distribute tasks on a specific server, but I cannot find anything about this specific problem. I use RabbitMQ as my broker.

Answer 1

You can use each celeryd instance from the queue named after the worker host name:

celeryd -l info -n worker1.example.com -Q celery,worker1.example.com 

sets the hostname worker1.example.com and will be consumed from the queue with the same name, as well as by default (with the name celery ).

Then, to direct the task to a specific employee, you can use:

 task.apply_async(args, kwargs, queue="worker1.example.com") 

Similarly, to send a repeat:

 task.retry(queue="worker1.example.com") 

or redirect the repetition of one employee:

 task.retry(queue=task.request.hostname)