I'm new to Celery / AMQP and trying to come up with a task / queue / worker project to meet the following requirements.
I have several types of tasks "for the user": for example, TaskA, TaskB, TaskC. Each of these per-user tasks reads / writes data for one specific user in the system. Therefore, at any time, I may need to create tasks User1_TaskA, User1_TaskB, User1_TaskC, User2_TaskA, User2_TaskB, etc. I need to make sure that for each user there are no two tasks, any tasks the type is running at the same time. I want a system in which no worker can execute User1_TaskA at the same time that the user is executing another user, User1_TaskB or User1_TaskC, but when executing User1_TaskA, other workers should not block the simultaneous execution of User2_TaskA, User3_TaskA, etc.
I understand that this can be implemented using some external locking mechanism (for example, in the database), but I hope that a more elegant project / queue / working project will work.
I believe that one of the possible solutions is to implement the queues in the form of custom buckets, so when you run the working commands, it will indicate how many buckets will be created, and each "desktop" is tied to one bucket. Then the "intermediate worker" removed tasks from the main task queue and assigned them to the queue with the slave code through, say, a hash / mod scheme. Thus, UserA tasks always end up in the same queue, and several tasks for UserA will support each other. I do not like this approach, since it will require a certain number of buckets to be determined ahead of time, and it does not seem to allow (easily) adding workers dynamically. It seems that there should be a better way - suggestions would be greatly appreciated.
source share