Celery - Schedule a periodic task at the end of another task

Question

I want to schedule a periodic task with Celery dynamically at the end of another task group.

I know how to create (static) periodic jobs with Celery:

CELERYBEAT_SCHEDULE = { 'poll_actions': { 'task': 'tasks.poll_actions', 'schedule': timedelta(seconds=5) } } 

But I want to dynamically create periodic tasks from my tasks (and, perhaps, I have the opportunity to stop these periodic tasks when some condition is achieved (all tasks are completed).

Sort of:

 @celery.task def run(ids): group(prepare.s(id) for id in ids) | execute.s(ids) | poll.s(ids, schedule=timedelta(seconds=5)) @celery.task def prepare(id): ... @celery.task def execute(id): ... @celery.task def poll(ids): # This task has to be schedulable on demand ... 

Answer 1

A simple solution to this requires that you can add / remove bit scheduler entries. Regarding the answer to this question ...

How to dynamically add / remove periodic celery tasks (celerybeat)

It was impossible. I doubt that it became available in the interim, because ...

Here you combine two concepts. The concept of "Event Driven Work" and the idea of ​​"Batch Schedule Driven Work" (this is really the first time that an event occurs on a schedule). If you really think what you are doing here, you will find that there is a rather complicated set of edge cases. Messages are distributed by nature, what happens when groups generated from different messages start to create conflicting records? What do you do when you are under the mountains of the previously planned kruft?

When working with messaging systems, you really want to create recursive trees. Work spindles that do something and spawn more messages to do more things. Cycles (intended or otherwise) away from them ultimately reach the base cases and end.

The answer to what you are actually trying to achieve is to re-encode your problem within the limitations of your messaging system and asynchronous work environment.