Python dependency batch processing

Question

I am looking for the best way to create a task scheduler for different types of tasks. CJobs are streaming, and some tasks must be completed before the next step in this process can be completed. Currently, all this is managed through a database table ... which, I think, is in order. But if there is a better way to manage addictions, I'm all ears.

Preferably, I would like to do this in python. I see there a parallel python module that looks great, but I'm worried about this dependency problem between jobs.

Can someone recommend something that does what I need to do or how to do it?

Thank you very much!

D

UPDATE: this should be done on a server cluster with a limited set of available workers ... 1 per port. Does it support celery or SCON?

Answer 1

I had great success with Celery

Answer 2

Luigi looks very interesting. It allows you to create workflows - sets of related jobs that are managed by Luigi. It also has a simple web interface that provides a dependency graph.

Answer 3

SCons may be useful for this.

It is biased towards software development (compilation, binding, etc.), but you can easily define new result classes, new commands, and new source classes so that it processes your data (and dependencies) correctly.

Based on the update, you probably need something like BuiltBot .