Concurrency parallel task with dependencies in Python like GNU Make

Question

I'm looking for a method or perhaps a philosophical approach to how to do something like GNU Make inside python. We are currently using makefiles to do the processing, because makefiles are very good at parallel runs with one option changed: -j x. In addition, gnu make already has dependency stacks built into it, so adding a secondary processor or being able to handle more threads just means updating this single option. I want the same power and flexibility in python, but I don't see it.

As an example:

all: dependency_a dependency_b dependency_c dependency_a: dependency_d stuff dependency_b: dependency_d stuff dependency_c: dependency_e stuff dependency_d: dependency_f stuff dependency_e: stuff dependency_f: stuff 

If we perform the standard single-stream operation (-j 1), the order of work may be:

 dependency_f -> dependency_d -> dependency_a -> dependency_b -> dependency_e \ -> dependency_c 

For two streams (-j 2) we can see:

 1: dependency_f -> dependency_d -> dependency_a -> dependency_b 2: dependency_e -> dependency_c 

Does anyone have any suggestions for an already created package or approach? I am completely open if this is a pythonic solution / approach.

Please in advance!

Answer 1

You might want to watch jug . This is a task-based parallelization framework that includes dependency tracking.

Answer 2

Take a look at Waf as well, it's less complicated than Scons.

Waf is a Python-based framework for customizing, compiling, and installing an Application. Here are perhaps the most important features of Waf:

Automatic assembly order: the assembly order is calculated from input and output files, including automatic dependencies: tasks to be performed by hashing files and commands Performance: tasks are executed in parallel automatically Flexibility: new commands can be added very easily through subclasses Features: support for many programming languages ​​and compilers are included by default Documentation: application based on a reliable model, Waf Book and API documents Python support: from Python 2.4 to 3.2 (Jython 2.5 and PyPy also supported)

(from the website)

Answer 3

You should use Scons as it already does the calculations you want, and you can undermine it to do just about anything (e.g. Make).

Answer 4

Take a look at the Scons . This is a replacement for GNU Make, written in Python.

SCons is a software design tool, that is, an excellent alternative to the classic "Make" build tool, which we all know and love.

SCons is implemented as a Python script and a set of modules, while SCons "configuration files" are actually executed as Python scripts. This gives SCons many powerful features, rather than those found in other software build tools.

Answer 5

redo -j

"Smaller, simpler, more powerful and more reliable than done. Implementing djb redo " in Python.