Parallel.For in Python

Question

Is there something similar to C # that is excellent Parallel.For in Python? I just want to make a calculation, for example

[simu(c) for c in clusterSizes] 

parallel. What is the easiest way to archive?

PS: I tried joblib, but in my cases it just starts, starts and starts processes until I restart my machine.

Answer 1

In python 3, there is a parallel mapping in concurrent.futures (in the standard library). I think it was even backported as a module for python 2.7. change http://pypi.python.org/pypi/futures

As noted in another answer, threads won't help. Instead, you need to use several processes.

editing from documents looks so simple:

 with concurrent.futures.ProcessPoolExecutor() as executor: for result in executor.map(simu, clusterSizes) pass # save result 

Answer 2

joblib is based on multiprocessing and has the same problem on Windows that @EwyynTomato points out in the comments: the main entry point to the script should be protected by checking __name__ == "__main__" .

 from joblib import Parallel, delayed if __name__ == "__main__": result = Parallel(n_jobs=-1)(delayed(simu)(c) for c in clusterSizes) # process result 

Answer 3

in the multiprocessing package, you can create a Pool object and use the map method:

see http://docs.python.org/library/multiprocessing.html

as with joblib, you must protect the main entry point with __name__ == "__main__" , otherwise you will create a fork (and you will have to restart your computer when importing your script).

The function you are configuring must be selectable.

Answer 4

I wrote a small package that could help:

 from python-parallel-collections import parallel #equal to [simu(c) for c in clusterSizes] #performed in parallel using multiple processes #the results are only evaluated on demand, such as with the call to list below parallel_gen = parallel(clusterSizes) lazy_results = parallel_gen.map(simu) evaluated_results = list(lazy_results) 

This is Python 2.7 and requires

 pip install futures pip install python-parallel-collections 

let me know what you think https://github.com/gterzian/Python-Parallel-Collections

Answer 5

Use a different language. Due to GIL (Global Interpreter Lock), non-IO parallelism in Python is not very efficient.