Perform many functions in parallel and collect all the results in a list

Question

I have a very intense processor load:

def entity_intersections(ent, collidable): intersections = [] for line1, line2 in product(ent.shape, collidable.shape): pair_intersections = find_intersections(line1 + ent.position, ent.velocity, ent.acceleration, line2 + collidable.position, collidable.velocity, collidable.acceleration, ent, collidable) intersections.extend(pair_intersections) return intersections 

I want all calls on find_intersections to find_intersections executed in parallel, so that they run faster, and collect the results together (after all the executions have completed). Which library would allow me to do this, given that find_intersections is a pure function ?

An indication of how to generate these parallel executions, as well as bring together the results, would be greatly appreciated.

Answer 1

The easiest way is to use the multiprocessing module:

 class FindIntersectionsWrapper(object): def __init__(self, ent, collidable): self.ent = ent self.collidable = collidable def __call__(self, dims): line1, line2 = dims return find_intersections( line1 + self.ent.position, self.ent.velocity, self.ent.acceleration, line2 + self.collidable.position, self.collidable.velocity, self.collidable.acceleration, self.ent, self.collidable) def entity_intersections(ent, collidable): find_inter = FindIntersectionsWrapper(ent, collidable) pool = multiprocessing.Pool() return pool.map(find_inter, product(ent.shape, collidable.shape)) 

The helper function find_intersections_wrapper() needed because Pool.map() expects a function with a single argument.

You might want to migrate the pool creation from entity_intersections() to have the overhead of creating a process pool only once.

Change The class is used instead of closing, since the caller passed to Pool.map() must be selected on Windows.

Answer 2

You should use threading class

http://docs.python.org/library/threading.html#module-threading

There are some good examples on this page: http://www.tutorialspoint.com/python/python_multithreading.htm