Python multiprocessing and numpy shared array

Question

I have a problem that looks like this:

import numpy as np C = np.zeros((100,10)) for i in range(10): C_sub = get_sub_matrix_C(i, other_args) # shape 10x10 C[i*10:(i+1)*10,:10] = C_sub 

Thus, apparently, it is not necessary to run this as a sequential calculation, since each submatrix can be calculated independently. I would like to use the multiprocessing module and create up to 4 processes for the for loop. I read several multiprocessing guides, but couldn't figure out how to use this to solve my problem.

thanks for the help

Answer 1

An easy way to parallelize this code is to use Pool processes:

 pool = multiprocessing.Pool() results = pool.starmap(get_sub_matrix_C, ((i, other_args) for i in range(10))) for i, res in enumerate(results): C[i*10:(i+1)*10,:10] = res 

I used starmap since the get_sub_matrix_C function has more than one argument ( starmap(f, [(x1, ..., xN)]) calls f(x1, ..., xN) ).

Please note, however, that serialization / deserialization can take considerable time and space, so you may need to use a lower-level solution to avoid this overhead.

---

It looks like you are using an outdated version of python. You can replace starmap with a regular map , but then you must provide a function that takes one parameter:

 def f(args): return get_sub_matrix_C(*args) pool = multiprocessing.Pool() results = pool.map(f, ((i, other_args) for i in range(10))) for i, res in enumerate(results): C[i*10:(i+1)*10,:10] = res 

Answer 2

The following recipe may possibly do the job. Feel free to ask.

 import numpy as np import multiprocessing def processParallel(): def own_process(i, other_args, out_queue): C_sub = get_sub_matrix_C(i, other_args) out_queue.put(C_sub) sub_matrices_list = [] out_queue = multiprocessing.Queue() other_args = 0 for i in range(10): p = multiprocessing.Process( target=own_process, args=(i, other_args, out_queue)) procs.append(p) p.start() for i in range(10): sub_matrices_list.extend(out_queue.get()) for p in procs: p.join() return sub_matrices_list C = np.zeros((100,10)) result = processParallel() for i in range(10): C[i*10:(i+1)*10,:10] = result[i]