There is an efficient way to share a sparse read-only matrix in multiprocessing in python

Question

I find many similar questions, but do not answer. For a simple array, there is multiprocessing. For a sparse matrix or any other arbitrary object, I find manager.namespace. So I tried the code below:

from scipy import sparse from multiprocessing import Pool import multiprocessing import functools def myfunc(x,ns): return ns.A[x,:]*ns.A*ns.A[:,x] manager = multiprocessing.Manager() Global = manager.Namespace() pool=Pool() Global.A=sparse.rand(10000,10000,0.5,'csr') myfunc2=functools.partial(myfunc,ns=Global) r=pool.map(myfunc2, range(100)) 

The code works, but is not efficient. In fact, only 4 out of 16 workers work. The reason is that, I think, a manager allows only one worker to access data at a time. Since the data is read-only, I really don't need a lock. So is there a better way to do this?

ps, I saw people talking about copy-on-write fork (). I really don't understand what it is, but it does not work. If I first generate A and do Pool (), each process will have a copy of A.

Thanks in advance.

Answer 1

The namespace object property is updated only when it is explicitly assigned . Good explanations are given here .

Edit: And looking at the implementation (in multiprocessing/managers.py ), it looks like it is not using shared memory. He simply dries the objects and sends them to the child upon request. This is probably why it takes so long.

Do you accidentally create a pool with more workers than your processor has cores? (Ie using the processes argument of the Pool constructor). This is usually not a good idea.

There are several other things you can try:

• Write the sparse matrix to a file and let each workflow read the file. The OS is likely to put the file in the buffer cache, so the performance of this can be much better than you think.
• A possible improvement is the use of a memory mapped file using the mmap module.