Python shared file sharing file in memory

Question

I am implementing a multiprocessor program in python, and for each of the subprocesses they should all read part of the file.

Since reading a file from disk is expensive, I want to read it only once and put it into shared memory.

1. If I use mmap, it can work with fork, but I can’t find a way to exchange the mmaped file between processes in the multiprocessing module.

2. If I read in a file in str and saved the line in sharedctypes.RawArray ('c', str), an error may occur if str has \ 0, the RawArray generated is truncation of the file.

Any idea?

Answer 1

Can you use multiprocessing managers? Make the mmped file an attribute of the NameSpace object returned by the Namespace() function, and pass a link to this for each of the processes.

 from multiprocessing import Manager mgr = Manager() ns = mgr.Namespace() ns.df = my_dataframe # now just give your processes access to ns, ie most simply # p = Process(target=worker, args=(ns, work_unit)) 

(my answer is mostly copied from here )