Python multiprocessing lock issue

Question

I want to add a list of dicts along with the python multiprocessing module.

Here is a simplified version of my code:

#!/usr/bin/python2.7 # -*- coding: utf-8 -*- import multiprocessing import functools import time def merge(lock, d1, d2): time.sleep(5) # some time consuming stuffs with lock: for key in d2.keys(): if d1.has_key(key): d1[key] += d2[key] else: d1[key] = d2[key] l = [{ x % 10 : x } for x in range(10000)] lock = multiprocessing.Lock() d = multiprocessing.Manager().dict() partial_merge = functools.partial(merge, d1 = d, lock = lock) pool_size = multiprocessing.cpu_count() pool = multiprocessing.Pool(processes = pool_size) pool.map(partial_merge, l) pool.close() pool.join() print d 

• I get this error when running this script. How can i solve this?

  RuntimeError: Lock objects should only be shared between processes through inheritance

• is the lock in merge function needed in this condition? or will python take care of this?

• I think that map should do something from one list to another list, and not dump all things in one list to one object. So is there a more elegant way to do such things?

Answer 1

Next, you should use cross-platform (that is, on Windows) both in Python 2 and 3. It uses a process pool initializer to set the dict manager as global in each child process.

FYI:

• Using locks is not required with the dict manager.
• The number of processes in Pool by default corresponds to the number of CPUs.
• If you are not interested in the result, you can use apply_async instead of map .

 import multiprocessing import time def merge(d2): time.sleep(1) # some time consuming stuffs for key in d2.keys(): if key in d1: d1[key] += d2[key] else: d1[key] = d2[key] def init(d): global d1 d1 = d if __name__ == '__main__': d1 = multiprocessing.Manager().dict() pool = multiprocessing.Pool(initializer=init, initargs=(d1, )) l = [{ x % 5 : x } for x in range(10)] for item in l: pool.apply_async(merge, (item,)) pool.close() pool.join() print(l) print(d1)