Python function mapper display function error

Question

I have a simple multiprocessing example that I am trying to create. The normal version of the map () function works, but when it changes to Pool.map, I get a strange error:

from multiprocessing import Pool from functools import partial x = [1,2,3] y = 10 f = lambda x,y: x**2+y # ordinary map works: map(partial(f,y=y),x) # [11, 14, 19] # multiprocessing map does not p = Pool(4) p.map(partial(f, y=y), x) Exception in thread Thread-2: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 504, in run self.__target(*self.__args, **self.__kwargs) File "/usr/lib/python2.7/multiprocessing/pool.py", line 319, in _handle_tasks put(task) PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed 

Etching Error? What it is?

Answer 1

Pool.map arguments must be legible. The module level functions are distinguished , but partial(f, y=y) not defined at the module level and therefore is not selected.

There is a workaround:

 def g(x, y=y): return f(x, y) p.map(g, x) 

---

Functions made with functools.partial were not available . However, with Python2.7 or higher, you can also define g (at the module level) with functools.partial:

 import multiprocessing as mp import functools def f(x, y): return x**2 + y x = [1,2,3] y = 10 g = functools.partial(f, y=y) if __name__ == '__main__': p = mp.Pool() print(p.map(g, x)) 

gives [11, 14, 19] . But note, to get this result, f had to be defined using def , not lambda . I think this is because pickle relies on "fully qualified" name references to look up object object values.