EDIT: I confirmed that this is a bug in Python. This is an error http://bugs.python.org/issue10332 (I filed a new error, in response to which the attendant pointed me to 10332). I copied the multiprocessing directory from the original Python repository to the project directory, and the test file now works correctly.
This seemingly simple program does not work for me unless I remove the maxtasksperchild parameter. What am I doing wrong?
from multiprocessing import Pool import os import sys def f(x): print "pid: ", os.getpid(), " got: ", x sys.stdout.flush() return [x, x+1] def cb(r): print "got result: ", r if __name__ == '__main__': pool = Pool(processes=1, maxtasksperchild=9) keys = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] result = pool.map_async(f, keys, chunksize=1, callback=cb) pool.close() pool.join()
When I run it, I get:
$ python doit.py pid: 6409 got: 1 pid: 6409 got: 2 pid: 6409 got: 3 pid: 6409 got: 4 pid: 6409 got: 5 pid: 6409 got: 6 pid: 6409 got: 7 pid: 6409 got: 8 pid: 6409 got: 9
And it freezes. That is, a new worker for processing the 10th element is not generated.
In another terminal, I see:
$ ps -C python PID TTY TIME CMD 6408 pts/11 00:00:00 python 6409 pts/11 00:00:00 python <defunct>
This is done on Ubuntu 11.10, where python 2.7.2+ is running (installed from ubuntu packages).