Python 2.7: error "do not start a new thread" from "multiprocessing.Pool"

Question

Here is my situation. The code is almost the same as for the example in the docs :

from multiprocessing import Pool
import numpy as np

def grad(x0, y): return 0 # does some computational-heavy work actually

if __name__ == '__main__':

    class UnrollArgs:
        def __init__(self, func):
            self.func = func

        def __call__(self, args):
            return self.func(*args)

    def batch_grad(x0, y, processes=4):
        g = Pool(processes).map(UnrollArgs(grad), [(x0, yi) for yi in y])
        return np.sum([gi for gi in g], axis=0) / len(y)

yI go to batch_gradhas 50 elements and Pool.mapgives an error:

Error: Cannot start a new thread.

From Google I know that this is usually caused by the fact that you are trying to start too many threads. Maybe it's just me, but I think the documentation is a multiprocessing.Poolbit incomplete. In particular, I do not understand how to control the number of threads that need to be started. The term "stream" is not even mentioned in the class documentation Pool.

The integral argument multiprocessing.Poolis the number of running processes , not threads.

, ?

: , , .

Answer 1

, Pool s. , , @ChongMa , Python, . , , .

: a) Pool.close(), Pool , :

def batch_grad(x0, y, processes=4):
    pool = Pool(processes)
    g = pool.map(UnrollArgs(grad), [(x0, yi) for yi in y])
    pool.close()
    return np.sum([gi for gi in g], axis=0) / len(y)

b) Pool - Pool batch_grad :

def batch_grad(x0, y, pool=None):
    if pool is None:
        pool = Pool(4)
    g = pool.map(UnrollArgs(grad), [(x0, yi) for yi in y])
    return np.sum([gi for gi in g], axis=0) / len(y)

# then call your function like so
p = Pool(4)
batch_grad(your_x0, your_y, p)

, .