The print function crashes the multiprocessing program

Question

In the following code, I try to create an isolated technical system in which changes in the global variables of a worker do not reflect other workers.

To achieve this, each new process is created each time a task is created, and for parallel execution, the creation of the processes themselves is controlled ThreadPoolExecutor.

import time
from concurrent.futures import ThreadPoolExecutor
from multiprocessing import Pipe, Process


def task(conn, arg):
  conn.send(arg * 2)


def isolate_fn(fn, arg):

  def wrapped():
    parent_conn, child_conn = Pipe()
    p = Process(target=fn, args=(child_conn, arg), daemon=True)
    try:
      p.start()
      r = parent_conn.recv()
    finally:
      p.join()
    return r

  return wrapped


def main():
  with ThreadPoolExecutor(max_workers=4) as executor:
    pair = []

    for i in range(0, 10):
      pair.append((i, executor.submit(isolate_fn(task, i))))

      # This function makes the program broken.
      # 
      print('foo')

    time.sleep(2)

    for arg, future in pair:
      if future.done():
        print('arg: {}, res: {}'.format(arg, future.result()))
      else:
        print('not finished: {}'.format(arg))

  print('finished')

main()

This program works fine until I turn on the function print('foo')inside the loop. If the function exists, some tasks remain incomplete, and worse, this program itself does not end.

The results are not always the same, but the following result is typical:

foo
foo
foo
foo
foo
foo
foo
foo
foo
foo
arg: 0, res: 0
arg: 1, res: 2
arg: 2, res: 4
not finished: 3
not finished: 4
not finished: 5
not finished: 6
not finished: 7
not finished: 8
not finished: 9

Why is this program so fragile?

I am using Python 3.4.5.

Answer 1

from multiprocessing import set_start_method

... rest of your code here ....

if __name__ == '__main__':
    set_start_method('spawn')
    main()

Stackoverflow python , , . ( 2.4 3.2 python)

, python multiprocessing.set_start_method. "spawn" "forkserver" - .

MultiProcessingPool , .

Btw. " " , , .

Answer 2

ThreadPoolExecutor , . , ?