Empty python process hangs on join [sys.stderr.flush ()]

Question

Python Guru I need your help. I came across a rather strange behavior: empty python. The process freezes when it connects . It looks like it is blocking some blocked resources.

Env:

• Python Version: 3.5.3
• OS: Ubuntu 16.04.2 LTS
• Kernel: 4.4.0-75-common

Description of the problem:

1) I have a log with a thread for processing messages in the background and a queue for this thread. Logger source code (slightly simplified).

2) And I have a simple script that uses my logger (just code to display my problem):

import os
from multiprocessing import Process
from my_logging import get_logger


def func():
    pass


if __name__ == '__main__':

    logger = get_logger(__name__)
    logger.start()
    for _ in range(2):
        logger.info('message')

    proc = Process(target=func)
    proc.start()
    proc.join(timeout=3)
    print('TEST PROCESS JOINED: is_alive={0}'.format(proc.is_alive()))

    logger.stop()
    print('EXIT')

script . script "proc" ( script ). "proc" .

, script:

$ for i in {1..100} ; do /opt/python3.5.3/bin/python3.5 test.py ; done

:

Strace :

strace: Process 25273 attached
futex(0x2275550, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, ffffffff

, . , process.py, 269 (python3.5.3), STDERR:

...
267    util.info('process exiting with exitcode %d' % exitcode)
268    sys.stdout.flush()
269    sys.stderr.flush()
...

269 , script .

:

logging.StreamHandler sys.stderr .

, STDERR, STDERR.

:

• python2.7. python2.7. , .
• , .

? ? - ?

Answer 1

, : http://bugs.python.org/issue6721

Answer 2

: ... "proc" .

TEST PROCESS:0 JOINED: is_alive=True

     

time.sleep(5) def func():.
   proc.join(timeout=3), .

:
30 , , proc.join(timeout=3). , .

, , Process/Thread time.sleep(0.05) -.

---

• from multiprocessing import Queue from queue import Queue.

  
     multiprocessing.Queue
     ( ) .

• class QueueHandler(logging.Handler):

  self.queue.put_nowait(record)
  

  class QueueListener(object):
  ...
  def stop(self):
      ...
  

  ,

  class QueueHandler(logging.Handler):
    def __init__(self):
        self.stop = Event()
        ...
  

• def _monitor(self): while ....
  , self._thread

  class QueueListener(object):
  ...
  def stop(self):
       self.handler.stop.set()
       while not self.queue.empty():
           time.sleep(0.5)
       # Don't use double flags
       #self._stop.set()
       self.queue.put_nowait(self._sentinel)
       self._thread.join()