Python: Multiprocessing and Queries

Question

The following is a snippet of code that I run to use multiprocessing, which runs an HTTP request in parallel. After starting on the console, it hangs on "request.get (url)" and does not go forward or does not throw an error.

def echo_100(q): ... print "before" r = requests.get(url) print "after" ... q.put(r) q = multiprocessing.Queue() p = multiprocessing.Process(target=echo_100,args=(q)) p.start() p.join() resp = q.get() 

Answer 1

On Mac OS, there seem to be some errors reading proxy settings from the operating system. I don’t know the exact details, but sometimes it causes queries to freeze when using multiprocessing. You can try to work around the problem by completely disabling proxies, for example:

 session = requests.Session() session.trust_env = False # Don't read proxy settings from OS r = session.get(url) 

This fixed it for me.

Answer 2

If you do not clear the queue, I mean that you do not take elements from the queue, the process freezes after a while. Multiprocessor queues are supported by unnamed FIFOs (pipes) on Linux platforms. The maximum size limit for pipes. This means that if the process writing to the handset and no other process reads data from the channel, after a while the recording process will freeze, trying to put more data into the pipe (it can be seen on the internal call of the recording system).

I suspect that you are not getting the item from the queue, so the queue started working after a while, as a result of which subsequent child processes are at an impasse.

Now, if the child process freezes, the parent process can also freeze if it tries to connect (p.join ()) to the child (inside it will call waitpid to join the child process).

Answer 3

I had the same problem with the project. I found that removing import ipdb calls in all of my modules fixes the problem. I am not sure why this import caused this problem, but eliminating these import operations completely fixed this. Only the import problem caused the problem, I didn’t even use anything from the ipdb package.

UPDATE: This happens both in Python 2.7.10 and 3.5.0, and only when importing ipdb ; everything is fine if you import pdb . I wrote a related question to ask why this is happening here

Hope this also solves your problem.