Python ZMQ and multiprocessing raises zmq.error.ZMQError: interrupted system call

Question

I have a Python script where I bind several (e.g. 5) ZMQ receiver sockets like this:

receiver_1 = context.socket(zmq.PULL)
receiver_1.bind("tcp://*:5555")
...
receiver_5 = context.socket(zmq.PULL)
receiver_5.bind("tcp://*:5559")

receivers = [receiver_1, ..., receiver_5]

Then I launch several instances of the Google Compute Engine and plug in the corresponding sender sockets.

I would like to extract from these sockets in parallel, and therefore I am trying to do this using a multiprocessor pool. The code looks something like this:

def recv_result(i):
    result_str = receivers[i].recv()
    return cPickle.loads(result_str)

pool = multiprocessing.Pool()
while True:
    results = pool.map(recv_result, [i for i in range(len(receivers))])
    # break when all results have been received
    ...

The error that I get when running the script is as follows:

Traceback (most recent call last):
  ...
  File ...
    results = pool.map(recv_result, [i for i in range(len(receivers))])
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 227, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 528, in get
    raise self._value
zmq.error.ZMQError: Interrupted system call

I also tried to implement the same functionality using multprocessing.Process, but I am getting essentially the same error, albeit more randomly.

, , - GCE, script ( , GCE ). - - , !

Answer 1

:

• , ZeroMQ - , .
• . / ZeroMQ, .
• /, zmq, , . , .
• zeromq , . , .