Transferring multiple arrays through multiprocessing.

Question

I use multiprocessing.Queueto pass numpy arrays float64between python processes. This works great, but I'm worried that it may not be as effective as it could be.

According to the documentation, multiprocessingobjects placed on Queuewill be pickled. a call picklein the numpy array results in a textual representation of the data, so null bytes are replaced with a string "\\x00".

>>> pickle.dumps(numpy.zeros(10)) "cnumpy.core.multiarray\n_reconstruct\np0\n(cnumpy\nndarray\np1\n(I0\ntp2\nS'b'\np3\ntp4\nRp5\n(I1\n(I10\ntp6\ncnumpy\ndtype\np7\n(S'f8'\np8\nI0\nI1\ntp9\nRp10\n(I3\nS'<'\np11\nNNNI-1\nI-1\nI0\ntp12\nbI00\nS'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'\np13\ntp14\nb."

I am worried that this means that my arrays are expensively converted to something 4 times the original value, and then converted back to another process.

Is there a way to pass data through the queue in raw form?

, , , .

!

Answer 1

numpy, pickle ( , ). pickle .

import numpy
import cPickle as pickle

N = 1000
a0 = pickle.dumps(numpy.zeros(N))
a1 = pickle.dumps(numpy.zeros(N), protocol=-1)

print "a0", len(a0)   # 32155
print "a1", len(a1)   #  8133

, , , , , cPickle ( ).

:
, . . , , , , - . , , . , . , Eelco, 60 , , . , , , , .

Answer 2

, Numpy

. - , , .