Python matrix multiplication: how to handle very large matrices?

Question

a = numpy.zeros((17770,5)) b = numpy.zeros((5,20000)) ma = numpy.matrix(a) mb = numpy.matrix(b) 

That is, ma.shape = (17770,5) , mb.shape = (5,20000) , both are numpy.matrix .

I need ma*mb . But I get the error "ValueError: array is too big" .

Are these matrices too large to be multiplied by Python?

BTW, I tested with python2.6.6 / 32bit / 3GB RAM

Answer 1

I can calculate ma*mb on my machine ( Python 2.7.1 |EPD 7.0-2 (64-bit) on 64-bit Ubuntu).

Make sure you use 64-bit Python on a 64-bit OS, as the 17770x20000 double-precision matrix requires 2.8 GB of RAM, which is superior (or very close) to what most 32-bit pen platforms can use.

Depending on your requirements, the use of single-point floats ( numpy.float32 ) may also be possible.

Finally, if your matrices are sparse or structured, you might want to study them to reduce memory requirements.