Unable to allocate 1.6 GB in Python

Question

This code MemoryError :

 from pylab import complex128 import numpy x = numpy.empty(100000000, dtype=complex128) # 100 millions complex128 

I have Win7 64 with 8 GB of RAM (at least 5.3 GB when running this code). I am using Python 2.7 (Anaconda) and I think this is a 32 bit version. Even with 32 bits, we can handle 1.6 GB!

Do you know how to solve this?

PS: I expected an array of 100 million elements, each of which used 16 bytes (128 bits) to use 16 * 100 million = 1.6 GB. This is confirmed:

 x = numpy.empty(1000000, dtype=complex128) # 1 million here print x.nbytes >>> 16000000 # 16 MB 

Answer 1

The problem was solved using 64-bit Python.

It is even possible to create a single array larger than 5 GB.

Note: when I create an array that should use 1,600,000,000 bytes (with 100 million elements in the complex128 array), the actual memory usage is not much "bigger": 1,607,068 KB ...

Answer 2

I know an old question. And I know that there are many similar questions. ex. Memory for python.exe on Windows 7 python 32 - does Numpy use only half of the available memory? But none of them seem to really solve the problem.

Using the hint provided here by fooobar.com/questions/294623 / ... , I think I finally fixed this problem.

First you need to install "Microsoft Visual C ++ Express Edition 2008". You can follow the instructions given here: http://blog.victorjabur.com/2011/06/05/compiling-python-2-7-modules-on-windows-32-and-64-using-msvc-2008- express /

The download URL for Microsoft Visual C ++ Express Edition 2008 in the above blog article is dead. But here you can find the Visual C ++ 2008 Express URL Download link Dead? .

(EDIT) I confirmed that the linker that comes with msvc-2010-express also works.

Then start the Visual Studio 2008 Command Prompt from the Start menu → Microsoft Visual C ++ 2008 Express Edition → Visual Studio Tools → Visual Studio 2008 Command Prompt

Then run the following commands:

 cd bin editbin.exe /LARGEADDRESSAWARE "C:\Python27\python.exe" 

This will set the IMAGE_FILE_LARGE_ADDRESS_AWARE flag in the python executable. With this magic, a 32-bit python will use up to 4 GB (instead of ~ 2 GB on windows).

According to MSDN:

On 64-bit versions of Windows, 32-bit applications marked with the IMAGE_FILE_LARGE_ADDRESS_AWARE flag have 4 GB of address space available.

Now

 x = numpy.empty(100000000, dtype=complex128) 

works on my 64-bit computer running Windows 7 with 32-bit Python 2.7.

I really hope that the official binary version of python will be sent with this FLAG, which is already installed, since there is no harm in this, but a huge benefit!

As MSDN says:

Setting this flag and then launching the application on a system that does not support 4GT does not affect the application.