Why can I use a large buffer as a vector, but not with a new one in Windows?

Question

I am using 64 bit Windows 7 Pro and Visual Studio 2010 Pro.

I am trying to allocate and use a buffer larger than 4 GB (to collect data at a high data rate).

Allocating and writing a buffer as a byte vector works fine. Allocating a buffer as an array of bytes works fine, but writing to this array quickly fails. (The last message printed is "buffer allocated.")

A note on vector partitioning does not fix the problem.

Below is my test program:

#include <iostream> #include <vector> #include <BaseTsd.h> using namespace std; int main() { const ULONG64 BUF_SIZE = 4 * 1024ULL * 1024ULL * 1024ULL; { vector<unsigned __int8> v(BUF_SIZE); cout << "vector allocated" << endl; for (ULONG64 i = 0; i < BUF_SIZE; ++i) { v[i] = 0xff; } cout << "vector written" << endl; } { unsigned __int8* buffer = new unsigned __int8[BUF_SIZE]; cout << "buffer allocated" << endl; for (ULONG64 i = 0; i < BUF_SIZE; ++i) { buffer[i] = 0xff; } cout << "buffer written" << endl; delete[] buffer; } return 0; } 

UPDATE: I believe this is a compiler error. Take a look here: http://connect.microsoft.com/VisualStudio/feedback/details/553756/invalid-check-for-maximum-array-size-in-x64-compiler-c2148

Answer 1

I just tried to compile this code with VS2010 Pro (64-bit), and the compiler generated a C2148 error for new call:

 error C2148: total size of array must not exceed 0x7fffffff bytes 

I compiled it from the command line after running vcvarsx86_amd64.bat . It seems that the restriction given here may somehow come into play. Changing new to [BUF_SIZE-1] allowed him to compile and run (although this is even more than the number 0x7fffffff discussed in these links).