How to handle an array of size 1,000,000,000 in C ++?

Question

I need to process 3D cube data. Its amount can be several billion. I understand that I can not allocate so much memory in Windows. Therefore, I am thinking about disk operations with an embedded database. Is there a better way to do this? Maybe something in boost?

Update: I will eventually have to provide graph viewing functions.

Update2: The following article seemed to be a good solution using a memory mapped file. I will try it and update again. http://www.codeproject.com/Articles/26275/Using-memory-mapped-files-to-conserve-physical-mem

Answer 1

Perhaps you saved the data more efficiently (see Bentley's Pearl Programming), is this rare data ?!

If not, memory mapped files (MMFs) are your friend and allow you to map MMF fragments to memory that you can access like any other memory.

Use CreateFileMapping and MapViewOfFile to draw a piece into your process.

Answer 2

The first and easiest step is to break the data into pieces. The size of the piece depends on your needs: it can be the smallest or largest fragment that can be drawn at a time, or for which geometry can be built, or the optimal size for compression.

As soon as you work with managed chunks, a memory problem will be prevented. A stream of pieces (loading and unloading / saving) as needed.

During the load / save process, you may need compression and / or a database. Even something simple, such as RLE and SQLite (a separate table with coordinates and a data block), can save a little space. Better compression allows you to work with larger block sizes.

Depending on the usage, it may be possible to store the compressed fragments in memory and only unpack the modification for a short time (or when they can be changed). If your data is read-only, downloading it and decompressing it only if necessary will be very useful.

Dividing data into pieces also has side benefits, such as an extremely simple form for axes, which allows you to generate geometry (marching cubes, etc.) for working on isolated data fragments (simplifies flows) and makes saving / loading process much easier.

Answer 3

try VirtualAlloc from <windows.h> .

https://msdn.microsoft.com/en-us/library/windows/desktop/aa366887%28v=vs.85%29.aspx

its quite useful for large arrays if they fit into your RAM, after which the answer 0xC0000022L is probably the best solution