MemoryMappedFiles: how much memory can be allocated for files

Question

I have large rawdata CT files that can exceed 20-30 GB in size. For most of our modern computers in the department, we have a maximum of 3 GB. But for data processing we need to go through all the data available. Of course, we could do this by sequentially viewing the data using the read and write functions. But sometimes it is necessary to store some data in memory.

Currently, I have my own memory management, where I created the so-called MappableObject. Each rawdata file contains, say, 20,000 structures, each of which shows different data. Each MappableObject refers to a location in the file.

In C #, I created a somewhat partially working mechanism that automatically updates mps and, if necessary, discards data. Since then I know MemoryMappedFiles, but in .NET 3.5 I refused to use it because I knew that in .NET 4.0 it would be available initially.

So, today I tried MemoryMappedFiles and found out that it is not possible to allocate as much memory. If I have a 32-bit system and I want to allocate 20 GB, it does not work due to the excess of the size of the logical address space. This is somehow clear to me.

But is there a way to handle large files like mine? What other chances do I have? How do you guys decide these things?

Thanks Martin

Answer 1

The only limitation that I know of is the size of the largest kind of file that can be matched, which is limited by the address space. A memory mapped file may be larger than the address space. Windows needs to display the file representation in an adjacent fragment of your process address space, so the size of the largest match is the size of the largest free fragment of the address space. The only limitation on the total file size is the file system itself.

Take a look at this article: Working with large memory-related files

Answer 2

"Memory mapped", you cannot map 20 gigabytes to a virtual address space of 2 gigabytes. Getting 500 MB on a 32-bit operating system is difficult. Remember that this is not a good solution if you do not need heavy random access to the file data. Which should be difficult when you need to share views. Sequential access through a regular file is unsurpassed with very modest memory usage. Also beware of the cost of marshaling data from an MMF; you still pay for a copy of the managed structure or the cost of marshaling.

Answer 3

You can read the file sequentially, you cannot store more than 2 GB of data in memory.

You can display blocks of a file at a time, preferably blocks that are multiples of your structure.

eg. File - 32 GB. The memory card displays 32 MB of the file at a time and analyzes it. Once you click on the end of these 32MB, hover the next 32MB of the file and continue until you reach the end of the file.

I'm not sure what the optimal display size is, but this is an example of how this can be done.

Answer 4

You are both right. At first I tried to use a file with memory without a file. It doesn’t work there. If I have an existing file. I can display as much memory as I want. The reason I wanted to use MemoryMappedFiles without the actual existing file is because it should be automatically deleted when the stream is deleted. It is not supported by MemoryMappedFile.

Now I saw that to get the expected result, I can do the following:

// Create the stream FileStream stream = new FileStream( "D:\\test.dat", FileMode.Create, FileAccess.ReadWrite, FileShare.ReadWrite, 8, FileOptions.DeleteOnClose // This is the necessary part for me. ); // Create a file mapping MemoryMappedFile x = MemoryMappedFile.CreateFromFile( stream, "File1", 10000000000, MemoryMappedFileAccess.ReadWrite, new MemoryMappedFileSecurity(), System.IO.HandleInheritability.None, false ); // Dispose the stream, using the FileOptions.DeleteOnClose the file is gone now stream.Dispose(); 

At least when you look at the first result, it looks good to me.

Thanks.