Memory allocation and memory in memory

Question

Given the following test code (x64 environment):

static void Test() { string fileName = @"d:\map"; long length = new FileInfo(fileName).Length; using (var file = MemoryMappedFile.CreateFromFile(fileName, FileMode.Open, "mapFile", length, MemoryMappedFileAccess.ReadWrite)) { byte* bytePtr = (byte*)0; var view = file.CreateViewAccessor(0, length, MemoryMappedFileAccess.ReadWrite); view.SafeMemoryMappedViewHandle.AcquirePointer(ref bytePtr); long count = (long)(length / sizeof(int)); long sum = 0; long step = count / 2000; int* ptr = (int*)&bytePtr[0]; long currentCount = 0 ; Parallel.For(0, count, new ParallelOptions { MaxDegreeOfParallelism = 8 }, (i) => { Interlocked.Add(ref sum, ptr[i]); Interlocked.Increment(ref currentCount) ; if (currentCount % step == 0) Console.Write("\r{0:0.00}%", (100 * currentCount / (float)count)); }); Console.WriteLine(sum); view.Dispose(); } } 

Given that "d: \ map" is a 40-gigabyte file, there is a very strange behavior with random access via the "ptr" pointer.

The physical memory of the system is fully used, and everything slows down, and this process takes more than 2 hours.

When I have serial (and single-threaded) access, the physical memory used does not exceed 1 GB and the process takes about 8 minutes.

My question is: when using a file with memory mapping, is "real" memory used? Not just a busy virtual address space?

I am trying to understand the physical memory consumption when using a memory mapped file.

Answer 1

Memory mapped files use virtual memory. You will not have problems displaying more gigabytes of VM space than you have RAM in a 64-bit operating system. The point of the virtual memory operational system with the requirement, the amount of memory required for all running processes, always significantly exceeds the amount of RAM.

Comparing it with RAM costs money when demand comes to the game. The processor interrupts the program and screams for help when it tries to access a virtual memory address that is not displayed in RAM. A page error is triggered.

If you did not spend this money on getting at least 40 GB of RAM, then you will inevitably pay the cost of the OS for these errors. This requires highlighting the RAM page and filling it with the contents of the file. Perf goes south when it needs to cancel the pre-mapped RAM and save its contents to a file. This is followed by reusing the freed RAM page and loading the contents of the file from the corresponding file offset. Very deer, drive slow. A problem known as "beating."

This is much less of a problem with sequential access to memory, one page error is suitable for 4096 bytes of serial access, and you will have a good chance that the head of the disk reader is still in the right place when you turn off the page error.