Advantages of anonymous mmap over malloc under memory pressure

Question

I am running some processing code for a large array (on a Pentium running Linux). Array sizes are large enough to exchange processes. While this works, perhaps because I try to keep reading and writing contiguous. However, I will soon have to process large arrays. In this case, switching to anonymous mmapped blocks will help?

If you would, please explain why.

In my shallow understanding, mmap implements a memory mapping file installed from the tmpfs section, which, under the pressure of memory, returns to the exchange mechanism. I would like to understand how mmap does this better than standard malloc (for the sake of argument or reason, I assume it is really better, I don’t know if that is).

Note. Please do not offer to get 64-bit or more RAM. This, unfortunately, is not an option.

Answer 1

The memory that supports your malloc() allocations is handled by the kernel in much the same way as the memory that supports private anonymous mappings created with mmap() . In fact, for large distributions, malloc() will create an anonymous mapping to mmap() to return it anyway, so you are unlikely to see a big difference explicitly using mmap() yourself.

At the end of the day, if your working set exceeds the size of physical memory, you will need to use swap, and whether you use anonymous mappings created using mmap() or malloc() , do not change this. The best thing you can do is try to reorganize your algorithm so that it has good link locality , which will reduce the degree to which you are exchanged.

You can also try to give the kernel some clues about your memory usage with the madvise() system call.

Answer 2

The main difference is that when using malloc(3) -ed input buffers, you ask the kernel to copy data from files mapped to files that are already in memory, and when using mmap(2) you simply use these pages. The first approach doubles the amount of physical memory needed to back up both your and embedded buffers, while the second approach shares this physical memory and only increases the number of virtual mappings for the useland process.