Reading a file with fread () in the reverse order causes a memory leak?

Question

I have a program that basically does this:

• Opening a binary file
• Reads the file back (back, I mean that it starts near the EOF and ends reading at the beginning of the file, that is, it reads the file from right to left) using 4 MB chunks
• Closes a file

My question is: why does memory consumption look lower, although there are no obvious memory leaks in my attached code?

Here is the source of the program that was launched to get the image above:

#include <stdio.h> #include <string.h> int main(void) { //allocate stuff const int bufferSize = 4*1024*1024; FILE *fileHandle = fopen("./input.txt", "rb"); if (!fileHandle) { fprintf(stderr, "No file for you\n"); return 1; } unsigned char *buffer = new unsigned char[bufferSize]; if (!buffer) { fprintf(stderr, "No buffer for you\n"); return 1; } //get file size. file can be BIG, hence the fseeko() and ftello() //instead of fseek() and ftell(). fseeko(fileHandle, 0, SEEK_END); off_t totalSize = ftello(fileHandle); fseeko(fileHandle, 0, SEEK_SET); //read the file... in reverse order. This is important. for (off_t pos = totalSize - bufferSize, j = 0; pos >= 0; pos -= bufferSize, j ++) { if (j % 10 == 0) { fprintf(stderr, "reading like crazy: %lld / %lld\n", pos, totalSize); } /* * below is the heart of the problem. see notes below */ //seek to desired position fseeko(fileHandle, pos, SEEK_SET); //read the chunk fread(buffer, sizeof(unsigned char), bufferSize, fileHandle); } fclose(fileHandle); delete []buffer; } 

I also have the following observations:

• Despite the fact that the amount of RAM used is irregular to 1 GB, the entire program uses only full execution of only 5 MB.
• Commenting out the call to fread() out , the memory leak disappears . This is strange as I don’t allocate anything anywhere near it, which could cause a memory leak ...
• In addition, reading a file usually instead of the opposite (= comment on fseeko() out) speeds up memory leak . This is a supernatural part.

Additional Information...

• The following does not help:
  • Checking the results of fread() does not give anything unusual.
  • Upgrading to regular, 32-bit fseek and ftell .
  • setbuf(fileHandle, NULL) like setbuf(fileHandle, NULL) .
  • setvbuf(fileHandle, NULL, _IONBF, *any integer*) like setvbuf(fileHandle, NULL, _IONBF, *any integer*) .
• Compiled with g ++ 4.5.3 on Windows 7 via cygwin and mingw; without any optimizations, just g++ test.cpp -o test . Both represent this behavior.
• The file used in the tests was 4 GB long, full of zeros.
• A strange pause in the middle of the diagram can be explained by some kind of temporary I / O hang, not related to this issue.
• Finally, if I finish reading in an infinite loop ... memory usage stops increasing after the first iteration.

---

I think this is due to some kind of internal cache building until it is filled with the whole file. How does it really work behind the scenes? How can I prevent this in a portable way?

Answer 1

I think this is an OS problem (or even a problem with using OS resources) than a problem with your program. Of course, it uses only 5 MB of memory: 1 MB for itself (libs, stack, etc.) and 4 MB for the buffer. Whenever you do fread (), the OS seems to “link” part of the file to your process and doesn't seem to release it at the same speed. Since the memory usage on your computer is low, this is not a problem: the OS simply keeps the data it already reads hanging up for longer than necessary, possibly assuming your application can read it soon and then it will not have to bind it again.

If the memory pressure were higher than the operating system, it is likely to cancel the memory faster, so that the leap in memory usage history will be less.

Answer 2

I had the same problem, although in Java, but in this context it does not matter. I solved this by reading much larger pieces at a time. I also read 4 MB chunks, but when I increased it to 100-200 MB, the problem disappeared. Perhaps this will do for you. I am on Windows 7.