Problem
I have a rather complicated application for image processing, where one of the submodules needs to load huge binary bitmaps into memory. In fact, up to 96 GB (which means an image of 888 888 x 888 888 pixels). Disks - 2xSSD raid0 with read / write speed of about 1 GB / s. It loads an image into a vector (each element represents a line in a bitmap image) of smart pointers to a vector with bytes (each element represents 8 pixels). The strange problem is that after reloading and cleaning vectors (I see that the memory is actually full and cleared without memory leak), each iteration takes more time and more time. Special memory cleaning takes a very long time.
Test
I made some simple test applications to test this isolated and from different angles. Replacing smart pointers with source pointers gave the same weird behavior. Then I tried using my own arrays instead of the vector, and that did the trick. After 100 iterations of loading / cleaning 24 GB, the time increased dramatically when using vectors, while the implementation of the array was stable for a while. Below is a test application filling memory with 24 GB of garbage instead of loading the actual image with the same results. Tests performed on Windows 10 Pro with 128 GB of RAM and built-in with Visual Studio version 5.0 upgrade.
This function uses vectors to load / clear:
void SimpleLoadAndClear_Vector(int width, int height) { time_t start_time, end_time; // Load memory time(&start_time); cout << "Loading image into memory..."; auto width_bytes = width / 8; auto image = new vector<vector<unsigned char>*>(height); for (auto y = 0; y < height; y++) { (*image)[y] = new vector<unsigned char>(width_bytes); auto row_ptr = (*image)[y]; for (auto b = 0; b < width_bytes; b++) { (*row_ptr)[b] = 0xFF; } } cout << "DONE: "; time(&end_time); auto mem_load = (int)difftime(end_time, start_time); cout << to_string(mem_load) << " sec" << endl; // Clear memory time(&start_time); cout << "Clearing memory..."; for (auto y = 0; y < height; y++) { delete (*image)[y]; } delete image; cout << "DONE: "; time(&end_time); auto mem_clear = (int)difftime(end_time, start_time); cout << to_string(mem_clear) + " sec" << endl; }
This function uses arrays to load shed:
void SimpleLoadAndClear_Array(int width, int height) { time_t start_time, end_time; // Load memory time(&start_time); cout << "Loading image into memory..."; auto width_bytes = width / 8; auto image = new unsigned char*[height]; for (auto y = 0; y < height; y++) { image[y] = new unsigned char[width_bytes]; auto row_ptr = image[y]; for (auto b = 0; b < width_bytes; b++) { row_ptr[b] = 0xFF; } } cout << "DONE: "; time(&end_time); auto mem_load = (int)difftime(end_time, start_time); cout << to_string(mem_load) << " sec" << endl; // Clear memory time(&start_time); cout << "Clearing memory..."; for (auto y = 0; y < height; y++) { delete[] image[y]; } delete[] image; cout << "DONE: "; time(&end_time); auto mem_clear = (int)difftime(end_time, start_time); cout << to_string(mem_clear) + " sec" << endl; }
This is the main function to call the above load / clear functions:
void main() { auto width = 455960; auto height = 453994; auto i_max = 50; for (auto i = 0; i < i_max; i++){ SimpleLoadAndClear_Vector(width, height); } }
The test results from the vector version look like this after 50 iterations (obviously, loading / cleaning time is increasing more and more):
Loading image into memory...DONE: 19 sec Clearing memory...DONE: 24 sec Loading image into memory...DONE: 40 sec Clearing memory...DONE: 20 sec Loading image into memory...DONE: 27 sec Clearing memory...DONE: 39 sec Loading image into memory...DONE: 35 sec Clearing memory...DONE: 24 sec Loading image into memory...DONE: 27 sec Clearing memory...DONE: 34 sec Loading image into memory...DONE: 33 sec Clearing memory...DONE: 29 sec Loading image into memory...DONE: 27 sec Clearing memory...DONE: 35 sec Loading image into memory...DONE: 32 sec Clearing memory...DONE: 33 sec Loading image into memory...DONE: 28 sec Clearing memory...DONE: 37 sec Loading image into memory...DONE: 31 sec Clearing memory...DONE: 35 sec Loading image into memory...DONE: 30 sec Clearing memory...DONE: 38 sec Loading image into memory...DONE: 31 sec Clearing memory...DONE: 38 sec Loading image into memory...DONE: 31 sec Clearing memory...DONE: 41 sec Loading image into memory...DONE: 32 sec Clearing memory...DONE: 40 sec Loading image into memory...DONE: 33 sec Clearing memory...DONE: 42 sec Loading image into memory...DONE: 35 sec Clearing memory...DONE: 43 sec Loading image into memory...DONE: 34 sec Clearing memory...DONE: 46 sec Loading image into memory...DONE: 36 sec Clearing memory...DONE: 47 sec Loading image into memory...DONE: 35 sec Clearing memory...DONE: 49 sec Loading image into memory...DONE: 37 sec Clearing memory...DONE: 50 sec Loading image into memory...DONE: 37 sec Clearing memory...DONE: 51 sec Loading image into memory...DONE: 39 sec Clearing memory...DONE: 51 sec Loading image into memory...DONE: 39 sec Clearing memory...DONE: 53 sec Loading image into memory...DONE: 40 sec Clearing memory...DONE: 52 sec Loading image into memory...DONE: 40 sec Clearing memory...DONE: 55 sec Loading image into memory...DONE: 41 sec Clearing memory...DONE: 56 sec Loading image into memory...DONE: 41 sec Clearing memory...DONE: 59 sec Loading image into memory...DONE: 42 sec Clearing memory...DONE: 59 sec Loading image into memory...DONE: 42 sec Clearing memory...DONE: 60 sec Loading image into memory...DONE: 44 sec Clearing memory...DONE: 60 sec Loading image into memory...DONE: 44 sec Clearing memory...DONE: 63 sec Loading image into memory...DONE: 44 sec Clearing memory...DONE: 63 sec Loading image into memory...DONE: 45 sec Clearing memory...DONE: 64 sec Loading image into memory...DONE: 46 sec Clearing memory...DONE: 65 sec Loading image into memory...DONE: 45 sec Clearing memory...DONE: 67 sec Loading image into memory...DONE: 47 sec Clearing memory...DONE: 69 sec Loading image into memory...DONE: 47 sec Clearing memory...DONE: 70 sec Loading image into memory...DONE: 48 sec Clearing memory...DONE: 72 sec Loading image into memory...DONE: 48 sec Clearing memory...DONE: 74 sec Loading image into memory...DONE: 49 sec Clearing memory...DONE: 74 sec Loading image into memory...DONE: 50 sec Clearing memory...DONE: 74 sec Loading image into memory...DONE: 50 sec Clearing memory...DONE: 76 sec Loading image into memory...DONE: 51 sec Clearing memory...DONE: 78 sec Loading image into memory...DONE: 53 sec Clearing memory...DONE: 78 sec Loading image into memory...DONE: 53 sec Clearing memory...DONE: 80 sec Loading image into memory...DONE: 54 sec Clearing memory...DONE: 80 sec Loading image into memory...DONE: 54 sec Clearing memory...DONE: 82 sec Loading image into memory...DONE: 55 sec Clearing memory...DONE: 91 sec Loading image into memory...DONE: 56 sec Clearing memory...DONE: 84 sec Loading image into memory...DONE: 56 sec Clearing memory...DONE: 88 sec
The test result from the version of the array looks like this after 50 iterations (obviously, the loading / cleaning time is stable and does not increase more):
Loading image into memory...DONE: 18 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 17 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 17 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 17 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 25 sec Loading image into memory...DONE: 27 sec Clearing memory...DONE: 17 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 17 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 17 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 17 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 25 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 25 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 25 sec Clearing memory...DONE: 19 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 25 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 25 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 25 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 25 sec Loading image into memory...DONE: 25 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 25 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 25 sec Clearing memory...DONE: 17 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 25 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 25 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 25 sec Loading image into memory...DONE: 25 sec Clearing memory...DONE: 19 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 25 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec
Questions
- Is this Windows that does not handle memory operations well when dealing with huge std :: vectors?
- Is it std :: vectors that just does crappy with huge data by design?
- I didn’t understand something at all?
- Is there any other obvious std container that I should have used instead (do I need to access image data by index in x and y from different threads)?
- Any other good explanation and suggested solution?