I have a situation where my main thread (producer) allocates a huge chunk of memory to a heap for a task, does some work on this buffer and then provides a buffer to worker threads (consumers) for further processing (which first compress this data, and then burn them to disk). When a worker thread runs on a job, it frees up the memory that the manufacturer received for the task.
However, there may be a situation where my main thread allocates too much memory, and therefore my system begins to replace other programs on disk to meet the memory requirement. As the disk becomes busy, workflows make it harder to write to the disk (and ultimately free up memory), while manufacturers continue to allocate more memory for other tasks. This, after all, kills my system performance.
What could be a good design for this problem?
In addition, if you suspend the main thread by pre-programming the memory request, in advance, this is an option, how can I arrive at a reliable number?
source
share