All geek questions in one place

Java FIFO Queue with disk overflow

I am working / preparing an application based on a producer / consumer model. In my case, there will be one manufacturer that generates several million (non-trivial) tasks, and there will be a customized number of consumers.

The relationship between producer and consumer is primarily queue based. However, I am worried about memory consumption: it is possible that the number of tasks will exceed the available memory for the JVM. Therefore, I would like to have a Queue implementation that only saves the number of elements of the "top-X" queue in memory and saves the rest on disk. This does not have to be resilient in the sense that he does not need to survive restarting the program.

I searched around, but cannot find the Queue implementation that seems to be widely used (there seem to be a few that I call realistic concepts, but I am worried about the future support / further development of these implementations). I also look at external message queue applications, but (1) I don’t want to start a second external process and (2) even embedding a message broker inside the same JVM process seems a bit “the hardest” for this requirement.

Does anyone know of any well-supported future library that provides this functionality?

Rgds

+4
source share
2 answers

Well, JMS seems like an obvious solution. I do not think that you will find something solid to solve this problem, since JMS solves it and is a standard solution.

Please note, however, that Java also has BoundedQueues to solve this problem: measure the queue to make sure that it does not work with OOME when the queue is full, and manufacturers will be blocked when trying to send a message in the full queue limit, until any task will not be removed from the queue by one of the consumers.

+1
source

Having a producer block, when there are more than enough tasks to consume, is usually more efficient than allowing the queue to grow and consume more memory. for example, if your queue is placed in a queue, it can be several times faster than a queue that does not.

0
source

Source: https://habr.com/ru/post/1388794/

More articles:


All Articles