Java Outofmemory error: how to make deque from a vector?

Question

I am new to Java and really need your help.

I am currently using a queue, the receiving thread puts data in this queue, and the parser is reading this. But the problem is that the receiver can receive at an incredible maximum speed, for example, . 3000 / s, while the parser only analyzes 100 / s.

EDIT: I checked, the first turn remains at 100 or so, and after ten seconds it starts to grow at 100 for the second, and the crash is at 2000 or so. Is it possible that there is a memory leak?

My code (in a closed loop)

byte[] data = new byte[1024]; System.arraycopy(udpPacket.getData(), 0, data, 0, 1024); queue.offer(data); 

The heap is filled up too fast and I get an outofmemory exception. I think the problem is that the queue is created using a linked list, and all pointers should be stored on the heap.

I know a version of C that does the same thing (using a buffer) but has much better performance, but due to deployment problems we can only use Java.

Answer 1

If you get 3000 / sec, but only the 100 / sec process, sooner or later you will run out of memory. Can I suggest you use more threads for parsing?

For the queue, see LinkedBlockingDeque and LinkedBlockingQueue . There are also high-performance in-line queue implementations.

Answer 2

Since data arrives 30 times faster than it is processed, you can extend HeapSize using

java -Xms<initial heap size> -Xmx<maximum heap size> if the transfer is complete before your memory is exhausted.

• Or, as you suggested, flush the data to disk and handle the delay.
• Otherwise, you have to optimize your parser

Answer 3

If the producer produces more data, then the consumer can handle it, then the data will begin to accumulate, and as a result you will encounter OutOfMemory problems. This will depend on (1) the difference in speed between the producer and the consumer, (2) the amount of data that you must process.

I suggest you limit the number of items in the queue. Use BlockingDeque LinkedBlockingDeque to limit queue capacity and lock your loop when the limit is reached. Thus, the queue acts as a cache for the parser.

Answer 4

I think the problem is that the queue is created using a linked list, and all pointers should be stored on the heap.

I do not think so. I believe that the real problem is the mismatch between the speed at which your system receives input and the speed that it can handle. If you cannot handle at least the average input speed, you will eventually run out of memory, no matter how you represent the queue.

You must either improve processing speed, reduce input speed, or ... delete data.

Answer 5

Another way to do this would be to fetch the data when the queue gets too large, and keep the sample rate so that we can simulate the original data.

Answer 6

When you run java , you can use the -Xmx option to make more memory available to the virtual machine. For example, java -Xmx512m will allow the VM to allocate up to 512 MB of memory. (The default value is pretty small).

But if you allocate memory and populate the list with data and never delete it, you will end up with running out of memory, no matter what language you use.

Answer 7

I have a theory, the implementation of ArrayDeque (at least in Oracle JDK, I'm not sure about Android) seems to never release popups.

In other words, pop-up slots are simply zero. While the elements added to the tail will make the "inner array" grow forever, which will cause problems sooner or later.

This code is from Oracle JDK 1.8.0_144:

 public E pollFirst() { int h = head; @SuppressWarnings("unchecked") E result = (E) elements[h]; // Element is null if deque empty if (result == null) return null; elements[h] = null; // Must null out slot head = (h + 1) & (elements.length - 1); return result; } 

Spells for me. :(

If my analysis is correct, it seems that ArrayDeque never intended for the "real" FIFO queue and is not suitable for this purpose. (Unfortunately, I need this goal now)

I am currently studying LinkedList (which also implements Deque ).