All geek questions in one place

MappedByteBuffer sliding window

Are there any ways to make MappedByteBuffer have a sliding window in the file. I have a very large file (20 GB), but I want to do only 100 MB at a time. I tried this by simply discarding the old buffer and creating a new one from the chanel, but this eats the memory as the old buffers do not seem to be reused.

Any ideas?

+6
source share
1 answer

You can force the old buffer to free its memory immediately with

((DirectBuffer) buffer).cleaner().clean();

Disclaimer: I used this only with an update of Sun / Oracle / OpenJDK Java 6 18 and later. It may not be available or may work correctly with older versions or other platforms. Thanks @EJP.

If you do not have a 32-bit OS, I would just copy the entire file into memory (using several mappings). It will be much more efficient and easy to manage. In this situation, I only clear ByteBuffers as part of closing the file (in unit tests)

You can use TB of virtual memory and use very little physical or even disk space. In this example, I am displaying 8 TB of virtual memory on a machine with 24 GB of memory and a 120 GB disk.

http://vanillajava.blogspot.com/2011/12/using-memory-mapped-file-for-huge.html

In short: on a 64-bit virtual machine, the machine is surprisingly cheap, this is not what you need to worry about.

BTW: Most 64-bit machines are actually limited to 48-bit virtual memory. This is a limit of 256 TB, not 16 EB (18,000,000 TB), which they can theoretically solve.

You may find this library interesting. I have not mentioned this before, as this may not be suitable for you, but you can find an approach and some of the methods that I use in interesting code.

https://github.com/peter-lawrey/Java-Chronicle

+3
source

Source: https://habr.com/ru/post/910012/

More articles:


All Articles