Well, one day of experimentation showed that the 32-bit JVM does not work with IOException: Map failed at 1 GB no matter what. To get around Size exceeds Integer.MAX_VALUE with display on 64-bit machines, you should use several buffers of available size, for example. 100 mb each is fine. This is because the buffers are an integer address .
Regarding this issue, you can save all such buffers at once in memory, i.e. no need to close one buffer => zero before allocating the next one to effectively increase the file size, as shown in the following demo
import Utils._, java.nio._, java.io._, java.nio.channels.FileChannel object MmfDemo extends App { val bufAddrWidth = 25 /*in bits*/ // Every element of the buff addresses a long val BUF_SIZE_WORDS = 1 << bufAddrWidth ; val BUF_SIZE_BYTES = BUF_SIZE_WORDS << 3 val bufBitMask = BUF_SIZE_WORDS - 1 var buffers = Vector[LongBuffer]() var capacity = 0 ; var pos = 0 def select(pos: Int) = { val bufn = pos >> bufAddrWidth // higher bits of address denote the buffer number //println(s"accessing $pos = " + (pos - buf * wordsPerBuf) + " in " + buf) while (buffers.length <= bufn) expand pass(buffers(bufn)){_.position(pos & bufBitMask)} } def get(address: Int = pos) = { pos = address +1 select(address).get } def put(value: Long) { //println("writing " + value + " to " + pos) select(pos).put(value) ; pos += 1 } def expand = { val fromByte = buffers.length.toLong * BUF_SIZE_BYTES println("adding " + buffers.length + "th buffer, total size expected " + format(fromByte + BUF_SIZE_BYTES) + " bytes") // 32bit JVM: java.io.IOException: "Map failed" at the following line if buf size requested is larger than 512 mb // 64bit JVM: IllegalArgumentException: Size exceeds Integer.MAX_VALUE buffers :+= fc.map(FileChannel.MapMode.READ_WRITE, fromByte, BUF_SIZE_BYTES).asLongBuffer() capacity += BUF_SIZE_WORDS } def rdAll(get: Int => Long) { var firstMismatch = -1 val failures = (0 until parse(args(1))).foldLeft(0) { case(failures, i) => val got = get(i) if (got != i && firstMismatch == -1) {firstMismatch = i; println("first mismatch at " +format(i) + ", value = " + format(got))} failures + ?(got != i, 1, 0) } ; println(format(failures) + " mismatches") } val raf = new RandomAccessFile("""C:\Temp\mmf""", "rw") val fc = raf.getChannel try { if (args.length < 1) { println ("usage1: buf_gen <len in long words>") println ("usage1: raf_gen <len in long words>") println("example: buf_gen 30m") println("usage2: raf_rd <size in words>") println("usage3: buf_rd <size in words>") } else { val t1 = System.currentTimeMillis args(0) match { case "buf_gen" => raf.setLength(0) (0 until parse(args(1))) foreach {i => put(i.toLong)} case "raf_gen" => raf.setLength(0) (0 until parse(args(1))) foreach {i =>raf.writeLong(i.toLong)} //fc.force(true) case "rd_raf" => rdAll{i => raf.seek(i.toLong * 8) ; raf.readLong()} case "rd_buf" => rdAll(get) case u =>println("unknown command " + u) } ; println("finished in " + (System.currentTimeMillis - t1) + " ms") } } finally { raf.close ; fc.close buffers = null ; System.gc /*GC needs to close the buffer*/} } object Utils { val formatter = java.text.NumberFormat.getIntegerInstance def format(l: Number) = formatter.format(l) def ?[T](sel: Boolean, a: => T, b: => T) = if (sel) a else b def parse(s: String) = { val lc = s.toLowerCase() lc.filter(_.isDigit).toInt * ?(lc.contains("k"), 1000, 1) * ?(lc.contains("m"), 1000*1000, 1) } def eqa[T](a: T, b: T) = assert(a == b, s"$a != $b") def pass[T](a: T)(code: T => Unit) = {code(a) ; a} }
at least on Windows. Using this program, I managed to create the mmf file more than my machine memory (not to mention the JVM-Xmx, which do not play any role in these matters at all). Just slow down the creation of the file by selecting any text in the Windows console with the mouse (the program will stop until you release the choice), because otherwise Windows will output all other critical critical employees to the page file, and your computer will die in a swoon.
BTW, the computer dies when I write only to the end of the file, and Windows can cut out my unused gigabyte blocks. Also, I noticed that the block I'm writing is really readable
Next conclusion
adding 38th buffer, total size expected 12,480,000,000 bytes adding 39th buffer, total size expected 12,800,000,000 bytes
accompanied by the following system requests
5:24,java,"QueryStandardInformationFile",mmf,"SUCCESS","AllocationSize: 12 480 000 000, EndOfFile: 12 480 000 000, NumberOfLinks: 1, DeletePending: False, Directory: False" 5:24,java,"SetEndOfFileInformationFile",mmf,"SUCCESS","EndOfFile: 12 800 000 000" 5:24,java,"SetAllocationInformationFile",mmf,"SUCCESS","AllocationSize: 12 800 000 000" 5:24,java,"CreateFileMapping",mmf,"FILE LOCKED WITH WRITERS","SyncType: SyncTypeCreateSection, PageProtection: " 5:24,java,"QueryStandardInformationFile",mmf,"SUCCESS","AllocationSize: 12 800 000 000, EndOfFile: 12 800 000 000, NumberOfLinks: 1, DeletePending: False, Directory: False" 5:24,java,"CreateFileMapping",mmf,"SUCCESS","SyncType: SyncTypeOther" 5:24,java,"ReadFile",mmf,"SUCCESS","Offset: 12 480 000 000, Length: 32 768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal" 5:24,java,"ReadFile",mmf,"SUCCESS","Offset: 12 480 032 768, Length: 32 768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal" 5:24,java,"ReadFile",mmf,"SUCCESS","Offset: 12 480 065 536, Length: 32 768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal" 5:24,java,"ReadFile",mmf,"SUCCESS","Offset: 12 480 098 304, Length: 32 768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal" 5:24,java,"ReadFile",mmf,"SUCCESS","Offset: 12 480 131 072, Length: 20 480, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal" skipped 9000 reads 5:25,java,"ReadFile",mmf,"SUCCESS","Offset: 12 799 836 160, Length: 32 768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal" 5:25,java,"ReadFile",mmf,"SUCCESS","Offset: 12 799 868 928, Length: 32 768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal" 5:25,java,"ReadFile",mmf,"SUCCESS","Offset: 12 799 901 696, Length: 32 768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal" 5:25,java,"ReadFile",mmf,"SUCCESS","Offset: 12 799 934 464, Length: 32 768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal" 5:25,java,"ReadFile",mmf,"SUCCESS","Offset: 12 799 967 232, Length: 32 768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal"
but this is another story.
It turns out that this answer is duplicated by Peter Lawrey's , except that my question is about "Map Error" and "Integer Range Exceeded" when comparing large buffers, while the original question is related to OutOfMem in the JVM, which has nothing to do with I / O .