Byte length exceeded [] (upper limit of int) - java.lang.ArrayIndexOutOfBoundsException

Question

I have a ByteArrayOutputStream object that causes the following error:

java.lang.ArrayIndexOutOfBoundsException at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:113) 

I am trying to upload a file consisting of several concerts by writing bytes [] of fragments of 250 MB each one at a time.

I can observe how the byte grows in size and as soon as it reaches a length of 2147483647, the upper limit is int, it explodes on the following line:

 stream.write(buf); 

stream - ByteArrayOutputStream, buf is what I write to the stream in 250 MB blocks.

I planned to do

 byte result[] = stream.toByteArray(); 

At the end. Is there another method that I can try that will support byte array sizes exceeding the upper limit of int?

Answer 1

Arrays in Java simply cannot exceed int boundaries.

From JLS section 15.10 :

The type of each dimension expression in DimExpr must be a type that is convertible (Section 5.1.8) to an integral type or a compile-time error occurs. Each expression undergoes unary numeric promotion (§). The elevated type must be int or a compile-time error occurs; this means, in particular, that the type of the dimension expression does not have to be long.

Similarly in the JVM specification for arraylength length :

arrayref must have a type reference and must reference an array. It is popped from the operand stack. The length of the found array is determined. This length is pushed onto the operand stack as an int .

This basically provides the maximum size of arrays.

Actually, it’s not entirely clear what you are going to do with the data after loading it, but I would try not to load all this into memory for starters.

Answer 2

Use multiple arrays. When you reach the limit, use ByteArrayOutputStream.toByteArray() and reset with ByteArrayOutputStream.reset() .

Answer 3

Using ByteArrayOutputStream to write multiple GiB data is not a good idea, since everything should be stored in the computer's memory. As you can see, the byte array is limited to 2 ^ 31 bytes (2GiB).

In addition, the buffer used to store this data does not grow if you write more data in it, so if the buffer used is full, you need to create a new one (usually double size), and all data must be copied from the old buffer to the new one.

My advice: use RandomAccessFile and save the data you get to a file. Through RandomAccessFile you can work with data files larger than 2GiB.