Saving a large array of a large array

Question

I'm basically looking for a way to save and reload several large arrays (about 5 million short ) in a quick way on Android. My application should save them in such a way that I can return to them much later, so I can’t just keep them in my memory ...

So far I have tried to convert them to a byte[] array, and they seem to be saved successfully, but I can’t get the data back, this is how my saving code works (it actually separates the functions simplified here):

 ByteArrayOutputStream baos = new ByteArrayOutputStream(); ObjectOuputStream oos = new ObjectOutputStream(baos); oos.writeObject(data); oos.close(); baos.close(); FileOutputStream fos = new FileOutputStream(filename); // valid absolute path fos.write(baos.toByteArray()); fos.close(); 

And part of the download where I get stuck, how do I get short[] from byte[] ? I also read that the database can also work, but is it fast enough?

I looked around Stackoverflow and Google and did not seem to find anyone with a similar problem, or at least a solution for it, but as a newbie, I might have missed something obvious ...

Answer 1

If data is short[] and you want to write it all to a file, do not use a buffer in memory, write it directly.

 ObjectOutputStream oos = new ObjectOutputStream(new FileOutputStream(filename)); try { oos.writeObject(data); } finally { oos.close(); } 

If you write it using an ObjectOutputStream , you will need to read it through an ObjectInputStream , as it not only writes clean data, but also some type information. If you enter short[] , you get back short[] (you can skip these bytes, but you will have to analyze what the stream actually writes). This also applies if your ObjectOutputStream written to ByteArrayOutputStream .

If you don't want to handle this mess, do what ObjectOutputStream basically:

 DataOutputStream dos = new DataOutputStream(new FileOutputStream(filename)); try { for (int i = 0; i < data.length; i++) { dos.writeShort(data[i]); } } finally { dos.close(); } 

DataOutputStream writes simple data, so you can just read the data directly as byte[] if you want. If byte order matters: it uses Big-Endian.

---

Since this approach writes single bytes instead of byte[] cartridges, it benefits from using a BufferedOutputStream between them. The default buffer is 8 KB, but it can increase, which can give even better results. Writing data now converts short to byte in memory, and as soon as enough data is available, the entire fragment will be transferred to the low-level functions of the OS to write it.

 int bufferSize = 32 * 1024; DataOutputStream dos = new DataOutputStream( new BufferedOutputStream( new FileOutputStream(filename), bufferSize) ); try { for (int i = 0; i < data.length; i++) { dos.writeShort(data[i]); } } finally { dos.close(); } 

Answer 2

Use file streams directly without buffering through ByteArrayOutputStream / ByteArrayInputStream . byte[] is stored in memory and is not suitable for large data potions.

Writing to new ObjectOutputStream(new FileOutputStream(filename)) , you can return your objects using new ObjectInputStream(new FileInputStream(filename)) . If you want, you can also do some buffering as indicated by Christopher, answer .

Answer 3

Perhaps you could use a method here: an array of bytes for a short array and back in java and just write and read bytearray. I also recommend using BufferedOutputStream instead of just FileOutputStream.