All geek questions in one place

How to write big binary data quickly in Java?

I am writing an STL file consisting of an 80 byte header, 4 byte integer, and then a record of 50 bytes, each of which consists of floats and a short integer.

Using RandomAccessFile, I can easily write data, but it's terribly slow. This uses the same interface as the DataOutputStream. If there is a simple way to buffer the data output stream, I could use it, but the annoying part is the need to write out all the records, and at the end to count the number of triangles, output and write this integer in bytes 81-84.

A simple but slow way, just focusing on most of the work that writes every aspect:

public static void writeBinarySTL(Triangle t, RandomAccessFile d) throws IOException {
    d.writeFloat((float)t.normal.x);
    d.writeFloat((float)t.normal.y);
    d.writeFloat((float)t.normal.z);
    d.writeFloat((float)t.p1.x);
    d.writeFloat((float)t.p1.y);
    d.writeFloat((float)t.p1.z);
    d.writeFloat((float)t.p2.x);
    d.writeFloat((float)t.p2.y);
    d.writeFloat((float)t.p2.z);
    d.writeFloat((float)t.p3.x);
    d.writeFloat((float)t.p3.y);
    d.writeFloat((float)t.p3.z);
    d.writeShort(0);
}

- , /?

, STL , Java, , . , , , , little-endian?

, , randomaccess, 81 .

, , . - BufferedWriter. - . , SSD, , Java. , 96 , 0,5 , 196 1,5 .

, nio , @sturcotte06. , 50 .

public static void writeBinarySTL2(Shape3d s, String filename) {
    java.nio.file.Path filePath = Paths.get(filename);

    // Open a channel in write mode on your file.
    try (WritableByteChannel channel = Files.newByteChannel(filePath, StandardOpenOption.CREATE)) {
        // Allocate a new buffer.
        ByteBuffer buf = ByteBuffer.allocate(50 * 1024);
        ArrayList<Triangle> triangles = s.triangles;
        // Write your triangle data to the buffer.
        for (int i = 0; i < triangles.size(); i += 1024) {
            for (int j = i; j < i + 1024; ++j)
                writeBinarySTL(triangles.get(j), buf);
            buf.flip(); // stop modifying buffer so it can be written to disk
            channel.write(buf);  // Write your buffer data.
        }
        channel.close();
    } catch (IOException e) {
        e.printStackTrace();
    }
}

WRITE (, , ), CREATE, , , .

Sphere902_bin2.stl

java.nio.file.NoSuchFileException: Sphere902_bin2.stl
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:79)
        at 
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
        at 
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
        at

sun.nio.fs.WindowsFileSystemProvider.newByteChannel(WindowsFileSystemProvider.java:230)            java.nio.file.Files.newByteChannel(Files.java:361)            java.nio.file.Files.newByteChannel(Files.java:407)           at edu.stevens.scad.STL.writeBinarySTL2 (STL.java:105)

, , , , , :

public static void writeBinarySTL(Triangle t, ByteBuffer buf) {
    buf.putInt(Integer.reverseBytes(Float.floatToIntBits((float)t.normal.x)));
    buf.putInt(Integer.reverseBytes(Float.floatToIntBits((float)t.normal.y)));
    buf.putInt(Integer.reverseBytes(Float.floatToIntBits((float)t.normal.y)));
    buf.putInt(Integer.reverseBytes(Float.floatToIntBits((float)t.p1.x)));
    buf.putInt(Integer.reverseBytes(Float.floatToIntBits((float)t.p1.y)));
    buf.putInt(Integer.reverseBytes(Float.floatToIntBits((float)t.p1.z)));
    buf.putInt(Integer.reverseBytes(Float.floatToIntBits((float)t.p2.x)));
    buf.putInt(Integer.reverseBytes(Float.floatToIntBits((float)t.p2.y)));
    buf.putInt(Integer.reverseBytes(Float.floatToIntBits((float)t.p2.z)));
    buf.putInt(Integer.reverseBytes(Float.floatToIntBits((float)t.p3.x)));
    buf.putInt(Integer.reverseBytes(Float.floatToIntBits((float)t.p3.y)));
    buf.putInt(Integer.reverseBytes(Float.floatToIntBits((float)t.p3.z)));
    buf.putShort((short)0);
}

MWE, , :

package language;
import java.io.*;
import java.nio.*;
import java.nio.file.*;
import java.nio.channels.*;

public class FastWritenio {
    public static void writeUsingPrintWriter() throws IOException {
        PrintWriter pw = new PrintWriter(new FileWriter("test.txt"));
        pw.print("testing");
        pw.close();
    }
    public static void writeUsingnio(int numTrials, int bufferSize, int putsPer) throws IOException {
        String filename = "d:/test.dat";
        java.nio.file.Path filePath = Paths.get(filename);
        WritableByteChannel channel = Files.newByteChannel(filePath, StandardOpenOption.CREATE, StandardOpenOption.WRITE, StandardOpenOption.READ);
        ByteBuffer buf = ByteBuffer.allocate(bufferSize);
        for (int t = 0; t < numTrials; ++t) {
            for (int i = 0; i < putsPer; i ++) {
                buf.putInt(i);
            }
            buf.flip(); // stop modifying buffer so it can be written to disk
            channel.write(buf);  // Write your buffer data.
            // Without this line, it crashes:  buf.flip();
            // but with it this code is very slow.
        }
        channel.close();
    }
    public static void main(String[] args) throws IOException {
        writeUsingPrintWriter();
        long t0 = System.nanoTime();
        writeUsingnio(1024*256, 8*1024, 2048);
        System.out.println((System.nanoTime()-t0)*1e-9);

    }
}
+4
3

nio ByteBuffer:

public static void writeBinarySTL(Triangle t, ByteBuffer buf) {
    buf.putFloat((float)t.normal.x);
    // ...
}

// Create a new path to your file on the default file system.
Path filePath = Paths.get("file.txt");

// Open a channel in write mode on your file.
try (WritableByteChannel channel = Files.newByteChannel(filePath, StandardOpenOption.WRITE)) {
    // Allocate a new buffer.
    ByteBuffer buf = ByteBuffer.allocate(8192);

    // Write your triangle data to the buffer.
    writeBinarySTL(t, buf);

    // Flip from write mode to read mode.
    buf.flip();

    // Write your buffer data.
    channel.write(buf);
}

ByteBuffer

ByteBuffer javadoc

WriteableByteChannel javadoc

javadoc

EDIT:

public static boolean tryWriteBinarySTL(Triangle t, ByteBuffer buf) {
    final int triangleBytes = 50; // set this.
    if (buf.remaining() < triangleBytes) {
       return false;
    }

    buf.putFloat((float)t.normal.x);
    // ...
    return true;
}

// Create a new path to your file on the default file system.
Path filePath = Paths.get("file.txt");

// Open a channel in write mode on your file.
try (WritableByteChannel channel = Files.newByteChannel(filePath, StandardOpenOption.WRITE)) {
    // Allocate a new buffer.
    ByteBuffer buf = ByteBuffer.allocate(8192);

    // Write your triangle data to the buffer.
    for (Triangle triangle : triangles) {
        while (!tryWriteBinarySTL(triangle, buf) ) {
            // Flush buffer.
            buf.flip();
            while (buf.hasRemaining()) {
                channel.write(buf);
            }

            buf.flip();
        }
    }

    // Write remaining.
    buf.flip();
    channel.write(buf);
}
+5

:

DataOutputStream dos = new DataOutputStream(new BufferedOutputStream(...));
+5

DataOutputStreamnot a particularly quick fix. Create a simple old one FileOutputStream, place it around it BufferedOutputStream, and then write your own code to write your data in bytes. Classes Floatand Doublehave helper functions for this. doubleToLongBits , for example.

If this is not fast enough for you, then read on NIO2.

+1
source

Source: https://habr.com/ru/post/1688731/

More articles:


All Articles