Split a large file into pieces

Question

I have a method that takes a file and chunk size and returns a list of fragmented files. But the main problem is that my line in the file can be broken, for example, in the main file, I have the following lines:

|1|aaa|bbb|ccc|
|2|ggg|ddd|eee|

After splitting, I could have in one file:

|1|aaa|bbb

In another file:

|ccc|2|
|ggg|ddd|eee|

Here is the code:

public static List<File> splitFile(File file, int sizeOfFileInMB) throws    IOException {
  int counter = 1;
  List<File> files = new ArrayList<>();

  int sizeOfChunk = 1024 * 1024 * sizeOfFileInMB;
  byte[] buffer = new byte[sizeOfChunk];

  try (BufferedInputStream bis = new BufferedInputStream(new FileInputStream(file))) {
    String name = file.getName();

    int tmp = 0;
    while ((tmp = bis.read(buffer)) > 0) {
        File newFile = new File(file.getParent(), name + "."
                + String.format("%03d", counter++));
        try (FileOutputStream out = new FileOutputStream(newFile)) {
            out.write(buffer, 0, tmp);
        }

        files.add(newFile);
    }
  }

  return files;
}

Should I use the RandomAccessFile class for the above purposes (the main file is really large - more than 5 GB)?

Answer 1

If you do not mind having pieces of different lengths (<= sizeOfChunk, but closer to it), then here is the code:

public static List<File> splitFile(File file, int sizeOfFileInMB) throws IOException {
    int counter = 1;
    List<File> files = new ArrayList<File>();
    int sizeOfChunk = 1024 * 1024 * sizeOfFileInMB;
    String eof = System.lineSeparator();
    try (BufferedReader br = new BufferedReader(new FileReader(file))) {
        String name = file.getName();
        String line = br.readLine();
        while (line != null) {
            File newFile = new File(file.getParent(), name + "."
                    + String.format("%03d", counter++));
            try (OutputStream out = new BufferedOutputStream(new FileOutputStream(newFile))) {
                int fileSize = 0;
                while (line != null) {
                    byte[] bytes = (line + eof).getBytes(Charset.defaultCharset());
                    if (fileSize + bytes.length > sizeOfChunk)
                        break;
                    out.write(bytes);
                    fileSize += bytes.length;
                    line = br.readLine();
                }
            }
            files.add(newFile);
        }
    }
    return files;
}

- , ​​ . , . "splitFile" .