Efficient method of reading line strings from a file

Question

Assuming I have a log file of 15 GB and I would like to iterate over the lines \ n of completed lines from this file. The standard java standard / third-party version provides a clean interface for this operation.

Note that I'm looking for a NIO-based solution, preferably using the Memory Mapped file access method, as shown in this question How to create a Java string from the contents of a file? would be an ideal solution if it did not load the entire byte buffer into memory before returning a new String () instance of the buffer. This approach does not work in this case due to input size.

Thanks,
Maksim.

Answer 1

Do you find using BufferedReader ? From the documentation:

Reads text from a character input stream, buffering characters to ensure efficient reading of characters, arrays, and strings .

It has a clean interface for getting \n -terminated strings ( BufferedReader.readLine() ) and should be quite efficient as it is buffered.

Answer 2

IMHO you do not need NIO for this task. Use a regular BufferedReader:

BufferedReader reader = new BufferedReader(new FileReader("myfile.log"));

Then user reader.readLine() .

Answer 3

It is not based on NIO, but I would look at the Guava CharStreams.readLines (InputSupplier, LineProcessor) method . He does what you want, I would say:

 File file = ... Foo result = CharStreams.readLines(Files.newReaderSupplier(file, Charsets.UTF_8), new LineProcessor<Foo>() { public boolean processLine(String line) { // do stuff for this line return true; // or false if you want to stop processing here } public Foo getResult() { return result; // if you create some result when processing the lines } }); 

In this case, a callback is used to process each line in the file sequentially. It does not load the next line into memory until you process the current one. If you do not want to create any single result object when reading rows, you can simply use LineProcessor<Void> and have getResult() return null .