How can I read a large file gradually?

Question

I am having trouble reading a file using java. This is absolutely huge (2.5G), and setting my memory doesn't help. All data is on one line, so I can’t read it one line at a time. What I would like to do is read the file until I find a specific line, for example "<| start |>" or "<| end |>", and then print the data between these lines to clear the memory and I can continue reading the rest of the file. So what I'm mostly looking for is a reader type that starts reading from a specific start line and stops reading the stop line. Can anybody help me?

Answer 1

You need to open the Reader (for example, a BufferedReader FileInputStream InputStreamReader FileInputStream ) and read the fragments at a time using read(char[], int, int) or read(char[]) . It is up to you to take care to find the token - including when it starts in one piece and ends in another. Also keep in mind that read() may not fill the buffer; you need to use the return value to see how much data it actually wrote to the array.

Answer 2

I would see if Scanner is suitable for your data. You can use the useDelimiter method to change the patterns used to tokenize input.

Answer 3

Try this pseudo code:

  char [] start = {'<','|','s','t','a','r','t','|','>' }; char [] start = {'<','|','e','n','d','|','>' }; char [] buff = new char[9]; while( true ) { char c = readChar(); if( c == '<' ) { buff = readChars( 9 ) ; if( buff == start ) { inside = true ; skip( 9 ); // start } else if( buff == end ) { inside = false; skip(7); // end } } if( inside ) { print( char ) ; } } 

The idea is to read until you find the marker and raise the flag, when the flag is on, you print the value, if you find the final token, you end the flag.

There must be several ways to encode the previous pseudocode. I will update this answer later.