I had a similar problem with this guy while processing a 4 MB log file . I actually process several files at the same time, but since I keep getting this exception, I decided to just test it for one file:
val temp = Source.fromFile("./datasource/input.txt")
val dummy = new PrintWriter("test.txt")
var itr = 0
println("Default Buffer size: " + Source.DefaultBufSize)
try {
for( chr <- temp) {
dummy.print(chr.toChar)
itr += 1
if(itr == 75703) println("Passed line 85")
if(itr % 256 == 0){ print("..." + itr); temp.reset; System.gc; }
if(itr == 75703) println("Passed line 87")
if(itr % 2048 == 0) println("")
if(itr == 75703) println("Passed line 89")
}
} finally {
println("\nFalied at itr = " + itr)
}
What I always get is that it will fail at itr = 75703, while my output file will always be 64 KB (more precisely 65536 bytes). No matter where I put temp.reset or System.gc, all experiments end up the same.
It seems that the problem depends on the memory allocation, but I can not find useful information about this problem. Any idea on how to solve this problem?
All your help is much appreciated.
EDIT: , , BufferedInputStream.