I am running Delphi RAD Studio XE2. I recently played with file streams and found interesting results that led me to this question.
What is the optimal buffer size for TStreamReader in Delphi? For example, I upload a 1GB file of 2 million lines in the form of doubleTABdoubleTABdouble . If I load it into a TStringList using the following code, I get significantly different results for different buffer sizes. By the results, I mean the processing speed and the use of RAM.
reader := TStreamReader.Create(fileLocation, TEncoding.UTF8, True, NumBytes); try stringList.BeginUpdate; try stringList.Clear; while not reader.EndOfStream do stringList.Add(reader.ReadLine); finally stringList.EndUpdate; end; finally reader.Free; end; end;
The optimal buffer size seems to be between 1024 and 4096. If it is set to less than 1024, it seems to slow down linearly and seems to use more RAM. If it is set above 4096, it seems to slow down exponentially.
Why do I see this behavior and how to determine the optimal buffer size for a task? Also, what is the maximum buffer size?
Edit
I executed the following code to extract the runtime using the above file size:
startTime := Now(); myStreamReader := TStreamReader.Create(fileLocation, TEncoding.UTF8, True, numBytes); myStringList := TStringList.Create; try myStringList.BeginUpdate; try myStringList.Clear; while not myStreamReader.EndOfStream do myStringList.Add(myStreamReader.ReadLine); finally myStringList.EndUpdate; end; finally myStreamReader.Free; end; processTime := Now() - startTime; myStringList.Free;
An example runtime was retrieved as:
Buffer Size 32. Done in 69s Buffer Size 64. Done in 69s Buffer Size 96. Done in 69s Buffer Size 128. Done in 70s Buffer Size 160. Done in 60s Buffer Size 192. Done in 57s Buffer Size 224. Done in 52s Buffer Size 256. Done in 50s Buffer Size 512. Done in 44s Buffer Size 768. Done in 40s Buffer Size 1024. Done in 39s Buffer Size 1280. Done in 41s Buffer Size 1536. Done in 44s Buffer Size 1792. Done in 40s Buffer Size 2048. Done in 39s Buffer Size 2304. Done in 41s Buffer Size 2560. Done in 41s Buffer Size 2816. Done in 42s Buffer Size 3072. Done in 43s Buffer Size 3328. Done in 43s Buffer Size 3584. Done in 45s Buffer Size 3840. Done in 44s Buffer Size 4096. Done in 45s Buffer Size 4352. Done in 47s Buffer Size 4608. Done in 46s Buffer Size 4864. Done in 46s Buffer Size 5120. Done in 48s Buffer Size 5376. Done in 49s Buffer Size 5632. Done in 51s Buffer Size 5888. Done in 51s Buffer Size 6144. Done in 52s Buffer Size 6400. Done in 54s Buffer Size 6656. Done in 53s Buffer Size 6912. Done in 55s Buffer Size 7168. Done in 55s Buffer Size 7424. Done in 56s Buffer Size 7680. Done in 57s Buffer Size 7936. Done in 65s Buffer Size 8192. Done in 62s Buffer Size 8448. Done in 63s Buffer Size 8704. Done in 64s Buffer Size 8960. Done in 64s Buffer Size 9216. Done in 66s Buffer Size 9472. Done in 66s Buffer Size 9728. Done in 68s Buffer Size 9984. Done in 68s Buffer Size 10240. Done in 69s
Regarding the use of RAM, a buffer size below 256 has led to the full use of RAM with a capacity of 5 GB and buffer sizes above 1024 with a total use of about 3.5 GB. For example, the use of RAM with a buffer of 2kb, 4kb and 8kb; talk to:
