Which consumes less resources and faster File.AppendText or File.WriteAllText stores first in StringBuilder?

Question

I need to write thousands of dynamically generated lines to a text file. I have two options that consumes less resources and faster than the other?

a. Using StringBuilder and File.WriteAllText

StringBuilder sb = new StringBuilder(); foreach(Data dataItem in Datas) { sb.AppendLine(String.Format("{0}, {1}-{2}",dataItem .Property1,dataItem .Property2,dataItem .Property3)); } File.WriteAllText("C:\\example.txt", sb.ToString(), new UTF8Encoding(false)); 

B. Using File.AppendText

 using(StreamWriter sw = File.AppendText("C:\\example.txt")){ foreach (Data dataItem in Datas) { sw.WriteLine(String.Format("{0}, {1}-{2}",dataItem .Property1,dataItem .Property2,dataItem .Property3)); } } 

Answer 1

Your first version, which puts everything in a StringBuilder and then writes it, will consume most of the memory. If the text is very large, you may have a shortage of memory. It may be faster, but it may also be slower.

The second option will use much less memory (basically, only the StreamWriter buffer) and will work very well. I would recommend this option. It works well - perhaps better than the first method - and does not have the same potential for running out of memory.

You can speed it up significantly by increasing the size of the output buffer. Instead

 File.AppendText("filename") 

Create a stream with:

 const int BufferSize = 65536; // 64 Kilobytes StreamWriter sw = new StreamWriter("filename", true, Encoding.UTF8, BufferSize); 

A 64K buffer size gives much better performance than the default 4K buffer size. You can go more, but I found that more than 64K gives minimal performance, and on some systems it can actually slow down performance.

Answer 2

You have at least one other choice using File.AppendAllLines()

 var data = from item in Datas select string.Format("{0}, {1}-{2}", item.Property1, item.Property2, item.Property3); File.AppendAllLines("Filename", data, new UTF8Encoding(false)); 

This will theoretically use less memory than your first approach, since only one line at a time is buffered in memory.

Most likely, it will be almost the same as your second approach. I will just show you the third option. The only advantage of this is that you can give it a Linq sequence, which can be useful sometimes.

I / O speed will overshadow any other considerations, so you should focus on minimizing memory usage, as described above, and also taking into account the dangers of premature optimization!)

This means using your second approach or the one I put here.