Need to speed up a C # application that converts timestamps

Question

im having a hard time getting a small application to work faster. I'm not a developer, and it took me a while to get this to work as it is. Can anyone offer any suggestions or alternative code to speed up this process, its adoption takes about 1 hour to process a 10 m input file.

the code is below and here is an example of an input file.

4401, imei: 0000000000,2012-09-01 12: 12: 12.9999

using System; using System.Globalization; using System.IO; class Sample { public static void Main(string[] args) { if (args.Length == 0) { return; } using (FileStream stream = File.Open(args[0], FileMode.Open, FileAccess.Read, FileShare.ReadWrite)) { using (StreamReader streamReader = new StreamReader(stream)) { System.Text.StringBuilder builder = new System.Text.StringBuilder(); while (!streamReader.EndOfStream) { var line = streamReader.ReadLine(); var values = line.Split(','); DateTime dt = new DateTime(); DateTime.TryParse(values[2], out dt); values[2] = Convert.ToString(dt.Ticks); string[] output = new string[values.Length]; bool firstColumn = true; for (int index = 0; index < values.Length; index++) { if (!firstColumn) builder.Append(','); builder.Append(values[index]); firstColumn = false; } File.WriteAllText(args[1], builder.AppendLine().ToString()); } } } } } 

Answer 1

The biggest performance hit is every time a line is read, the entire file (processed so far) is written back to disk. For a quick win, try moving the StringBuilder out of the loop:

  System.Text.StringBuilder builder = new System.Text.StringBuilder(); using (FileStream stream = File.Open(args[0], FileMode.Open, FileAccess.Read, FileShare.ReadWrite)) { using (StreamReader streamReader = new StreamReader(stream)) { while (!streamReader.EndOfStream) { var line = streamReader.ReadLine(); var values = line.Split(','); DateTime dt = new DateTime(); DateTime.TryParse(values[2], out dt); values[2] = Convert.ToString(dt.Ticks); string[] output = new string[values.Length]; bool firstColumn = true; for (int index = 0; index < values.Length; index++) { if (!firstColumn) builder.Append(','); builder.Append(values[index]); firstColumn = false; } builder.AppendLine(); } } } File.WriteAllText(args[1], builder.ToString()); 

If you want to reorganize a further change in the comma separation logic:

  System.Text.StringBuilder builder = new System.Text.StringBuilder(); using (FileStream stream = File.Open(args[0], FileMode.Open, FileAccess.Read, FileShare.ReadWrite)) { using (StreamReader streamReader = new StreamReader(stream)) { while (!streamReader.EndOfStream) { var line = streamReader.ReadLine(); var values = line.Split(','); DateTime dt = new DateTime(); DateTime.TryParse(values[2], out dt); values[2] = Convert.ToString(dt.Ticks); builder.AppendLine(string.Join(",", values)); } } } File.WriteAllText(args[1], builder.ToString()); 

Edit: To avoid memory usage, uninstall Stringbuilder and use another FileStream to write to disk. Your proposed solution (using List ) will still use a significant amount of memory and will probably break larger files:

  using (FileStream input = File.Open(args[0], FileMode.Open, FileAccess.Read, FileShare.ReadWrite)) using (FileStream output = File.Create(args[1])) { using (StreamReader streamReader = new StreamReader(input)) using (StreamWriter streamWriter = new StreamWriter(output)) { while (!streamReader.EndOfStream) { var line = streamReader.ReadLine(); var values = line.Split(','); DateTime dt = new DateTime(); DateTime.TryParse(values[2], out dt); values[2] = Convert.ToString(dt.Ticks); streamWriter.WriteLine(string.Join(",", values)); } } } 

Answer 2

here is what i found, i can fix it and process large files.

thanks to @Muzz and @Vache for the help.

 string line = ""; System.IO.StreamReader file = new System.IO.StreamReader("c:/test.txt"); List<string> convertedLines = new List<string>(); while ((line = file.ReadLine()) != null) { string[] lineSplit = line.Split(','); DateTime dt = new DateTime(); DateTime.TryParse(lineSplit[2], out dt); lineSplit[2] = Convert.ToString(dt.Ticks); string convertedline = lineSplit[0] + "," + lineSplit[1] + "," + lineSplit[2]; convertedLines.Add(convertedline); } file.Close(); File.WriteAllLines("c:/newTest.txt", convertedLines);