Writing to a file in the background

Question

For a method that writes to a text file

public void WriteToFile( ) { var file = "C:\\AsyncTest.txt"; var writer = File.Exists( file ) ? File.AppendText( file ) : File.CreateText( file ); writer.WriteLine( "A simulated entry" ); writer.Close(); } 

I need to simulate a scenario in which this method could be called in a loop, perhaps dozens of times, and should run asynchronously.

So I tried calling the method in a new thread like this (where writer is the class in which WriteToFile lives)

 //in a loop... Thread thread = new Thread( writer.WriteToFile ); thread.Start( ); 

Which works fine once, but throws an IO exception that the file is used by another process in subsequent iterations. Which really makes sense, actually, but I don't know how to get around this.

I tried using Join () like this

 Thread thread = new Thread( writer.WriteToFile ); thread.Start( ); thread.Join(); 

But this blocks the calling thread until all the merged threads have completed, what kind of target hit is not?

I tried using ThreadPool.QueueUserWorkItem(writer.WriteToFile); but getting the same IO exception.

I tried to use lock

 private object locker = new object(); public void WriteToFile( ) { lock(locker){ //same code as above } } 

But it had no visible effect.

I also tried using the Task class to no avail.

So, how can I “stack” these background streams for writing to a single file without conflicts without blocking the calling stream?

Answer 1

You can use something like:

 // To enqueue the write ThreadPool.QueueUserWorkItem(WriteToFile, "A simulated entry"); // the lock private static object writeLock = new object(); public static void WriteToFile( object msg ) { lock (writeLock) { var file = "C:\\AsyncTest.txt"; // using (var writer = File.Exists( file ) ? File.AppendText( file ) : File.CreateText( file )) { // As written http://msdn.microsoft.com/it-it/library/system.io.file.appendtext(v=vs.80).aspx , File.AppendText will create the // file if it doesn't exist using (var writer = File.AppendText( file )) { writer.WriteLine( (string)msg ); } } } 

And please use using with files!

Answer 2

Another option is to create a queue. Let the main thread put the lines in the queue and have a constant background thread that reads the queue and writes to the file. It is really easy to do.

 private BlockingCollection<string> OutputQueue = new BlockingCollection<string>(); void SomeMethod() { var outputTask = Task.Factory.StartNew(() => WriteOutput(outputFilename), TaskCreationOptions.LongRunning); OutputQueue.Add("A simulated entry"); OutputQueue.Add("more stuff"); // when the program is done, // set the queue as complete so the task can exit OutputQueue.CompleteAdding(); // and wait for the task to finish outputTask.Wait(); } void WriteOutput(string fname) { using (var strm = File.AppendText(filename)) { foreach (var s in OutputQueue.GetConsumingEnumerable()) { strm.WriteLine(s); // if you want to make sure it written to disk immediately, // call Flush. This will slow performance, however. strm.Flush(); } } } 

The background thread waits for a wait in the output queue, so it does not use CPU resources, unless it actually outputs data. And since other threads just need to queue something, there is essentially no expectation.

See my blog, Simple Multithreading, Part 2 for more information.

Answer 3

You can handle the lock as you tried, but you need to include the using statement:

 private readonly object _lockObj = new Object(); public void WriteToFile( ) { var file = "C:\\AsyncTest.txt"; lock (_lockObj) { using (StreamWriter writer = File.AppendText( file )) { writer.WriteLine( "A simulated entry" ); } } } 

In addition, you do not need to use CreateText , because AppendText will create the file if it does not exist. Finally, I had problems with this code, and also that the lock would be released before Windows releases the resource. This is rare, but it happens, so I just add a little retry logic looking for this particular exception.

Answer 4

• Dispose of your threads.
• Use the features of TPL or async / await.

For instance:

 Task.Run(() => File.WriteAllText("c:\\temp\test.txt", "content")); 

This will start the write operation asynchronously, without worrying about threads.

In addition, streams and stream recordings provide WriteAsync methods that you can use and wait for.

UPDATE

To avoid "blocking" the problem, just do not do blocking :) Blocking occurs if you try to write to the same file from different streams. You can use the File.Open () methods and specify the mode so that it blocks the stream and waits until the file is writable.

But blocking is bad. Therefore, I advise you, if you want to write from multiple threads, create a queue and put your write tasks in this queue. You can safely set multiple threads (use ConcurrentQueue<T> ). Then you consume this queue in the background task and simply write what you have in the queue into your file - one element at a time.

What it is: multiple publishers, one user [file recording], super lightweight, no locks required.