Multiple I / O Files

Question

Sorry for the chronic name.

I have a project that I am working on and I would appreciate any suggestions as to how I should do IO stuff.

Ok, I have 3 text files. One file contains many lines of text. This is the same for the other two files. Let them be called File1, File2, and File3.

I need to create a text file, for explanation, I will call it Result.txt.

Here's what you need to do:

• Extract the first line of text from File1 and add it to Result.txt.

• Extract the first line of text from File2 and add it to the end of the first line in Result.txt.

• Extract the first line of text from File3 and add it to the end of the first line in Result.txt.

• Create a new line in Result.txt

• Repeat from 1 to 4.

Note. These files can be quite large.

Anyone have any ideas on how best to approach this?

thanks

-

Thank you all for your very helpful answers. I learned a lot from your tips and code samples!

Answer 1

I think you can use the producer / consumer philosophy here. You can have a stream (producer) reading each line from your three source files, combining 3 lines and placing the result in a queue (in memory). Meanwhile, another thread (consumer) constantly reads from this queue and writes the result.txt file.

1: producer thread
   Reads line n from file 1,2 and 3
   concatenates the contents of the 3 lines and push_back in the queue

2: consumer thread
   Check if the queue is empty. 
   If not, pop the first item in the queue and write to the result.txt

Answer 2

int i = 0;
using (StreamWriter result = new StreamWriter("result.txt"),
    StreamReader file1 = new StreamReader("file1.txt"),
    StreamReader file2 = new StreamReader("file1.txt"),
    StreamReader file3 = new StreamReader("file1.txt"))
{
    while(file1.Peek() != -1)
    {
        result.Write(file1.ReadLine());
        result.Write(file2.ReadLine());
        result.WriteLine(file3.ReadLine());
        if (i++ % 100 == 0) result.Flush();
    }
}

Answer 3

:

using (StreamWriter result = new StreamWriter("result.txt"))
{
    StreamReader file1 = new StreamReader("file1.txt");
    StreamReader file2 = new StreamReader("file2.txt");
    StreamReader file3 = new StreamReader("file3.txt");
    while (!file1.EndOfStream || !file2.EndOfStream || !file3.EndOfStream)
    {
        result.Write(file1.ReadLine() ?? "");
        result.Write(file2.ReadLine() ?? "");
        result.WriteLine(file3.ReadLine() ?? "");
    }
}

- , :

• : ,
• , StringBuilder
• StringBuilder , n , .
• , async.
• Do this until both threads are done.

You need to learn how to synchronize two streams, but it's fun, and in my particular case we got a good performance boost.

EDIT : new version for a copy of Yuri:

object locker = new object();
using (StreamWriter result = new StreamWriter("result.txt"))
{
    StreamReader file1 = new StreamReader("file1.txt");
    StreamReader file2 = new StreamReader("file2.txt");
    StreamReader file3 = new StreamReader("file3.txt");

    const int SOME_MAGICAL_NUMBER = 102400; // 100k?
    Queue<string> packets = new Queue<string>();
    StringBuilder buffer = new StringBuilder();
    Thread writer = new Thread(new ThreadStart(() =>
    {
        string packet = null;
        while (true)
        {
            Monitor.Wait(locker);
            lock (locker)
            {
                packet = packets.Dequeue();
            }
            if (packet == null) return;
            result.Write(packet);
        }
    }));
    writer.Start();

    while (!file1.EndOfStream || !file2.EndOfStream || !file3.EndOfStream)
    {
        buffer.Append(file1.ReadLine() ?? "");
        buffer.Append(file2.ReadLine() ?? "");
        buffer.AppendLine(file3.ReadLine() ?? "");

        if (buffer.Length > SOME_MAGICAL_NUMBER)
        {
            lock (locker)
            {
                packets.Enqueue(buffer.ToString());
                buffer.Length = 0;
                Monitor.PulseAll(locker);
            }
        }
    }

    lock (locker)
    {
        packets.Enqueue(buffer.ToString());
        packets.Enqueue(null); // done
        Monitor.PulseAll(locker);
    }
    writer.Join();
}

Answer 4

It looks pretty simple. Using binary reading instead of text (line by line) can speed up the process.