C # Multithreaded IO file (read)

Question

We have a situation where our application needs to process a number of files and instead execute synchronously, we would like to use multithreading to divide the workload between different threads.

Each work item:
1. Open the file for reading only
2. Process data in a file
3. Write processed data to the dictionary

We would like each file to work on a new stream? Is this possible, and we must use ThreadPool or create new threads, bearing in mind that each element of the "work" takes only 30 ms, but it is possible that hundreds of files will need to be processed.

Any ideas to make this more effective are welcome.

EDIT: We are currently using ThreadPool for this. If we have 500 files to process, we cycle through the files and allocate each “processing unit” in threadpool using QueueUserWorkItem.

Can threadpool be used for this?

Answer 1

I would suggest you use ThreadPool.QueueUserWorkItem(...), in this, the flows are controlled by the .net system and infrastructure. The chances of you starting your own flow are much higher. Therefore, I would recommend that you use Threadpool provided by .net. It is very easy to use,

ThreadPool.QueueUserWorkItem(new WaitCallback(YourMethod), ParameterToBeUsedByMethod); 

YourMethod(object o){ Your Code here... }

, , http://msdn.microsoft.com/en-us/library/3dasc8as%28VS.80%29.aspx

,

Answer 2

(, 4), 4 . 400 , 100 . , .

-, , , .

Answer 3

, , , Parallel Extensions (PEX)

var filesContent = from file in enumerableOfFilesToProcess
                   select new 
                   {
                       File=file, 
                       Content=File.ReadAllText(file)
                   };

var processedContent = from content in filesContent
                       select new 
                       {
                           content.File, 
                           ProcessedContent = ProcessContent(content.Content)
                       };

var dictionary = processedContent
           .AsParallel()
           .ToDictionary(c => c.File);

PEX , - ( , !)

PEX .NET Framework 4.0, back-port to 3.5 Reactive Framework.

Answer 4

CCR (Concurrency ), . , , , , .

CCR, Interleave :

Arbiter.Activate(dispatcherQueue, Arbiter.Interleave(
    new TeardownReceiverGroup(Arbiter.Receive<bool>(
        false, mainPort, new Handler<bool>(Teardown))),
    new ExclusiveReceiverGroup(Arbiter.Receive<object>(
        true, mainPort, new Handler<object>(WriteData))),
    new ConcurrentReceiverGroup(Arbiter.Receive<string>(
        true, mainPort, new Handler<string>(ReadAndProcessData)))));

public void WriteData(object data)
{
    // write data to the dictionary
    // this code is never executed in parallel so no synchronization code needed
}

public void ReadAndProcessData(string s)
{
    // this code gets scheduled to be executed in parallel
    // CCR take care of the task scheduling for you
}

public void Teardown(bool b)
{
    // clean up when all tasks are done
}

Answer 5

, , , . .

• , , .
• . Queue.
• , . , , . . Dictionary , ManagedThreadId.
• , .
• , .

Answer 6

ThreadPool.QueueUserWorkItem . . .

Answer 7

ThreadPool , , ( Mutexes ) .

, ? , ThreadPool - . , , .

ThreadPool , . .

Hth

Answer 8

ThreadPool - . , , , . , , ThreadPool. ThreadPool, ~ 100. 400 , , ~ 100 , CPU. ,.NET Framework , , ThreadPool , , , ( -). ThreadPool , . , :

• System.Threading.Thread( SINGLE ThreadPool) ,

• FileStream BeginRead BeginWrite -. .NET API IO (IOCP).

2 , , , . -, , , .

, ... - 10-15% CPU, , , 80% + . . ThreadPool IOCP , , , , , , .

, , - , (50+ ) , 35 . , MSDN SocketAsyncEventArgs, , x , , .

, - :)