Using ThreadPool inside loops in C #

Question

I don’t know much about a thread, but let's say the following code (I’m more worried about using thread pools in a loop):

string[] filePaths = GetFilePaths(); foreach (string filePath in filePaths ) { ThreadPool.QueueUserWorkItem(DoStuff, filePath); } 

Is there any other way that this can be done?

EDIT:

NB Each execution of DoStuff, in turn, creates several subpions (about 200). These subtopics mimic system users and are only responsible for receiving and sending information via TCP.

Answer 1

Your current code may go wrong if a combination of the following two is executed:

• there are a lot of files.
• file processing (DoStuff) takes a considerable amount of time

Threadpool does not have enough capacity to balance the load and will create more and more threads that go beyond the optimal number.

If you can use Fx4, use TPL.

For earlier versions, rewrite your code to use fewer threads.

---

Edit since you are using Fx4:

The biggest gain may be to use System.Directory.EnumFiles() to replace Directory.GetFiles() .

Sketch:

 var files = System.Directory.EnumerateFiles(...); // deferred execution Parallel.ForEach(files, f => DoStuff(f)); // maybe use MaxDegree or CancelationToken // all files done here 

You can also wrap this. ForEach in one (one) try / catch, etc.

And if DoStuff() requires parallelism, you should also use TPL, perhaps go through a CancellationToken, etc. He would put all parallelism under the control of one scheduler.

You may need help fine-tuning, but it will also be much easier than without TPL.

Answer 2

You are right to worry. This current situation, depending on the number of elements in the loop, may exert too much pressure on the thread pool. I believe that I am incorrect here, in this situation the items will be queued, but as a result, most of the thread pool can be used, which will not necessarily lead to better performance.

Starting with .NET 4, there are some very simple alternatives to using a thread pool.

Parallel.ForEach will be relevant for you. Here is a link to the new parallelism features in .NET 4:

http://galratner.com/blogs/net/archive/2010/04/24/a-quick-lap-around-net-4-0-s-parallel-features.aspx

Update: by your right, which mentions 200 subtopics. I would advise that OS threads are not light objects. They have the overhead associated with them, and they can quickly offset any benefits from parallelism. Performing some tasks in parallel requires some consideration of how much work, what is the goal (to free the user interface, use all the cores, etc.), and whether it will ultimately work with processor intensity or with reference to IO. There are other factors, but I think they are very important. I would create another SO question describing what you are trying to solve using this level of parallelism so that you can get some design tips that are more specific to your problem.

Answer 3

It depends on what you are trying to execute: if you want the operations to be executed in a different thread, but not against the fact that they are performed in a strict order, you can simply do

 string[] filePaths = GetFilePaths(); ThreadPool.QueueUserWorkItem(DoStuff, filePaths); 

and put the foreach inside the DoStuff . This may be an acceptable solution depending on the expected values ​​of filePaths (for example, if all the paths are on the same device, trying to do it right away will not be faster, but maybe slower), and this is definitely the easiest.

If you definitely want to do them in parallel, then you should study the parallel task library (only .NET 4) and, in particular, Parallel.ForEach . Since limiting the number of simultaneous simultaneous tasks is a good idea, here is an example that shows how you can do this:

 var options = new ParallelOptions { MaxDegreeOfParallelism = 2 }; Parallel.ForEach(filePaths, options, i=> { DoStuff(i); }); 

Answer 4

Why don't you encapsulate your calls in Action<T> or delegate and put them in a thread queue in your loop. Then you can start (number) of threads, which will work until all actions from the queue are completed; This way you can control the number of threads used, and you don't need to worry about spawning too many of them.