Parallel.ForEach behavior changed?

Question

I have a report building class that uses parallel processing to create a batch of 800 reports.

Until yesterday, it worked fine with the following code:

Parallel.ForEach(reports, report => { this.Build(report); }); 

Note: for debugging purposes, I exclude code from a test written in a test project.

Yesterday a crash started. Digging a little with Resource Monitor showed that the number of threads associated with qtagent32.exe (test runner) was constantly increasing and ultimately led to a process failure. It seems that something suddenly began to block.

I managed to fix the problem with a small change to my code:

 Parallel.ForEach(reports, new ParallelOptions { MaxDegreeOfParallelism = 4 }, report => { this.Build(report); }); 

But I still wonder what has changed?

Answer 1

It only takes one item to get a little slower. If it’s really “the same code,” that is, the same reports, it may have been a database. A little more data can make the request a little slower.

TPL sits on top of ThreadPool, and ThreadPool will slowly add new threads.

Suppose one of your reports takes longer than a threshold (500 ms), then TP will create an additional stream. If your reports are competing for something (most likely disk I / O) than this additional stream, the chances are that other reports will exceed the threshold. The result is an avalanche.

The new Fx4 Threadpool is smarter than the previous one, but it is still a rough heuristic. Your MaxDegreeOfParallelism is the right solution.