Parallel.ForEach behaves as usual for everyone towards the end of the iteration

Question

I have this problem when I run something like this:

Parallel.ForEach(dataTable.AsEnumerable(), row =>
{
   //do processing
}

Assuming 500+ entries say 870. As soon as Parallel.ForEach completes 850, it seems to work sequentially, that is, only 1 operation at a time. He completed 850 operations very quickly, but when he approaches the end of the iteration, he becomes very slow and seems to perform as regular for everyone. I even tried for 2,000 entries.

Is there something wrong in my code? Please give some advice.

Below is the code I'm using

Sorry, I just wrote the wrong example. This is the correct code:

Task newTask = Task.Factory.StartNew(() =>
{
    Parallel.ForEach(dtResult.AsEnumerable(), dr =>
    {
        string extractQuery = "";
        string downLoadFileFullName = "";
        lock (foreachObject)
        {

            string fileName = extractorConfig.EncodeFileName(dr);
            extractQuery = extractorConfig.GetExtractQuery(dr);
            if (string.IsNullOrEmpty(extractQuery)) throw new Exception("Extract Query not found. Please check the configuration");

            string newDownLoadPath = CommonUtil.GetFormalizedDataPath(sDownLoadPath, uKey.CobDate);
            //create folder if it doesn't exist
            if (!Directory.Exists(newDownLoadPath)) Directory.CreateDirectory(newDownLoadPath);
            downLoadFileFullName = Path.Combine(newDownLoadPath, fileName);
        }
        Interlocked.Increment(ref index);

        ExtractorClass util = new ExtractorClass(SourceDbConnStr);
        util.LoadToFile(extractQuery, downLoadFileFullName);
        Interlocked.Increment(ref uiTimerIndex);
    });
});

Answer 1

:

IO:

• Database +

-. , , , IO . , , . , , .

, :

var options = new ParallelOptions { MaxDegreeOfParallelism = 2 };

Parallel.ForEach(dtResult.AsEnumerable(), options, dr =>
{
    //Do stuff
});

, :

• - .
• .

:

private static bool isDirectoryCreated;

//...

var options = new ParallelOptions { MaxDegreeOfParallelism = 2 };

Parallel.ForEach(dtResult.AsEnumerable(), options, dr =>
{
    string fileName, extractQuery, newDownLoadPath;

    lock (foreachObject)
    {
        fileName = extractorConfig.EncodeFileName(dr);

        extractQuery = extractorConfig.GetExtractQuery(dr);

        if (string.IsNullOrEmpty(extractQuery))
            throw new Exception("Extract Query not found. Please check the configuration");

        newDownLoadPath = CommonUtil.GetFormalizedDataPath(sDownLoadPath, uKey.CobDate);

        if (!isDirectoryCreated)
        {
            if (!Directory.Exists(newDownLoadPath))
                Directory.CreateDirectory(newDownLoadPath);

            isDirectoryCreated = true;
        }
    }

    string downLoadFileFullName = Path.Combine(newDownLoadPath, fileName);

    Interlocked.Increment(ref index);

    ExtractorClass util = new ExtractorClass(SourceDbConnStr);
    util.LoadToFile(extractQuery, downLoadFileFullName);

    Interlocked.Increment(ref uiTimerIndex);
});

Answer 2

, ..NET , .

- , . , , , .

, PLinq, , .

Answer 3

, parallelism . ( ) , Parallel.ForEach . , , . , 850 , , Thread 1 425 , Thread 2 - 425 . . : [0, 425, 426, 1, 2, 427, 3, 428, 429, 4,...].

(, ), , .

, , - , , , , . [0, 1, 2, 4, 3, 6, 5,...].

. , .

, . , , .

Parallel.ForEachimplemented in the first way: each thread is assigned its own group of elements for processing. For this, in another way, more overhead would be required, because the list of elements would have to be processed as a general queue with the resulting synchronization costs.