.net Main Problems of Parallel.ForEach

Question

I switched to .net Core for some projects and now I have a problem with Parallel.ForEach. Previously, I often had a list of id values ​​that I would then use to create web requests to get complete data. It will look something like this:

Parallel.ForEach(myList, l =>
{
    // make web request using l.id 
    // process the data somehow
});

Well, in .net Core, web requests should be tagged await, which means that the Parallel.ForEach action needs to be tagged async. But, marking the action of Parallel.ForEach as async, we have a method void asyncthat causes problems. In my specific case, this means that the response is returned to the application before all web requests in the Parallel loop are completed, which is inconvenient and causes errors.

Question: What are the alternatives to using Parallel.ForEach here?

One of the possible solutions that I found is to wrap the Parallel loop inside the task and wait for the task:

await Task.Run(() => Parallel.ForEach(myList, l =>
{
    // stuff here
}));

(found here: Parallel .ForEach and Task.Run and Task.WhenAll )

But this does not work for me. When I use this, I still get back to the application until the loop ends.

Another option:

var tasks = new List<Task>();
foreach (var l in myList)
{
    tasks.Add(Task.Run(async () =>
    {
         // stuff here
    }));
}
await Task.WhenAll(tasks);

It works, but is it the only option? It seems that the new .net Core has made Parallel.ForEach practically useless (at least when it comes to nested web calls).

Any help / advice is appreciated.

Answer 1

None of these 3 apporaches are good.

You should not use the class in this scenario Paralleleither Task.Run.

async:

private async Task HandleResponse(Task<HttpResponseMessage> gettingResponse)
{
     HttpResponseMessage response = await gettingResponse;
     // Process the data
}

Task.WhenAll:

Task[] requests = myList.Select(l => SendWebRequest(l.Id))
                        .Select(r => HandleResponse(r))
                        .ToArray();

await Task.WhenAll(requests);

Answer 2

Parallel.ForEach , : . - (, -) - , , . , .

- (, HttpWebRequest.GetResponseAsync), - - ( ). URL-. , , , , Semaphore. Semaphore - X, , ( ). :

static async Task ProcessUrls(string[] urls) {
    var tasks = new List<Task>();
    // semaphore, allow to run 10 tasks in parallel
    using (var semaphore = new SemaphoreSlim(10)) {
        foreach (var url in urls) {
            // await here until there is a room for this task
            await semaphore.WaitAsync();
            tasks.Add(MakeRequest(semaphore, url));
        }
        // await for the rest of tasks to complete
        await Task.WhenAll(tasks);
    }
}

private static async Task MakeRequest(SemaphoreSlim semaphore, string url) {
    try {
        var request = (HttpWebRequest) WebRequest.Create(url);

        using (var response = await request.GetResponseAsync().ConfigureAwait(false)) {
            // do something with response    
        }
    }
    catch (Exception ex) {
        // do something
    }
    finally {
        // don't forget to release
        semaphore.Release();
    }
}

Answer 3

, ref, , . .

Parallel.ForEach(myList, l =>
{

    // make web request using ref l.id 
    string id=l.id;
    WebRequest webRequest= MakeRequest(ref id);
    // process the data somehow
});

private WebRequest MakeRequest(ref string id)
{
  //make and return web request
}

Answer 4

I assume this code will work:

for (int i = 0; i < myList.Length; i++)
{
    var item = myList[i];

    var msg = await SendAsync(item.Id);
    //Post Process
}