What is the wait behavior inside a Parallel.ForEach () loop?

Question

I have an intensive computing program that I am trying to parallelize, however one of the limiting steps is the I / O operation, which is controlled by a phenomenally inefficient API. I do not control, but I have no choice but to use. It is imperative that my parallelization does not increase the number of I / O operations, or that any advantage is likely to disappear very quickly.

The layout is something like this: I have two classes: Fooand Bar, and for a calculation Foothat does not require a lot of calculations, I have to pass it an instance or several instances Barthat I import from another file in an extremely expensive I / O operation. I need a large number of instances Fooand Bar, and many of these instances Barwill be used to compute more than one instance Foo. As a result, I don’t want to discard instances Barafter computing each Foo, and I don’t want to import them more than once. Potentially, it's harder to complicate the 32-bit API, while my program must be 64-bit to avoidMemoryException, so it is handled by a locally hosted server that I communicate with using WCF.

Here is my suggested solution, but I'm extremely new to parallelization, and in particular I'm not sure how it awaitwill be handled inside the ForEach wrt loop, freeing up the processors:

ConcurrentDictionary<string, Task<Bar>> barList = new ConcurrentDictionary<string, Task<Bar>>();

Parallel.ForEach(fooList, foo =>
{
    if (!barList.ContainsKey(this.RequiredBarName))
    {
        Task<Bar> importBar = Task.Run(() => Import.BarByName(this.RequiredBarName));
        barList.Add(this.RequiredBarName,importBar);
    }
    this.RequiredBarTask = barList.TryGetValue(this.RequiredBarName);
    foo.CalculateStuff();
}

// where foo.CalculateStuff() looks something like this
async public void CalculateStuff()
{
    // do some stuff...
    Bar requiredBar = await this.RequiredBarTask;
    // do some more stuff with requiredBar
}

What happens when the code runs in await? Will ThreadPool choose another Task, or is the processor just idle? If I then arrange something WaitAll()out Parallel.ForEach(), can I parallelize all this efficiently? Does anyone have any better ideas on how I can implement this?

Edit to provide MCVE:

, API, , , - , API, - .

, , , "". , Dataflow , .

Channel Channel, Parallel.ForEach(); , , .

public abstract class Class
{
    public string Name {get;set;}
    public float[] Data {get;set;}

    async public Task CalculateData(IsampleService proxy){}
}

public class Channel : Class
{
    public Class[] ChildClasses {get;set;}

    async public override Task CalculateData(IsampleService proxy)
    {
        foreach(Class childClass in ChildClasses)
        {
            // not the real processing but this step could be anything. There is a class to handle what happens here, but it is unnecessary for this post.
            if(childClass.Data==null) await childClass.CalculateData(proxy);
            this.Data = childClass.Data;
        }
    }
}

public class Input : Class
{
    async public override Task CalculateData(IsampleService proxy)
    {
            this.Data = await proxy.ReturnData(this.Name);
    }
}

async public static Task ProcessDataForExport(Channel[] channelArray)
{
ChannelFactory<IsampleService> factory = new ChannelFactory<IsampleService>(new NetNamedPipeBinding(), new EndpointAddress(baseAddress));

IsampleService proxy = factory.CreateChannel();

Parallel.ForEach(channelArray, channel =>
    {
        channel.CalculateData();
    });
// Task.WhenAll() might be a better alternative to the Parallel.ForEach() here.
}

Answer 1

, ?

, await: , , Task, . .

ThreadPool ?

, . , ? , , , sure & hellip; .

-, , , (, Parallel.ForEach() ), - .

, await , . , , (*). await , , , , , - .:)

(*) (, & hellip; , , , , :))

- WaitAll() Parallel.ForEach(), ? - , ?

, . , , await Parallel.ForEach() - . await, .

, Parallel.ForEach(), , , , . . , , Parallel , , .

, , . , , , , , , . .

MCVE, . API , , .