Waiting for a lot of tasks

Question

I have a set of Task (there are a lot of them, about 400):

 IEnumerable<Task> tasks = ... 

I want to run them all at the same time and then wait for each of them. I use this piece of code to complete tasks:

 Task.Run(async () => { ... }); 

Each of the tasks will run asynchronous methods on its own, and so I need the async in lambda. Among these nested tasks, there are well-known HTTP requests that are sent, and HTTP responses that are received.

I tried two different ways to wait for all tasks to complete:

 await Task.WhenAll(tasks); 

and

 foreach (var task in tasks) { await task; } 

Which, a priori, look the same for me (but, of course, they seem to be wrong, I would not publish here in the first place ...).

The first method makes tasks run faster, but there are tons of A first chance exception of type 'System.Net.Sockets.SocketException' occurred in System.dll and other similar ones in the output window. Moreover, some tasks are still in the WaitingForActivation state after calling await Task.WhenAll() .

The second method is slower, and it seems that the tasks are not running at the same time (I get HTTP responses one by one, and the first method of waiting for tasks makes them come almost all at the same time). In addition, I see no first chance exception at all in the output window, when I use the foreach to wait for each task, and no task has a WaitingForActivation state after the loop.

I understand that the “best” way to wait for a set of tasks is to use WhenAll() (at least for reading), but why do these two methods behave differently? How can I solve this problem? Ideally, I would like the tasks to be executed quickly and be sure that everything is over (I have a try catch finally catch in lambda to handle server error, and I did not forget if(httpClient != null) httpClient.Dispose() in finally before anyone asks ...).

Any hints are welcome!

EDIT

Ok, I tried another. I added:

 .ContinueWith(x => System.Diagnostics.Debug.WriteLine("#### ENDED = " + index))); 

For each task, index is the Task number. When using the foreach I get:

 #### ENDED = 0 #### ENDED = 1 #### ENDED = 2 #### ENDED = 3 #### ENDED = 4 ... 

When using WhenAll() I get:

 #### ENDED = 1 #### ENDED = 3 #### ENDED = 0 #### ENDED = 4 #### ENDED = 8 ... 

Thus, using the foreach , all my tasks are executed synchronously ..., which perhaps explains why I do not get any first chance exception in the output window, since the system is not criticized at all by the algorithm.

EDIT2:

Sample code: http://pastebin.com/5bMWicD4

The public service is used here: http://timezonedb.com/

Answer 1

Two attempts are completely different.

The first attempt expects completion of all tasks and continuation of work. He will throw only after completing all tasks. The order of the results is uncertain and depends on which task ends first .

The second is waiting for each task one at a time, in the order in which they were placed in the task array, which, of course, is not what you want, and rather slow. It interrupts the wait with an exception if even one task fails. Other tasks will be lost.

This is not EXACTLY how to complete tasks synchronously, as some tasks will complete earlier than others, but you still have to check them all one at a time.

Here you should notice that Task.WhenAll does not block by itself. It returns a task that ends when all other tasks are completed. Calling await Task.WhenAll , you are waiting for the completion of this task. You can check this task status to make sure one or more subtasks have failed or been canceled, or process the results by calling ContinueWith.

You can also call Task.WaitAll instead of await Task.WhenAll to lock until all tasks have finished, or at least one of them cancels or cancels. This is somewhat similar to your second attempt, although it still avoids waiting for all tasks one by one.

The fact that you have many exceptions has nothing to do with how you expect. There are restrictions on the number of HTTP connections that you can make in the same domain (for example, an address) at a time, there may be timeout errors (usually caused by the connection limit) or other network-related problems.

The type of exceptions you get depends on whether you call await Task.WhenAll or Task.WaitAll . This post explains the problem , but, in short, Task.WaitAll will Task.WaitAll all exceptions and throw an AggregateException, and await Task.WhenAll will return only one of them.

By the way, what message do you get for a SocketException?

Answer 2

The behavior of your code has nothing to do with await . This is due to the way you iterate over the Task s collection. Most LINQ methods are lazy, which means that they actually execute their code only when you iterate through it.

So, this code runs each Task only after the previous one has completed:

 foreach (var task in tasks) { await task; } 

but this code runs all at once:

 foreach (var task in tasks.ToList()) { await task; } 

And since Task.WhenAll() does an internal equivalent ToList() , you get the same behavior as the second snippet above.