How to efficiently make 1000 web requests as fast as possible

Question

I need to make 100,000 simple (i.e. small Content-Length) web requests from a C # console application. What is the fastest way that I can do this (i.e., complete all requests as soon as possible) and what are the best practices to follow? I can’t shoot and forget, because I need to capture the answers.

Presumably, I would like to use async web query methods, however, I am wondering what impact the overhead on saving all Task extensions and sorting will be.

Memory consumption is not a common problem, the goal is speed.

Presumably, I would also like to use all available kernels.

So, I can do something like this:

 Parallel.ForEach(iterations, i => { var response = await MakeRequest(i); // do thing with response }); 

but it won’t make me faster than just the number of cores ...

I can do:

 Parallel.ForEach(iterations, i => { var response = MakeRequest(i); response.GetAwaiter().OnCompleted(() => { // do thing with response }); }); 

but how to save your program after ForEach . Holding all the Tasks and WhenAll , they seem bloated, do existing templates or helpers have any kind of task queue?

Is there a way to get better, and how should I handle throttling / error detection? For example, if the remote endpoint responds slowly, I do not want to continue sending it.

I understand that I also need:

 ServicePointManager.DefaultConnectionLimit = int.MaxValue 

Anything else needed?

Answer 1

The Parallel class does not work with async loop bodies, so you cannot use it. The body of the loop completes completely and returns the task. There is no advantage to parallelism.

This is a very simple problem. Use one of the standard solutions for processing a number of elements asynchronously with this DOP (this is good: http://blogs.msdn.com/b/pfxteam/archive/2012/03/05/10278165.aspx . Use the last code snippet).

You need to empirically determine the correct DOP. Just try different values. There is no theoretical way to get the best value, because it depends on many things.

The connection limit is the only limit that is in your way.

response.GetAwaiter (). Oncomppleted

Not sure what you tried to do there ... If you comment, I will explain the misunderstanding.

Answer 2

The operation you want to perform is

• Call I / O method
• Process result

You are correct that you should use the async version of the I / O method. What more, you only need 1 thread to start all the I / O operations. You cannot take advantage of parallelism here.

In the second part, you will get parallelism - processing the result, since it will be a processor-bound operation. Fortunately, async / await will do all the work for you. Console applications do not have a synchronization context. This means that part of the method after await will be executed in the thread pool thread, optimally using all CPU cores.

 private async Task MakeRequestAndProcessResult(int i) { var result = await MakeRequestAsync(); ProcessResult(result); } var tasks = iterations.Select(i => MakeRequestAndProcessResult(i)).ToArray(); 

To achieve the same behavior in an environment with a synchronization context (for example, WPF or WinForms), use ConfigureAwait(false) .

 var result = await MakeRequestAsync().ConfigureAwait(false); 

To wait for tasks to complete, you can use await Task.WhenAll(tasks) inside the async or Task.WaitAll(tasks) method in Main() .

Throwing 100,000 requests in a web service is likely to kill him, so you have to limit it. You can check the answers to this question to find some options on how to do this.

Answer 3

Parallel.ForEach should be able to use more threads than there are cores if you explicitly set the MaxDegreeOfParallelism property of the ParallelOptions parameter (in the ForEach overload where this parameter is) - see https://msdn.microsoft.com/en-us/library /system.threading.tasks.paralleloptions.maxdegreeofparallelism(v=vs.110).aspx

You can set this to 1000 to force it to use 1000 threads or more, but it can be inefficient due to the overhead of the threads. You can experiment (for example, a cycle from 100 to 1000 steps in 100 seconds to try to send 1000 requests each time and time start to the end) or even set up some kind of self-tuning algorithm.