Download multiple files quickly and efficiently (async)

Question

I have so many files that I need to download. Therefore, I am trying to take advantage of the new asynchronous functions, as shown below.

var streamTasks = urls.Select(async url => (await WebRequest.CreateHttp(url).GetResponseAsync()).GetResponseStream()).ToList();

var streams = await Task.WhenAll(streamTasks);
foreach (var stream in streams)
{
    using (var fileStream = new FileStream("blabla", FileMode.Create))
    {
        await stream.CopyToAsync(fileStream);
    }
}

What am I afraid of this code, it will cause a large amount of memory usage, because if there are 1000 files that contain a 2 MB file, so this code will load 1000 * 2 MB streams into memory?

I can lose something, or I'm absolutely right. If something is not missing, why is it better to wait for each request, and consuming a thread is the best approach?

Answer 1

. , ( , ).

​​ . - AsyncLock ( SemaphoreSlim). - TPL Dataflow MaxDegreeOfParallelism.

var block = new ActionBlock<string>(url =>
    {
        var stream = (await WebRequest.CreateHttp(url).GetResponseAsync()).GetResponseStream();
        using (var fileStream = new FileStream("blabla", FileMode.Create))
        {
            await stream.CopyToAsync(fileStream);
        }
    },
    new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 100 });

Answer 2

, async . async -, , ResponseStream.

, , async, .

, , . ResponseStream ( ) .

Answer 3

, . , .

foreach (var chunk in urls.Batch(5))
{
    var streamTasks = chunk
        .Select(async url => await WebRequest.CreateHttp(url).GetResponseAsync())
        .Select(async response => (await response).GetResponseStream());

    var streams = await Task.WhenAll(streamTasks);

    foreach (var stream in streams)
    {
        using (var fileStream = new FileStream("blabla", FileMode.Create))
        {
            await stream.CopyToAsync(fileStream);
        }
    }
}

- , .

public static IEnumerable<IEnumerable<T>> Batch<T>(this IEnumerable<T> source, int chunksize)
{
    while (source.Any())
    {
        yield return source.Take(chunksize);
        source = source.Skip(chunksize);
    }
}