Throttling concurrent asynchronous requests using a loop

Question

I am currently working on creating a large number of web API requests. I tried to complete asyncthis process to do this in a reasonable amount of time, however I cannot disconnect connections so as not to send more than 10request / second. I use a semaphore for throttling, but I'm not quite sure how it will work in this context, since I have a nested loop.

I essentially get a list of models, and each model has a list of days inside it. I need to make a request for every day inside the models. The number of days can be from 1up to 50, 99%at the time when it is just 1. Therefore, I want asynceach model, because there will be one of them 3000, but I want asyncdays in case it takes several days to be completed. I need to stop at or below 10requests / second, so I thought the best way to do this is to set a request limit of 10 for the whole operation. Is there a place where I can put a semaphore that will limit connections to the whole chain?

Each individual request must also make two requests for 2different pieces of data, and this API does not support any batch processing right now.

I am kind of new to C #, very new to asyncand very new to WebRequests/HttpClient, so any help is appreciated. I tried to add all the relevant codes here. If you need anything else, let me know.

public static async Task GetWeatherDataAsync(List<Model> models)
{
    SemaphoreSlim semaphore = new SemaphoreSlim(10);
    var taskList = new List<Task<ComparisonModel>>();

    foreach (var x in models)
    {
        await semaphore.WaitAsync();
        taskList.Add(CompDaysAsync(x));
    }

    try
    {
        await Task.WhenAll(taskList.ToArray());
    }
    catch (Exception e) { }
    finally
    {
        semaphore.Release();
    }
}

public static async Task<Models> CompDaysAsync(Model model)
{
    var httpClient = new HttpClient();
    httpClient.DefaultRequestHeaders.Authorization = new 
                Headers.AuthenticationHeaderValue("Token","xxxxxxxx");
    httpClient.Timeout = TimeSpan.FromMinutes(5);
    var taskList = new List<Task<Models.DateTemp>>();

    foreach (var item in model.list)
    {
        taskList.Add(WeatherAPI.GetResponseForDayAsync(item, 
            httpClient, Latitude, Longitude));
    }
    httpClient.Dispose();
    try
    {
        await Task.WhenAll(taskList.ToArray());
    }
    catch (Exception e) { }

    return model;
}

public static async Task<DateTemp> GetResponseForDayAsync(DateTemp date, HttpClient httpClient, decimal? Latitude, decimal? Longitude)
{
    var response = await httpClient.GetStreamAsync(request1);
    StreamReader myStreamReader = new StreamReader(response);
    string responseData = myStreamReader.ReadToEnd();
    double[] data = new double[2];
    if (responseData != "[[null, null]]")
    {
        data = Array.ConvertAll(responseData.Replace("[", "").Replace("]", "").Split(','), double.Parse);
    }
    else { data = null; };

    double precipData = 0;
    var response2 = await httpClient.GetStreamAsync(request2);
    StreamReader myStreamReader2 = new StreamReader(response2);
    string responseData2 = myStreamReader2.ReadToEnd();
    if (responseData2 != null && responseData2 != "[null]" && responseData2 != "[0.0]")
    {
        precipData = double.Parse(responseData2.Replace("[", "").Replace("]", ""));
    }
    date.Precip = precipData;

    if (data != null)
    {
        date.minTemp = data[0];
        date.maxTemp = data[1];
    }
    return date;
}

Answer 1

I think you do not fully understand what it is doing SemaphoreSlim.

• Your semaphore is a local variable based on a method, so calling each one GetWeatherDataAsync will cause calls 10to your API without waiting for another client.

• , , models.Count > 10, , , 11th , :

  var semaphore = new SemaphoreSlim(10);
  
  foreach (var item in Enumerable.Range(0, 15))
  {
      // will stop after 9
      await semaphore.WaitAsync();
      Console.WriteLine(item);
  }
  

, ( static) GetWeatherDataAsync Release finally.

Parallel.Foreach - , async ( async/await), , .