Using HttpClient to Download Asynchronous Files

Question

I have a service that returns a csv file in a POST request. I would like to load the specified file using asynchronous methods. Although I can get the file, my code has some unresolved issues and questions:

1) Is it really asynchronous?

2) Is there a way to find out the length of the content, even if it is being sent in chunked format? Think about the steps of progress).

3) How can I best track progress to complete the program exit until all work is completed.

using System; using System.IO; using System.Net.Http; namespace TestHttpClient2 { class Program { /* * Use Yahoo portal to access quotes for stocks - perform asynchronous operations. */ static string baseUrl = "http://real-chart.finance.yahoo.com/"; static string requestUrlFormat = "/table.csv?s={0}&d=0&e=9&f=2015&g=d&a=4&b=5&c=2000&ignore=.csv"; static void Main(string[] args) { while (true) { Console.Write("Enter a symbol to research or [ENTER] to exit: "); string symbol = Console.ReadLine(); if (string.IsNullOrEmpty(symbol)) break; DownloadDataForStockAsync(symbol); } } static async void DownloadDataForStockAsync(string symbol) { try { using (var client = new HttpClient()) { client.BaseAddress = new Uri(baseUrl); client.Timeout = TimeSpan.FromMinutes(5); string requestUrl = string.Format(requestUrlFormat, symbol); //var content = new KeyValuePair<string, string>[] { // }; //var formUrlEncodedContent = new FormUrlEncodedContent(content); var request = new HttpRequestMessage(HttpMethod.Post, requestUrl); var sendTask = client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead); var response = sendTask.Result.EnsureSuccessStatusCode(); var httpStream = await response.Content.ReadAsStreamAsync(); string OutputDirectory = "StockQuotes"; if (!Directory.Exists(OutputDirectory)) { Directory.CreateDirectory(OutputDirectory); } DateTime currentDateTime = DateTime.Now; var filePath = Path.Combine(OutputDirectory, string.Format("{1:D4}_{2:D2}_{3:D2}_{4:D2}_{5:D2}_{6:D2}_{7:D3}_{0}.csv", symbol, currentDateTime.Year, currentDateTime.Month, currentDateTime.Day, currentDateTime.Hour, currentDateTime.Minute, currentDateTime.Second, currentDateTime.Millisecond )); using (var fileStream = File.Create(filePath)) using (var reader = new StreamReader(httpStream)) { httpStream.CopyTo(fileStream); fileStream.Flush(); } } } catch (Exception ex) { Console.WriteLine("Error, try again!"); } } } } 

Answer 1

• "Is it really asynchronous?"

Yes, basically. The DownloadDataForStockAsync() method will return before the operation completes in the await response.Content.ReadAsStreamAsync() statement.

The main exception is near the end of the method where you call Stream.CopyTo() . This is not asynchronous, and because a potentially lengthy operation can lead to noticeable delays. However, in the console program you will not notice, because the continuation of the method is performed in the thread pool, and not in the original calling thread.

If you intend to port this code to a GUI infrastructure such as Winforms or WPF, you must change the read instruction await httpStream.CopyToAsync(fileStream);

2. Is there a way to find out the length of the content, even if it is being sent in chunked format? Think about the steps of progress).

Assuming the server contains Content-Length in the headers (and should), yes. It should be possible.

Note that if you used HttpWebRequest , the response object would have a ContentLength property giving you this value directly. Instead, you are using HttpRequestMessage , which I am less familiar with. But, as far as I can tell, you should have access to the Content-Length value as follows:

 long? contentLength = response.Content.Headers.ContentLength; if (contentLength != null) { // use value to initialize "determinate" progress indication } else { // no content-length provided; will need to display progress as "indeterminate" } 

3. What is the best way to track progress in order to stop the program from exiting until all work is completed.

There are many ways. I will point out that any sensible way would require modifying the DownloadDataForStockAsync() method so that it returns Task , not void . Otherwise, you do not have access to the created task. You still have to do this so that this is not a big deal. :)

The simplest would be to simply keep a list of all the tasks that you run, and then wait for them to exit:

 static void Main(string[] args) { List<Task> tasks = new List<Task>(); while (true) { Console.Write("Enter a symbol to research or [ENTER] to exit: "); string symbol = Console.ReadLine(); if (string.IsNullOrEmpty(symbol)) break; tasks.Add(DownloadDataForStockAsync(symbol)); } Task.WaitAll(tasks); } 

Of course, this requires that you explicitly maintain a list of each Task object, including those that have already been completed. If you intend to work for a long time and process a very large number of characters, this can be prohibitive. In this case, you can use the CountDownEvent object:

 static void Main(string[] args) { CountDownEvent countDown = new CountDownEvent(); while (true) { Console.Write("Enter a symbol to research or [ENTER] to exit: "); string symbol = Console.ReadLine(); if (string.IsNullOrEmpty(symbol)) break; countDown.AddCount(); DownloadDataForStockAsync(symbol).ContinueWith(task => countdown.Signal()) ; } countDown.Wait(); } 

This simply increments the CountDownEvent counter for each task being created and attaches a continuation to each task to decrease the counter. When the counter reaches zero, the event is set, allowing you to return to Wait() .