Error checking when using HttpClient to load asynchronous files

Question

This question is a continuation of Threading issues when using HttpClient to load asynchronous files .

To get the file transfer for asynchronous use using HttpClient, you need to add HttpCompletionOption.ResponseHeadersRead to the SendAsync request. That way, when this call ends, you can determine that everything was fine with the request and response headers by adding the EnsureSuccessStatusCode call. However, the data may still be transferred at this point.

How can you detect errors that occur after the headers return, but before the data transfer is complete? How would mistakes appear?

The following code example follows with a question question marked on line 109) with the comment: "// ***** WANT TO MAKE MORE ERRORS CHECKING HERE **"

using System; using System.Collections.Generic; using System.IO; using System.Net.Http; using System.Threading.Tasks; namespace TestHttpClient2 { class Program { /* Use Yahoo portal to access quotes for stocks - perform asynchronous operations. */ static string baseUrl = "http://real-chart.finance.yahoo.com/"; static string requestUrlFormat = "/table.csv?s={0}&d=0&e=1&f=2016&g=d&a=0&b=1&c=1901&ignore=.csv"; static void Main(string[] args) { var activeTaskList = new List<Task>(); string outputDirectory = "StockQuotes"; if (!Directory.Exists(outputDirectory)) { Directory.CreateDirectory(outputDirectory); } while (true) { Console.WriteLine("Enter symbol or [ENTER] to exit:"); string symbol = Console.ReadLine(); if (string.IsNullOrEmpty(symbol)) { break; } Task downloadTask = DownloadDataForStockAsync(outputDirectory, symbol); if (TaskIsActive(downloadTask)) { // This is an asynchronous world - lock the list before updating it! lock (activeTaskList) { activeTaskList.Add(downloadTask); } } else { Console.WriteLine("task completed already?!??!?"); } CleanupTasks(activeTaskList); } Console.WriteLine("Cleaning up"); while (CleanupTasks(activeTaskList)) { Task.Delay(1).Wait(); } } private static bool CleanupTasks(List<Task> activeTaskList) { // reverse loop to allow list item deletions // This is an asynchronous world - lock the list before updating it! lock (activeTaskList) { for (int i = activeTaskList.Count - 1; i >= 0; i--) { if (!TaskIsActive(activeTaskList[i])) { activeTaskList.RemoveAt(i); } } return activeTaskList.Count > 0; } } private static bool TaskIsActive(Task task) { return task != null && task.Status != TaskStatus.Canceled && task.Status != TaskStatus.Faulted && task.Status != TaskStatus.RanToCompletion; } static async Task DownloadDataForStockAsync(string outputDirectory, string symbol) { try { using (var client = new HttpClient()) { client.BaseAddress = new Uri(baseUrl); client.Timeout = TimeSpan.FromMinutes(5); string requestUrl = string.Format(requestUrlFormat, symbol); var request = new HttpRequestMessage(HttpMethod.Post, requestUrl); var response = await client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead); response.EnsureSuccessStatusCode(); using (var httpStream = await response.Content.ReadAsStreamAsync()) { var timestampedName = FormatTimestampedString(symbol, true); var filePath = Path.Combine(outputDirectory, timestampedName + ".csv"); using (var fileStream = File.Create(filePath)) { await httpStream.CopyToAsync(fileStream); } } // *****WANT TO DO MORE ERROR CHECKING HERE***** } } catch (HttpRequestException ex) { Console.WriteLine("Exception on thread: {0}: {1}\r\n", System.Threading.Thread.CurrentThread.ManagedThreadId, ex.Message, ex.StackTrace); } catch (Exception ex) { Console.WriteLine("Exception on thread: {0}: {1}\r\n", System.Threading.Thread.CurrentThread.ManagedThreadId, ex.Message, ex.StackTrace); } } static volatile string lastTimestampedString = string.Empty; static volatile string dummy = string.Empty; static string FormatTimestampedString(string message, bool uniquify = false) { // This is an asynchronous world - lock the shared resource before using it! lock (dummy) //lock (lastTimestampedString) { Console.WriteLine("IN - Thread: {0:D2} lastTimestampedString: {1}", System.Threading.Thread.CurrentThread.ManagedThreadId, lastTimestampedString); string newTimestampedString; while (true) { DateTime lastDateTime = DateTime.Now; newTimestampedString = string.Format( "{1:D4}_{2:D2}_{3:D2}_{4:D2}_{5:D2}_{6:D2}_{7:D3}_{0}", message, lastDateTime.Year, lastDateTime.Month, lastDateTime.Day, lastDateTime.Hour, lastDateTime.Minute, lastDateTime.Second, lastDateTime.Millisecond ); if (!uniquify) { break; } if (newTimestampedString != lastTimestampedString) { break; } //Task.Delay(1).Wait(); }; lastTimestampedString = newTimestampedString; Console.WriteLine("OUT - Thread: {0:D2} lastTimestampedString: {1}", System.Threading.Thread.CurrentThread.ManagedThreadId, lastTimestampedString); return lastTimestampedString; } } } } 

Answer 1

I copied and cleared the corresponding code a bit.

 var request = new HttpRequestMessage(HttpMethod.Post, requestUrl); var response = await client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead); response.EnsureSuccessStatusCode(); using (var httpStream = await response.Content.ReadAsStreamAsync()) { var timestampedName = FormatTimestampedString(symbol, true); var filePath = Path.Combine(outputDirectory, timestampedName + ".csv"); using (var fileStream = File.Create(filePath)) { await httpStream.CopyToAsync(fileStream); } } 

The question is, what if something goes wrong while reading a stream and copying it to a file?

All logical errors have already been considered as part of the HTTP request and response cycle: the server received your request, it decided that it is valid, it responded successfully (part of the response header), and now it has sent you the result (part of the response body).

The only errors that may occur right now are things like a server crash, loss of connection, etc. I understand that they will display as HttpRequestException , that is, you can write code like this:

 try { using (var httpStream = await response.Content.ReadAsStreamAsync()) { var timestampedName = FormatTimestampedString(symbol, true); var filePath = Path.Combine(outputDirectory, timestampedName + ".csv"); using (var fileStream = File.Create(filePath)) { await httpStream.CopyToAsync(fileStream); } } } catch (HttpRequestException e) { ... } 

The documentation says little, unfortunately. The source of the link is wrong. So, your best bet is to start from there and possibly log all exceptions that are not HttpRequestException if there is another type of exception that can be thrown while loading the response body.

Answer 2

If you want to narrow it down to the part that lies between the read header and reading the content, you will actually leave yourself with an asynchronous buffer:

 var httpStream = await response.Content.ReadAsStreamAsync(); 

If you look at what happens inside the method, you will see:

 public Task<Stream> ReadAsStreamAsync() { this.CheckDisposed(); TaskCompletionSource<Stream> tcs = new TaskCompletionSource<Stream>(); if (this.contentReadStream == null && this.IsBuffered) { this.contentReadStream = new MemoryStream(this.bufferedContent.GetBuffer(), 0, (int)this.bufferedContent.Length, false, false); } if (this.contentReadStream != null) { tcs.TrySetResult(this.contentReadStream); return tcs.Task; } this.CreateContentReadStreamAsync().ContinueWithStandard(delegate(Task<Stream> task) { if (!HttpUtilities.HandleFaultsAndCancelation<Stream>(task, tcs)) { this.contentReadStream = task.Result; tcs.TrySetResult(this.contentReadStream); } }); return tcs.Task; } 

CreateContentReadStreamAsync is the one that does all the reading, inside, it will call LoadIntoBufferAsync , which you can find here .

Basically, you can see that it encapsulates IOException and ObjectDisposedException , or ArgumentOutOfRangeException - this is a buffer larger than 2 GB (although I think it will be very rare).