Web Api + HttpClient: asynchronous module or handler completed while asynchronous operation has not yet completed

Question

I am writing an application that proxies some HTTP requests using the ASP.NET web API, and I am struggling to determine the source of the intermittent error. It looks like a race condition ... but I'm not quite sure.

Before I explain in more detail, this is the general communication flow of the application:

• The client makes an HTTP proxy request 1 .
• Proxy 1 redirects the contents of the HTTP request to Proxy 2
• Proxy 2 transfers the contents of the HTTP request to the target web application
• The target web application responds to the HTTP request and the response is transmitted (transmitted over the channels) to Proxy 2
• Proxy 2 returns the response of Proxy 1 , which in turn answers the original call by the Client .

Proxy applications are written to the ASP.NET RTM web API using .NET 4.5. The code for running the relay looks like this:

//Controller entry point. public HttpResponseMessage Post() { using (var client = new HttpClient()) { var request = BuildRelayHttpRequest(this.Request); //HttpCompletionOption.ResponseHeadersRead - so that I can start streaming the response as soon //As it begins to filter in. var relayResult = client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead).Result; var returnMessage = BuildResponse(relayResult); return returnMessage; } } private static HttpRequestMessage BuildRelayHttpRequest(HttpRequestMessage incomingRequest) { var requestUri = BuildRequestUri(); var relayRequest = new HttpRequestMessage(incomingRequest.Method, requestUri); if (incomingRequest.Method != HttpMethod.Get && incomingRequest.Content != null) { relayRequest.Content = incomingRequest.Content; } //Copies all safe HTTP headers (mainly content) to the relay request CopyHeaders(relayRequest, incomingRequest); return relayRequest; } private static HttpRequestMessage BuildResponse(HttpResponseMessage responseMessage) { var returnMessage = Request.CreateResponse(responseMessage.StatusCode); returnMessage.ReasonPhrase = responseMessage.ReasonPhrase; returnMessage.Content = CopyContentStream(responseMessage); //Copies all safe HTTP headers (mainly content) to the response CopyHeaders(returnMessage, responseMessage); } private static PushStreamContent CopyContentStream(HttpResponseMessage sourceContent) { var content = new PushStreamContent(async (stream, context, transport) => await sourceContent.Content.ReadAsStreamAsync() .ContinueWith(t1 => t1.Result.CopyToAsync(stream) .ContinueWith(t2 => stream.Dispose()))); return content; } 

Error that occurs intermittently:

The asynchronous module or handler is completed while the asynchronous operation has not yet been completed.

This error usually occurs during the first few requests to proxy applications, after which the error no longer appears.

Visual Studio never catches an exception on a throw. But the error can be detected in the Global.asax Application_Error event. Unfortunately, there is no stack trace in Exception.

Proxy applications are hosted in Azure Web roles.

Any help identifying the culprit would be appreciated.

Answer 1

Your problem is subtle: the async lambda, which you switch to PushStreamContent , is interpreted as async void (because the PushStreamContent constructor accepts only Action as parameters). So, there is a race condition between your module / handler and the completion of this async void lambda.

PostStreamContent detects the thread closes and treats it as the end of its Task (module / handler termination), so you just need to be sure that there are no async void methods that can still be executed after the thread is closed. async Task methods are fine, so this should fix it:

 private static PushStreamContent CopyContentStream(HttpResponseMessage sourceContent) { Func<Stream, Task> copyStreamAsync = async stream => { using (stream) using (var sourceStream = await sourceContent.Content.ReadAsStreamAsync()) { await sourceStream.CopyToAsync(stream); } }; var content = new PushStreamContent(stream => { var _ = copyStreamAsync(stream); }); return content; } 

If you want your proxies to scale a little better, I also recommend getting rid of all Result calls:

 //Controller entry point. public async Task<HttpResponseMessage> PostAsync() { using (var client = new HttpClient()) { var request = BuildRelayHttpRequest(this.Request); //HttpCompletionOption.ResponseHeadersRead - so that I can start streaming the response as soon //As it begins to filter in. var relayResult = await client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead); var returnMessage = BuildResponse(relayResult); return returnMessage; } } 

Your previous code blocks one thread for each request (until headers are received); Using async down to your controller level, you will not block the stream during this time.

Answer 2

A slightly simpler model is that you can simply use HttpContents directly and pass them inside the relay. I just downloaded a sample that illustrates how to asynchronously both requests and responses and without buffering the contents is relatively simple:

http://aspnet.codeplex.com/SourceControl/changeset/view/7ce67a547fd0#Samples/WebApi/RelaySample/ReadMe.txt

It is also useful to reuse the same instance of HttpClient, as this allows you to reuse connections if necessary.

Answer 3

I would like to add some wisdom to everyone who has landed here with the same error, but all of your code seems to be beautiful. Find any lambda expressions passed to the function in the call tree where it comes from.

I was getting this error when invoking JavaScript JSON to an MVC 5.x controller action. Everything I did up and down the stack was defined by async Task and called using await .

However, using the Visual Studio “Install Next Operator” function, I systematically skipped lines to determine which one called. I continued to drill local methods until I received a call to an external NuGet package. The called method accepted Action as a parameter, and the lambda expression passed for this Action was preceded by the async . As Stephen Cleary emphasizes in his answer, this is seen as an async void that MVC doesn't like. Fortunately, Async versions of the same methods were in the package. Switching to their use, as well as some subsequent calls to the same package, fixed the problem.

I understand that this is not a new solution to the problem, but I went through this thread several times in my search, trying to solve the problem, because I thought that I did not have async void or async <Action> calls and I wanted to help someone else avoid this.