HttpWebRequest slower with striped data

Question

I am using HttpWebRequest to connect to my embedded HTTP server. My problem is that it is much slower than connecting to the server via PostMan ( https://chrome.google.com/webstore/detail/postman-rest-client/fdmmgilgnpjigdojojpjoooidkmcomcm?hl=en ), which is probably in the functions in Chrome to request data.

The server is built using this example on MSDN ( http://msdn.microsoft.com/en-us/library/dxkwh6zw.aspx ) and uses a buffer size of 64. The request is an HTTP request with some data in the body.

When connected through PostMan, the request is divided into a bunch of pieces, and BeginRecieve () is called several times, each time receiving 64B and takes about 2 milliseconds. Except for the latter, which receives less than 64B.

But when connecting to my client using HttpWebRequest, the first BeginRecieve () callback takes 64B and takes about 1 ms, the next only 47B and takes almost 200 ms, and finally the third receives about 58B and takes 2 ms.

What happens to the second BeginRecieve? I note that the connection is established as soon as I start writing data to the HttpWebRequest input stream, but the data reception does not start until I call GetResponse ().

Here is my HttpWebRequest code:

var request = (HttpWebRequest)WebRequest.Create(url); request.Method = verb; request.Timeout = timeout; request.Proxy = null; request.KeepAlive = false; request.Headers.Add("Content-Encoding", "UTF-8"); System.Net.ServicePointManager.Expect100Continue = false; request.ServicePoint.Expect100Continue = false; if ((verb == "POST" || verb == "PUT") && !String.IsNullOrEmpty(data)) { var dataBytes = Encoding.UTF8.GetBytes(data); try { var dataStream = request.GetRequestStream(); dataStream.Write(dataBytes, 0, dataBytes.Length); dataStream.Close(); } catch (Exception ex) { throw; } } WebResponse response = null; try { response = request.GetResponse(); } catch (Exception ex) { throw; } var responseReader = new StreamReader(rStream, Encoding.UTF8); var responseStr = responseReader.ReadToEnd(); responseReader.Close(); response.Close(); 

What am I doing wrong? Why does it behave differently than an HTTP request from a web browser? This effectively adds 200 ms of lag in my application.

Answer 1

It looks like a typical Nagle algorithm case, faced with TCP Delayed Confirmation . In your case, you send a small Http request (~ 170 bytes according to your numbers). This is probably less than MSS (maximum segment size), which means the Nagle algorithm will start working. Perhaps the server delays the ACK, which leads to a delay of up to 500 ms. See links for more details.

You can disable Nagle through ServicePointManager.UseNagleAlgorithm = false (before issuing the first request), see MSDN .

Also see the Nagles algorithm is not suitable for small queries for a detailed discussion, including Wireshark analysis.

Note. In your answer, you are faced with the same situation when you write, write, read. When you switch to reading, you overcome this problem. However, I do not believe that you can instruct HttpWebRequest (or HttpClient, for that matter) to send small requests as a single TCP write operation. This will probably be a good optimization in some cases. Although this can lead to some additional copying of arrays, which negatively affects performance.

Answer 2

200 ms is a typical latency of the Nagle algorithm. This raises the suspicion that the server or client is using Nagling. You say you are using the sample from MSDN as the server ... Well, that’s it. Use the correct server or disable Nagling.

Assuming the HttpWebRequest built-in class has an unnecessary delay of 200 ms, it is very unlikely. Look at another place. Look at your code to find the problem.

Answer 3

It seems that the HttpWebRequest is just very slow.

The funny thing: I implemented my own HTTP client using Sockets, and I found the key to why HttpWebRequest is so slow. If I encoded ASCII headers into my own byte array and sent them to a stream, followed by an array of bytes encoded from my data, my Sockets based HTTP client behaved exactly like HttpWebRequest: first it fills one buffer with data (part of the header), then it partially uses another buffer (the rest of the header), waits 200 ms and then sends the rest of the data.

The code:

 TcpClient client = new TcpClient(server, port); NetworkStream stream = client.GetStream(); // Send this out stream.Write(headerData, 0, headerData.Length); stream.Write(bodyData, 0, bodyData.Length); stream.Flush(); 

The solution, of course, was to add two byte arrays before sending them to the stream. My application now behaves as expected.

Code with a single write stream:

 TcpClient client = new TcpClient(server, port); NetworkStream stream = client.GetStream(); var totalData = new byte[headerBytes.Length + bodyData.Length]; Array.Copy(headerBytes,totalData,headerBytes.Length); Array.Copy(bodyData,0,totalData,headerBytes.Length,bodyData.Length); // Send this out stream.Write(totalData, 0, totalData.Length); stream.Flush(); 

And the HttpWebRequest seems to send the header before I write to the request stream, so it can be implemented as my first code example. Does that make sense at all?

Hope this helps anyone who has the same problem!

Answer 4

Try the following: you need to get rid of IDisposables:

 var request = (HttpWebRequest)WebRequest.Create(url); request.Method = verb; request.Timeout = timeout; request.Proxy = null; request.KeepAlive = false; request.Headers.Add("Content-Encoding", "UTF-8"); System.Net.ServicePointManager.Expect100Continue = false; request.ServicePoint.Expect100Continue = false; if ((verb == "POST" || verb == "PUT") && !String.IsNullOrEmpty(data)) { var dataBytes = Encoding.UTF8.GetBytes(data); using (var dataStream = request.GetRequestStream()) { dataStream.Write(dataBytes, 0, dataBytes.Length); } } string responseStr; using (var response = request.GetResponse()) { using (var responseReader = new StreamReader(rStream, Encoding.UTF8)) { responseStr = responseReader.ReadToEnd(); } }