How to make a C # application using HttpWebRequest, behave like a violinist

Question

I have a console application that uses 20 or so threads to connect to a remote web server and sends arbitrary HTTP requests quite small in size, 100% on top of ssl. The remote web server is actually a complete load-balanced data center, full of high-availability systems that can handle hundreds of thousands of requests per second. This is not a server or bandwidth issue. With this in mind, I do not start it and do not affect how it is configured, so I could not make changes on the server side, even if I wanted to.

When you run the application with a script, the application runs very quickly. When he does not work as a violinist, he is really much slower, to the point that he is useless for the task. It seems that it lingers at some point quite early, but this may just be a deadlock problem, but not sure yet.

In any case, the violinist, being a proxy server, will undoubtedly modify my requests / connections in some way, which provides excellent bandwidth, but I have no idea what he is doing. I am trying to figure this out so that I can get my .net application to simulate fiddlers connection processing behavior without having to run it through fiddler

I pasted the connection code below.

using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.Net; using System.IO; namespace Redacted { public class HiveCommunicator { public static IResponse SendRequest(IRequest request) { ServicePointManager.DefaultConnectionLimit = 60; ServicePointManager.Expect100Continue = false; string hostUrlString = string.Empty; if (request.SiteID <= 0) hostUrlString = string.Format("{0}://{1}{2}", request.UseSSL ? "https" : "http", DataCenters.GetCenter(request.DataCenter), request.Path); else hostUrlString = string.Format("{0}://{1}{2}", request.UseSSL ? "https" : "http", DataCenters.GetCenter(request.DataCenter), string.Format(request.Path, request.SiteID)); HttpWebRequest webRequest = (HttpWebRequest)HttpWebRequest.Create(hostUrlString); switch (request.ContentType) { default: case ContentTypes.XML: webRequest.ContentType = "application/xml"; break; case ContentTypes.JSON: webRequest.ContentType = "application/json"; break; case ContentTypes.BINARY: webRequest.ContentType = "application/octet-stream"; break; } if (request.RequiresAuthorizationToken) { AuthorizationToken tok = HiveAuthentication.GetToken(request.SiteID); if (tok == null) { return null; } webRequest.Headers.Add(HttpRequestHeader.Authorization, tok.Token); } bool UsesRequestBody = true; switch (request.HttpVerb) { case HttpVerbs.POST: webRequest.Method = "POST"; break; case HttpVerbs.DELETE: webRequest.Method = "DELETE"; UsesRequestBody = false; break; case HttpVerbs.PUT: webRequest.Method = "PUT"; break; default: case HttpVerbs.GET: webRequest.Method = "GET"; UsesRequestBody = false; break; } HttpWebResponse webResponse = null; Stream webRequestStream = null; byte[] webRequestBytes = null; if (UsesRequestBody) { webRequestBytes = request.RequestBytes; webRequest.ContentLength = webRequestBytes.Length; webRequestStream = webRequest.GetRequestStream(); for (int i = 0; i < webRequest.ContentLength; i++) { webRequestStream.WriteByte(webRequestBytes[i]); } } try { webResponse = (HttpWebResponse)webRequest.GetResponse(); } catch (WebException ex) { webResponse = (HttpWebResponse)ex.Response; } if (UsesRequestBody) { webRequestStream.Close(); webRequestStream.Dispose(); } IResponse respReturn = request.ParseResponse(webResponse); webResponse.Close(); return respReturn; } } } 

Answer 1

I thank the people who tried to help. Unfortunately, this required a call to Microsoft Profesional Support.

Although I used ServicePointManager.Expect100Continue = false; This happened at the end of the application life cycle. After looking at the System.Net.Trace logs, we saw that the continue-100 header continues to be used (except when using a script). The solution was to include this in the application launch (in Main ())

I also tried to read the response stream before closing the request stream.

After it was fixed, everything accelerated. The application works much faster without a violinist than with, as I expected.

Several people said to call the HttpWebResponse. This class does not have a public Dispose method. I assume .Close () calls .Dispose () inside though.

Answer 2

You can play with Fiddler's Connection Settings to find out if the reason for Fiddler’s powerful bandwidth is reusing client connections. In this case, you might want to implement a shared secure pool of http connections, or just watch a movie or something like that. ^^

Answer 3

Take a wild guess here, but this may have to do with the simple setup of app.config:

 <system.net> <connectionManagement> <add address="*" maxconnection="40"/> </connectionManagement> </system.net> 

I had the same problem in a multi-threaded HTTP request application, and this solved this problem.

Answer 4

Given that your application sends "arbitrary small HTTP requests", this may help turn off the Nagle algorithm.

 ServicePointManager.UseNagleAlgorithm = true; 

From MSDN : a number of elements can affect performance when using HttpWebRequest, including:

• Class ServicePointManager .
• The DefaultConnectionLimit property.
• Property UseNagleAlgorithm .

The Nagle algorithm [...] accumulates sequences of small messages into larger TCP packets before data is sent over the network. [...] As a rule, for constant high volumetric throughput, performance is improved using the Nagle algorithm. But for applications with less bandwidth, performance degradation can be seen. [...] If an application uses low latency connections, this can help set this property to false.