Outbound connection queue detection for ASP.NET website

Question

Is there any way to detect when outgoing connections are running?

Our ASP.NET application has many outgoing requests to other web services. Recently, we have run into serious performance issues where calls to a specific endpoint took many seconds to complete or turn off time. The owners of this service did not see any performance issues at their end. When we analyzed the network traffic, we saw that indeed, HTTP requests completed in a timely manner. This is when we found out that our long waiting times and timeouts are related to the order of the connection.

Our first approach to fixing this is simply to increase the number of allowed outgoing connections to this endpoint, thus:

<system.net>
  <connectionManagement>
    <add address="http://some.endpoint.com" maxconnection="96" />
  </connectionManagement>
</system.net>

This has led to a sharp reduction in our calls to the end point. However, we noticed that this led to our overall incoming requests taking a lot longer. This is when we encountered Microsoft KB 821268 . Following the guidelines of the “rules of conduct", we have come to the following additional changes:

<processModel maxWorkerThreads="100" maxIoThreads="100" minWorkerThreads="50"/>
<httpRuntime minFreeThreads="704" minLocalRequestFreeThreads="608"/>      

This seemed to fix everything. Our calls some.endpoint.comwere still fast, and the response time was also reduced.

, , , , SQL Server. , , ; , some.endpoint.com , - .

, , , . , , , . , - .

? .

Answer 1

, , , , , , . , , . . HttpWebRequest Average Queue Time, @Simon Mourier ( performancecounters="enabled" ), , , ASP.NET - , , , . , powershell script, . . Linux . 2 ( Program.exe) :

,

> .\ThreadsTop.ps1 -ThreadStates -ProcMask Program

Threads states / process

Process Name    Initialized       Ready     Running     Standby  Terminated     Waiting  Transition     Unknown
------------    -----------       -----     -------     -------  ----------     -------  ----------     -------
Program                   0           0           0           0           0          22           0           0

> .\ThreadsTop.ps1 -ThreadWaitReasons -ProcMask Program

Legend:
 0  - Waiting for a component of the Windows NT Executive| 1  - Waiting for a page to be freed
 2  - Waiting for a page to be mapped or copied          | 3  - Waiting for space to be allocated in the paged or nonpag
ed pool
 4  - Waiting for an Execution Delay to be resolved      | 5  - Suspended
 6  - Waiting for a user request                         | 7  - Waiting for a component of the Windows NT Executive
 8  - Waiting for a page to be freed                     | 9  - Waiting for a page to be mapped or copied
 10 - Waiting for space to be allocated in the paged or nonpaged pool| 11 - Waiting for an Execution Delay to be resolve
d
 12 - Suspended                                          | 13 - Waiting for a user request
 14 - Waiting for an event pair high                     | 15 - Waiting for an event pair low
 16 - Waiting for an LPC Receive notice                  | 17 - Waiting for an LPC Reply notice
 18 - Waiting for virtual memory to be allocated         | 19 - Waiting for a page to be written to disk

Process Name      0   1   2   3   4   5   6   7   8   9  10  11  12  13  14  15  16  17  18  19
------------      -   -   -   -   -   -   -   -   -   -  --  --  --  --  --  --  --  --  --  --
Program           1   0   0   0   0   0  34   0   0   0   0   0   0   0   0   3   0   0   0   0

:

> .\ThreadsTop.ps1 -ThreadStates -ProcMask Program

Threads states / process

Process Name    Initialized       Ready     Running     Standby  Terminated     Waiting  Transition     Unknown
------------    -----------       -----     -------     -------  ----------     -------  ----------     -------
Program                   0           1           6           0           0          20           0           0

24.

> .\ThreadsTop.ps1 -ThreadWaitReasons -ProcMask Program

Legend:
 0  - Waiting for a component of the Windows NT Executive| 1  - Waiting for a page to be freed
 2  - Waiting for a page to be mapped or copied          | 3  - Waiting for space to be allocated in the paged or nonpag
ed pool
 4  - Waiting for an Execution Delay to be resolved      | 5  - Suspended
 6  - Waiting for a user request                         | 7  - Waiting for a component of the Windows NT Executive
 8  - Waiting for a page to be freed                     | 9  - Waiting for a page to be mapped or copied
 10 - Waiting for space to be allocated in the paged or nonpaged pool| 11 - Waiting for an Execution Delay to be resolve
d
 12 - Suspended                                          | 13 - Waiting for a user request
 14 - Waiting for an event pair high                     | 15 - Waiting for an event pair low
 16 - Waiting for an LPC Receive notice                  | 17 - Waiting for an LPC Reply notice
 18 - Waiting for virtual memory to be allocated         | 19 - Waiting for a page to be written to disk

Process Name      0   1   2   3   4   5   6   7   8   9  10  11  12  13  14  15  16  17  18  19
------------      -   -   -   -   -   -   -   -   -   -  --  --  --  --  --  --  --  --  --  --
Program           1   0   0   0   0   0  18   0   0   0   0   0   0   0   0   6   0   0   0   0

, , " " , .

script, - , (, ). , , Concurrency Visualizer, . system.net , , .

Answer 2

NagleAlgorithm

Nagling - TCP , TCP. , TCP .

Nagling TCP Delayed ACK, TCP . , ACK . RFC 1122 , 500 , ACK . ACK ACK , , ACK .

TL; DR

, "", , :

ServicePointManager.UseNagleAlgorithm = false;

Answer 3

, , . , SQL-, - .., , . - , .

, .., , , . . , .

, NewRelic: http://newrelic.com/application-monitoring .