WCF reliable messages: stuttering after maxPendingChannels enlarge

Question

We have a problem where during load testing, if we quickly call in one of our services, we get an error

"System.ServiceModel.ServerTooBusyException: A request to create a reliable session was denied by the destination server RM." Net.tcp: // local: 10511 / ParameterMonitorService “is too busy to process this request Please try again later. Unable to open channel.

We increased the value of maxPendingChannels from its default from 4 to 128, and then outside, and now the error has disappeared, instead of throwing an exception, the service will simply stop processing messages under load, and then start in a few minutes.

It seems that he will not drop anything, he just hangs for a while. The more we broke the service, the longer this restoration would probably take.

The service is configured as Per-Call with ConcurrencyMode Multiple. Other behavior settings:

<serviceThrottling maxConcurrentCalls="100" maxConcurrentSessions="100" maxConcurrentInstances="100"/>

<customBinding>

    <binding name="Services_Custom_Binding" openTimeout="00:00:20" sendTimeout="00:01:00">          
        <reliableSession  ordered="true" inactivityTimeout="00:10:00" maxPendingChannels="128" flowControlEnabled="true" />
        <binaryMessageEncoding>
          <readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384"
            maxBytesPerRead="4096" maxNameTableCharCount="16384" />            
        </binaryMessageEncoding>
        <tcpTransport maxPendingConnections="100" listenBacklog="100" />          
      </binding>
  </customBinding>

We are kind of stuck. Any help appreciated!

Answer 1

This is a classic performance tuning story. By reconfiguring the throttle for reliable sessions, you removed what used to be a bottleneck in the system and moved the bottleneck to another place in your system.

, , , - , , , , . Windows Performance Monitor , , .

, concurrency , - , , .

Answer 2

8 . , WCF , .

, , AdjustThreads , :

    Imports NLog
    Public Module AdjustThreads_

    Private _Logger As Logger = LogManager.GetCurrentClassLogger
    Private _MaxWorkers As Integer = 16
    Private _MaxCompletions As Integer = 16
    Public Sub AdjustThreads()
        Dim minworkerthreads As Integer = 0
        Dim maxworkerthreads As Integer = 0
        Dim mincompletionthreads As Integer = 0
        Dim maxcompletionthreads As Integer = 0
        Dim activeworkerthreads As Integer = 0
        Dim activecompletionthreads As Integer = 0
        Threading.ThreadPool.GetMinThreads(minworkerthreads, mincompletionthreads)
        Threading.ThreadPool.GetMaxThreads(maxworkerthreads, maxcompletionthreads)
        Threading.ThreadPool.GetAvailableThreads(activeworkerthreads, activecompletionthreads)
        Dim workers As Integer = maxworkerthreads - activeworkerthreads
        Dim completions As Integer = maxcompletionthreads - activecompletionthreads
        If workers > _MaxWorkers Then
            _MaxWorkers = _MaxWorkers
        End If
        If completions > _MaxCompletions Then
            _MaxCompletions = completions
        End If
        ' If current is (initially) 8, new threads only start twice a second.
        ' So, kick off a minimum of 16 and always increase by 50%
        Dim needworkers As Integer = _MaxWorkers * 3 \ 2
        Dim needcompletions As Integer = _MaxCompletions * 3 \ 2

        If needworkers > minworkerthreads OrElse
           needcompletions > mincompletionthreads Then
            _Logger.Info("Threadpool increasing workers to {0}, completions to {1}",
                         needworkers, needcompletions)
            Threading.ThreadPool.SetMinThreads(needworkers, needcompletions)
        End If
    End Sub
End Module

( 'End Module' , - ?)