Service fabric: calls between services are delayed?

Question

We are working on application software consisting of several different services, and the key part of our application is that these services must call each other in large volumes.

We had no problems until recently, when we increased the load on our application and found that it slows down significantly. After much research and timing, we found that the problem is that when we make many calls for one type of service (of which we have several instances), the calls seemed to be some delay between us calling the service, and the service actually begins to process the request.

We call between services as described by Microsoft here

To be more clear: ServiceA gets a reference to ServiceB, then calls ServiceB.GetResult (), we register the time when this method is called in ServiceA, and the first thing we do in GetResult () is the time during which the processing starts. When there is no load, only a few ms, as soon as we increase the load that we found there, so that between these times was 4-5 seconds .

Is this some kind of limit in the service? We have several instances of ServiceB, and resource use in the cluster practically does not happen, the processor fluctuates around 10%, and memory usage is about 1/4 on all nodes, but the service bandwidth is very low because it waits here.

Why is he waiting? Is there a certain limit to how different calls can be handled by a service at the same time? Did we do something with our message?

Thanks.

Answer 1

Setting MaxConcurrentCalls seemed to me the most necessary.

When connecting to the service:

FabricTransportSettings transportSettings = new FabricTransportSettings { MaxConcurrentCalls = 32 }; ServiceProxyFactory serviceProxyFactory = new ServiceProxyFactory( (c) => new FabricTransportServiceRemotingClientFactory(transportSettings)); service = serviceProxyFactory.CreateServiceProxy<T>(serviceUri); 

Creating service listeners:

 protected override IEnumerable<ServiceInstanceListener> CreateServiceInstanceListeners() { FabricTransportListenerSettings listenerSettings = new FabricTransportListenerSettings { MaxConcurrentCalls = 32 }; return new[] { new ServiceInstanceListener( (context) => new FabricTransportServiceRemotingListener(context,this,listenerSettings)) }; }