We have a class that passes messages to the stateless service of the Fabric service. This stateless service has one partition but with many replicas. The message must be sent to all replicas in the system. Therefore, we request FabricClient for one partition and the entire replica of this partition. We use standard HTTP communication (a stateless service has a Message Listener with a self-trained OWIN listener using WebListener / HttpSys) with a common HttpClient instance. During the load test, we get a lot of errors when sending messages. Please note that we have other services in one application, as well as communication (WebListener / HttpSys, ServiceProxy and ActorProxy).
Code where we see Exceptions (stacktrace below code sample):
private async Task SendMessageToReplicas(string actionName, string message)
{
var fabricClient = new FabricClient();
var eventNotificationHandlerServiceUri = new Uri(ServiceFabricSettings.EventNotificationHandlerServiceName);
var promises = new List<Task>();
Partition partition = (await fabricClient.QueryManager.GetPartitionListAsync(eventNotificationHandlerServiceUri).ConfigureAwait(false)).First();
string continuationToken = null;
do
{
var replicas = await fabricClient.QueryManager.GetReplicaListAsync(partition.PartitionInformation.Id, continuationToken).ConfigureAwait(false);
foreach(Replica replica in replicas)
{
promises.Add(SendMessageToReplica(replica, actionName, message));
}
continuationToken = replicas.ContinuationToken;
} while(continuationToken != null);
await Task.WhenAll(promises).ConfigureAwait(false);
}
private async Task SendMessageToReplica(Replica replica, string actionName, string message)
{
if(replica.TryGetEndpoint(out Uri replicaUrl))
{
Uri requestUri = UriUtility.Combine(replicaUrl, actionName);
using(var response = await _httpClient.PostAsync(requestUri, message == null ? null : new JsonContent(message)).ConfigureAwait(false))
{
string responseContent = await response.Content.ReadAsStringAsync().ConfigureAwait(false);
if(!response.IsSuccessStatusCode)
{
throw new Exception();
}
}
}
else
{
throw new Exception();
}
}
The following exception is thrown:
System.Fabric.FabricTransientException: Could not ping any of the provided Service Fabric gateway endpoints. ---> System.Runtime.InteropServices.COMException: Exception from HRESULT: 0x80071C49
at System.Fabric.Interop.NativeClient.IFabricQueryClient9.EndGetPartitionList2(IFabricAsyncOperationContext context)
at System.Fabric.FabricClient.QueryClient.GetPartitionListAsyncEndWrapper(IFabricAsyncOperationContext context)
at System.Fabric.Interop.AsyncCallOutAdapter2`1.Finish(IFabricAsyncOperationContext context, Boolean expectedCompletedSynchronously)
--- End of inner exception stack trace ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Company.ServiceFabric.ServiceFabricEventNotifier.<SendMessageToReplicas>d__7.MoveNext() in c:\work\ServiceFabricEventNotifier.cs:line 138
During the same period, we also see that this is an exception:
System.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 0 - An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full.) ---> System.ComponentModel.Win32Exception (0x80004005): An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full
at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)
at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
at System.Data.SqlClient.SqlConnection.OpenAsync(CancellationToken cancellationToken)
:
Event ID: 4231
Source: Tcpip
Level: Warning
A request to allocate an ephemeral port number from the global TCP port space has failed due to all such ports being in use.
Event ID: 4227
Source: Tcpip
Level: Warning
TCP/IP failed to establish an outgoing connection because the selected local endpoint was recently used to connect to the same remote endpoint. This error typically occurs when outgoing connections are opened and closed at a high rate, causing all available local ports to be used and forcing TCP/IP to reuse a local port for an outgoing connection. To minimize the risk of data corruption, the TCP/IP standard requires a minimum time period to elapse between successive connections from a given local endpoint to a given remote endpoint.
, Microsoft-Service Fabric ,
Event 4121
Source Microsoft-Service-Fabric
Level: Warning
client-02VM4.company.nl:19000/192.168.10.36:19000: error = 2147942452, failureCount=160522. Filter by (type~Transport.St && ~"(?i)02VM4.company.nl:19000") to get listener lifecycle. Connect failure is expected if listener was never started, or listener/its process was stopped before/during connecting.
Event 4097
Source Microsoft-Service-Fabric
Level: Warning
client-02VM4.company.nl:19000 : connect failed, having tried all addresses
:
Event 4096
Source Microsoft-Service-Fabric
Level: Error
client-02VM4.company.nl:19000 failed to bind to local port for connecting: 0x80072747
- , , , ? - ?