WCF client causes server to hang until connection fails

Question

The text below is an attempt to expand and add color to this question:

How can I prevent a bad client from uninstalling the entire service?

I essentially have this scenario: the WCF service is up and running with a client callback that has a direct, simple, one-way message, not very different from this:

public interface IMyClientContract { [OperationContract(IsOneWay = true)] void SomethingChanged(simpleObject myObj); } 

I call this method potentially thousands of times per second from the service until there will eventually be about 50 concurrently connected clients with minimal delay (<15 ms would be nice). This works fine until I set a breakpoint on one of the client applications connected to the server and then everything freezes after 2-5 seconds when the service freezes and no other clients receive any data for about 30 seconds or so long as the service logs a connection failure event and disconnects the intruder client. After that, all other clients continue to receive messages.

I did a study on configuring serviceThrottling, concurrency, setting minimum threadpool threads, secret WCF sauces and as many as 9 yards, but at the end of the day this MSDN article - basic WCF requirements, one-way calls, callbacks and events describe exactly the problem I encountered without making recommendations.

A third solution that allows the service to safely call the customer back is to set up callback operations as one-way operations. This allows the service to call back even when concurrency is configured to be single-threaded because there will be no response message to block.

but earlier in the article the problem that I see is described only from the point of view of the client

When one-way calls reach the service, they cannot be sent immediately and can be queued on the service side, which will be sent one at a time, all in accordance with the configured operation mode and concurrency mode session. How many messages (whether it is one-way or request-response) that the service wants to queue is the product of the configured channel and reliability mode. If the number of messages in the queue exceeds the bandwidth of the queue, the client will block even when a one-way call is issued

I can only assume that the opposite is true, the number of messages in the queue for the client has exceeded the bandwidth of the queue, and the threadpool is now full of threads trying to call this client, which are now blocked.

What is the right way to handle this? Should I explore a way to check how many messages are queued at the intercom level for each client and interrupt their connections after reaching a certain limit?

It seems that if the WCF service itself blocks the queue filling, then all the async / oneway / fire-and-forget strategies that I could ever implement inside the service will still be blocked whenever one client queue is full.

Answer 1

Update:

I implemented Fire-and-forget to call the client callback channel, and the server no longer blocks as soon as the buffer is filled by the client

MyEvent is an event with a delegate that matches one of the methods defined in the WCF client contract, when they connect, I essentially add a callback to the event

 MyEvent += OperationContext.Current.GetCallbackChannel<IFancyClientContract>().SomethingChanged 

etc., and then send this data to all clients, I do the following

 //serialize using protobuff using (var ms = new MemoryStream()) { ProtoBuf.Serializer.Serialize(ms, new SpecialDataTransferObject(inputData)); byte[] data = ms.GetBuffer(); Parallel.ForEach(MyEvent.GetInvocationList(), p => ThreadUtil.FireAndForget(p, data)); } 

in the ThreadUtil class, I essentially made the following change to the code defined in the fire-and-foget article

 static void InvokeWrappedDelegate(Delegate d, object[] args) { try { d.DynamicInvoke(args); } catch (Exception ex) { //THIS will eventually throw once the client WCF callback channel has filled up and timed out, and it will throw once for every single time you ever tried sending them a payload, so do some smarter logging here!! Console.WriteLine("Error calling client, attempting to disconnect."); try { MyService.SingletonServiceController.TerminateClientChannelByHashcode(d.Target.GetHashCode());//this is an IContextChannel object, kept in a dictionary of active connections, cross referenced by hashcode just for this exact occasion } catch (Exception ex2) { Console.WriteLine("Attempt to disconnect client failed: " + ex2.ToString()); } } } 

I have no good ideas on how to go and kill all pending packets that the server is still waiting to see if they will be delivered. As soon as I get the first exception, theoretically I should be able to go and stop all other requests in some queue, but this setting is functional and meets the goals.

Answer 2

I don't know much about client callbacks, but it seems like common problems with wcf code blocking. I often solve these problems by creating a BackgroundWorker and making a client call in a thread. During this time, the main thread counts how long the child thread takes. If the child has not finished in a few milliseconds, the main thread simply moves and leaves the stream (it eventually dies by itself, so there is no memory leak). This is basically what Mr. Graves offers with the phrase "fire and forget."