I ran into an extremely mysterious problem. I have a Windows service that monitors two MSMQ queues for input and sends messages to another MSMQ queue. Although the send operation seems instantaneous in terms of service, it actually receives the message exactly three (3) minutes later (as shown in the properties window in the MSMQ MMC). I tested this problem and didn’t listen to anything else on the other side so that I could see how messages swell. This is how the service sends messages:
var proxyFactory = new ChannelFactory<IOtherServerInterface>(new NetMsmqBinding(NetMsmqSecurityMode.None) { Durable = true, TimeToLive = new TimeSpan(1, 0, 0), ReceiveTimeout = TimeSpan.MaxValue }); IOtherServerInterface server = this.proxyFactory.CreateChannel(new EndpointAddress("net.msmq://localhost/private/myqueue")); var task = new MyTask() { ... }; using (TransactionScope scope = new TransactionScope(TransactionScopeOption.Required)) { server.QueueFile(task); scope.Complete(); }
The service runs on Windows Server 2008 R2. I also tested it on R1 and noticed the same behavior. Again, everything happens on one machine. All components are deployed there, so I don’t think it could be a network problem.
EDIT # 1:
I turned on WCF diagnostics, and what I noticed is very strange. An MSMQ datagram is usually written normally. However, after the message "message was closed", nothing happens. As if the service is waiting for something. After about 3 minutes and exactly when the MSMQ message arrives (according to the MMCM MSMQ), I see another trace message about the previous action. I suspect there is some kind of hindrance.
Let me tell you more about how services work. There is an IIS application that receives jobs from clients and transfers them to the MSMQ queue. From there, the restless service (MainService) picks them up and starts processing them. In some cases, a different service (AuxService) is required to complete the task, so the MainService sends a message (which is always delayed) to the AuxService. AuxService has its own incoming message queue, where it receives MSMQ messages, and when this is done, it sends an MSMQ message to MainService. Meanwhile, the thread that sent the message to the AuxService waits until it receives a signal or until it expires. There is a special queue in which MainService searches for messages from AuxServices. When a message is received, the aforementioned stream wakes up and resumes its activity.
Here is a view of the whole architecture:
- IIS Application -> Q1 -> MainService
- MainService → Q2 → AuxService
- AuxService → Q3 → MainService
Although all operations are marked by OneWay, I wonder if starting an MSMQ operation as part of another MSMQ operation is somehow illegal. This is similar to empirical evidence. If so, is it possible to change this behavior?
EDIT # 2:
Well, after some extra digging, it seems that WCF is the culprit. I have included both client code in MainService and server code in AuxService to use the MSMQ SDK directly, and it works as expected. The 3 minute timeout that I experienced was actually the time after which the MainService gave up and thought that the AuxService failed. Therefore, it seems that for some reason, the WCF refuses to complete the transfer until the current WCF activity exits.
Is it design or is it a mistake? Can this behavior be controlled?