The problem of designing parallel processing MSMQ

Question

I have a Windows service written in C # that is read from MSMQ and based on the type of message that it assigns to them the agents that process this message in the workflow. The application starts without agents and is created dynamically at runtime when messages arrive in MSMQ

Here is a basic figure of how this works:

If the agent workflow is busy doing work, the message is queued in its local queue. So far, so good. But if for some reason, if the service is stopped, the contents of the local queue are lost.

I am trying to figure out what might be the best way to handle this scenario. Right now, local queues are System.Concurrent.ConcurrentQueue . Maybe I could use Sql Ce db or some other persistent storage, but I'm worried about performance. Another thing, in my opinion, is to read MSMQ only when the agents are ready to process messages, but the problem is that I do not know which message MSMQ will contain.

What possible approaches can I take on this issue?

Answer 1

The following template is mainly implemented in your project: http://www.eaipatterns.com/MessageDispatcher.html

However, instead of using actual messaging, you choose to implement a dispatcher in multi-threaded code.

Rather, each handler should be an autonomous process with its own message queue. This is what ensures the durability of the message in case of failure. It also allows you to scale simply by hosting more instances of the handler.

Answer 2

I created a similar system dependent on Redis. The idea is that it provides quick access to data, isolated from the rest of the application, and will not be disconnected when my service is running. In addition, it ultimately saves my data to disk, so I get a good compromise between reliability and speed.

If you designed it so that each client reads it from its own message queue, which was hosted in Redis, you could keep the queue independent of the downtime of the service, and each workload was distributed the next time the service was started.

Answer 3

Why don't you just create two new msms queues to receive messages for Agenta and agentb and create a new agent that (transactionally) extracts the command from the main queue and sends the message to the appropriate agent queue?