How to implement a message queue solution

Question

I have a scenario where about 10 different messages will need to be queued and then deleted or processed. One subscriber will need all 10 messages, but another will only need 8 out of 10 messages. I am trying to figure out how best to configure this type of architecture. Do you create a queue for each type of message so that subscribers (a) can simply subscribe to the corresponding queues or do you drop them all in one queue and ignore messages that are not related to this subscriber? I want the solution to be flexible / scalable, etc.

process:

• 10 different xml messages will be queued on the IBM WebSphere MQ server.
• We will use .Net (most likely WCF, since WebSphere MQ 7.1 has been added to support WCF)
• We will delete the messages and load them into another database (most likely, SQL Server).
• The solution should scale well, because we will process a very large number of messages, and this can grow (probably 40-50 000 / hour). At least for us.

As always, they value information very much.

- S

Answer 1

OK, based on the comments, here is a suggestion that will scale and does not require significant changes in applications.

From the manufacturer’s side, I will copy the message selection criteria into the message property, and then post the message in the subject. The only change that is required here for the application is the message property. If for some reason you do not want to publish it using the built-in function, you can define an alias by topic. The application believes that it sends messages, but they really are publications.

On the consumer side, you have several options. One of them is to create administrative subscriptions for each application and use the selector in the subscription. Then the messages are sent to a dedicated queue for each consumer based on the selection criteria. Applications believe that they simply consume messages.

Alternatively, the application may simply subscribe to this topic. This gives you the option of a dynamic subscription that does not receive messages when the application is disconnected (if you really need it) or a long-term subscription functionally equivalent to an administrative subscription.

This solution will easily scale to the volumes you specify. Another option is that the manufacturer does not use properties. Here, the consumer application consumes all messages, breaks the message payload on each, and decides whether to process or ignore the message. In this decision, the manufacturer is still publishing the topic. Any decision associated with a direct queue makes the manufacturer know all the destinations. Add another consumer, change the manufacturer. In addition, there is a place for each destination.

In the worst case, the producer creates several messages, and the consumer must read each to decide whether he will be ignored. This parameter may have scaling problems, depending on how deep the selection criteria field is in the payload. A really long XPath expression = poor performance and no way to tune WMQ to compensate for this, since latency is all in the application at this point.

Best case, the producer sets the property of the message and publishes. Consumers choose by property in their subscription, or an administrative subscription does it for them. Whether this solution uses application signatures or administrative signatures does not matter in terms of scalability.

Answer 2

Queuing is relatively “cheap” in terms of resources, plus yes, it’s better to use the queue for each specific purpose, so it is probably best to separate them with the target client, if possible. Using a queue to selectively send messages based on some criteria (correlation identifier or some other thing) is usually a bad idea. The most efficient messaging scenario is the simplest: just pull messages out of the queue as they arrive, rather than viewing and receiving them selectively.

As for scaling, I can’t talk about Websphere MQ or other IBM products, but 40-50K messages per hour for MSMQ on Windows Server is especially difficult to process, so I would suggest that IBM could do it like Well. Usually the bottleneck is not the queue itself, but the process of processing and processing individual messages.