Message Queuing with Message Barrier?

Question

Is there an implementation of a message queue that allows you to split work into “batches” by inserting “message barriers” into the message flow? Let me explain. No messages after the message barrier has to be delivered to any consumers of the queue until all messages before the barrier are used. It looks like a synchronization point. I would also prefer that all consumers receive a notification when they reach the barrier.

Anything like this?

Answer 1

I do not know the existing widely available implementations, but if you allow me, I would suggest a very simple general implementation using a proxy , where:

• manufacturers write to the proxy / topic queue
• the proxy is redirected to the initial queue / topic until the proxy server reads the barrier message , after which:
  • the proxy server can notify subscribers of barriers to forward the barrier message to the original subject or
  • The proxy server can notify subscribers of the barrier queue :
    • periodically publish barrier messages until the barrier is cleared; this does not guarantee that all consumers will receive exactly one notification, although all will eventually clear the barrier (some of them can receive 0 notifications, others> 1 notifications - all depending on the type of scheduler used to distribute messages to consumers, for example, roundrobin )
    • using the dedicated section to notify each consumer exactly once per barrier
  • the proxy stops the redirection of any messages from the proxy queue until the barrier is cleared, that is, until the original queue is empty and / or all consumers confirm all messages in the queue / subject (if any) leading to a barrier
• proxy resumes forwarding

UPDATE

Thanking Miklos for the fact that in JMS the structure does not provide confirmation for asynchronous deliveries (what is called "confirmations" in JMS is a purely consumer concept and is not a proxy server as such).

So, under JMS , an existing implementation (which will be adapted for barriers) may already provide application-level confirmations through a "confirmation queue" (as opposed to the original queue - which will be a "request queue".) Consumers would have to acknowledge the execution of requests by sending messages confirmations to the proxy confirmation queue; the proxy server will use confirmation messages to track when the barrier has been cleared after it also redirected confirmation messages to the manufacturer.

If the existing implementation (which will be adapted for barriers) does not already provide application level confirmation through the "confirmation queue", you can either:

• use QueueBrowser , , which the:
  • you are dealing with queueus not events that
  • you want to synchronize upon delivery not confirmation of completion, but
  • normally synchronize with first delivery, even if the request was actually aborted and needs to be resent (even after the barrier has been cleared.) I think Miklos already indicated this problem from IIRC.
• otherwise , add the confirmation queue consumed by the proxy server and adapt the consumers to write confirmations to it (essentially, the JMS scenario is higher, except that the proxy server should not forward confirmation messages to the producer if your manufacturer does not need functionality.)

Answer 2

You can achieve this by using the topic for "Barrier communication" and the queue for the "packet elements" that are consumed using selective receivers.
Publishing a Barrier message to a topic ensures that all consumers receive their own copy of the Barrier message.

Each consumer will need two subscriptions:

• To the topic of the barrier
• Selective receiver for a packet queue using the selection criteria defined in the ban message.

The Barrier message must contain a packet key that should be applied to the queue consumer selection criteria.
e.g. batchId = n
or JMSMessageID <100
or JMSTimestamp <xxx

Whenever a barrier message is received,

• current queue consumer must be closed
• queue selection criteria must be changed using the contents of the Barrier message
• new selective consumer must be launched using modified selection criteria

If you intend to use a custom batch key for selection criteria such as "batchId" above, then it is assumed that all message producers can set this JMS property, otherwise the proxy server will have to use the message set property and re-queue where selective consumers are listened .

For more information about selective receivers, see the following links:

• http://java.sun.com/j2ee/1.4/docs/api/javax/jms/Message.html
• http://java.sun.com/j2ee/sdk_1.3/techdocs/api/javax/jms/QueueSession.html#createReceiver(javax.jms.Queue,%20java.lang.String)