Message processing exactly once

Question

Please consider the scenario as shown in the attached image:

• The portal (manufacturer) will send any message to the bus to which it is necessary to process several applications (consumer) - PAYROLLAPP, HELPDESK, etc.
• You can run multiple instances of consumer applications, these instances can also be added / deleted dynamically
• Now it is important to ensure that the message is processed only once, for each application , if it PAYROLLAPP -1 processes the message, PAYROLLAPP -2 DOES NOT process it; Of course, in the above diagram, HELPDESK - 1 should process it. In short, in the case of multiple instances, exactly one should process the message as soon as
• When I was looking for answers, most of the material concerned the creation of a "selective consumer" - a consumer who accepts / rejects a message based on some logic - note that there are no changes / additions / wrappers for applications shown in the diagram; the logic must be somewhere in the provider that controls the bus.

Please tell about the same.

Adding more details after Petter's answer:

• The items to the left of the left-hand row are “approaches” - Pure JMS, ESB, EAI
• Elements to the right of the line with dots to the right are "implementations"

Now the bulk is QUERIES:

• Whatever solution (pure JMS, ESB, EAI), part below the horizontal dashed line (application queue) to implement?
• How is ESB (JBoss ESB, etc.) used instead of "pure JMS (Active MQ, etc.)), help / interfere? Does ESB provide any advantage over JMS, which is only java (?). I confused - "ESB or JMS, even after links to topics such as: JMS and ESB - how are they related? . It has one answer that says: "JMS is not very suitable for integrating REST services, file systems, S / FTP, email, Hessian, SOAP, etc., which work best with an ESB that supports these types initially For example, if you have a process that dumps a 500 MB CSV file at midnight, and you want another system to download the file, analyze the CSV, and import it into the database, this can be easily performed by the ESB, whereas a solution with JMS will be poor. Similarly, load balancing REST services / switching to multiple exe Backend players can be enhanced with an ESB that supports HTTP / S natively. " This only added to my confusion!
• Is using an EAI framework (Apache Camel, etc.) completely different from pure JMS or ESB? If so, how and what are the pros / cons?
• I was told that the ESB will not help, BPM (or something else?) Needs to be used to determine and store the routing logic - is that true?

Answer 1

I see the point. This can be a little tricky with pure JMS.

What you essentially want to do is allow the portal to post messages in the thread, but not allow PAYROLLAPP to subscribe to this thread (since they will all receive a copy of the message). Thus, you will need some logic between what distributes the message from the subscription topic to one queue per application type. From this queue, normal load balancing (a competing consumer pattern) can be implemented using JMS.

Various JMS providers have special implementations that can solve this problem. ActiveMQ has its own virtual addresses . In WebSphere MQ there are server subscription subscriptions that can subscribe from topic to queue . In the event that your JMS provider has no way to handle this, you may need to add some routing middleware to your topology. Apache Camel is nice, lightweight, but there are many others that can set up some routing in the middle without affecting real applications.

Update for detailed questions

• The lines below the line should be required (if your applications use messaging). The "Some distrib. Logic" field is not required. The “Some routing logic” field can be an ESB or in this case be implemented on a messaging server, for example, ActiveMQ with virtual addresses (or WebSphere MQ or, possibly, RabbitMQ among others).

• In the field of integration there are many buzwords. Simplified (depending on who you ask - ESB can also be considered an architectural model, but let it be simple), ESB is a server application (or the topology of several servers in practice), which is a central element of the integration landscape. An ESB server simply contains logical and small message flows that receive messages (files, etc.) from one application and send them to many applications, convert them to other formats, encrypt, and convert from one transport protocol, such as HTTP / SOAP to file, etc ..

JMS is a rather confusing and missing word. Java has, to some extent, dominated the corporate messaging domain in recent years, so JMS is sometimes used largely as a synonym for messages. However, messages (or message ordering, asynchronous messages, MOM = message-oriented middleware, etc.) should simply be considered as a family of similar transport protocols with a central relay server. Is this not only a Java thing. Many of the successful ESB settings I worked with actually used the messaging engine

• In your situation, I would not go into the academic / philosophical differences between ESB and EAI software. Most likely, they will do the same for you. Instead, look at hard facts like price, support, resource, monitoring, technology. functions, learning curve, etc. Be it Camel / ServiceMix, Mule, JBoss ESB, Microsoft BizTalk, IBM Message Broker, Tibco, etc.

• Huh! Perhaps it was a seller? ESB will be very good. The Messaging server will also work in your case, for example, ActiveMQ, as already mentioned. BPM types are good for organizing semi-automated business processes or if there is a core business logic at the integration level. Otherwise, avoid extra complexity.

Answer 2

• Regardless of the solution (pure JMS, ESB, EAI), should a part be implemented under the horizontal dashed line (application queues)?

Consumers should be implemented in such a way as to work with the solution of your choice, but you do not need to worry about creating a queue for each consumer or distribution logic (assuming that consumers can consume directly from the selected tech)

• How does using ESB (JBoss ESB, etc.) instead of "pure JMS (Active MQ, etc.)) help / hinder? Does ESB provide any advantage over JMS, which is only java (?). I got confused - ESB or JMS, even after referring to topics such as: JMS and ESB - how are they related? There is one answer that says: "JMS is not very suitable for integrating REST services, file systems, S / FTP, electronic mail, Hessian, SOAP, etc., which are better handled by the ESB, which supports these types natively. you have a process that uploads a 500 MB CSV file at midnight, and you want another system to collect the file, analyze the CSV and import into the database, this can be easily reached by the ESB, whereas a solution with just JMS will be bad Similarly, integration of REST services with load balancing / fault tolerance to multiple backend instances can be improved with an ESB that supports HTTP / S natively. "This only added to my confusion !!!

My opinion is that the ESB will complicate this decision too much. He designed (among other things) to help integrate with various technologies, but simpler solutions do it too - for example - Apache Camel provides a very simple way to communicate using a huge variety of vehicles (including ActiveMQ).

Not all JMS implementations are designed to connect to other languages, but ActiveMQ uses this STOMP connector.

• Is using EAI infrastructure (Apache Camel etc.) different from JMS or ESB approach? If so, how and what are the pros / cons?

Apache Camel and JMS are additional technologies such as JMS and ESB. Camel (& Spring Integration) are lightweight, simple and portable. ESBs are much more heavyweight and usually result in more communication with the ESB / application server.

• I was told that the ESB would not help, should BPM (or something else?) Be used to define and store the "routing logic" - is that true?

It depends on your “routing” logic, it seems to me that you do not need routing logic, you just need guaranteed delivery to 1payroll consumer and 1 customer support customer. BPM will be more useful if you want to selectively publish data / call a service based on some characteristic of that data.

I highly recommend reading http://activemq.apache.org/virtual-destinations.html using them:

• Send messages to the ActiveMQ broker on a virtual topic, for example. VirtualTopic.X
• Register Payroll and Helpdesk users as consumers in the queues that ActiveMQ dynamically creates in the topic - for example. Consumer.Payroll.VirtualTopic.X. Both consumers of payroll should be registered on one line.
• ActiveMQ will automatically save a token that represents what each consumer set did not consume. This means that 100% of the messages will be processed by the payroll consumer, but the message will not be sent to> 1 payroll consumer.
• Add / remove users as you wish.

NBI believe other products, for example. Apache QPID provides similar functionality - I just am most aware of ActiveMQ and have been successful with this approach