Event-sourcing: when (and not) should the message queue be used?

Question

I am creating a project from scratch using event-sourcing with Java and Cassandra. My applications, on which we rely on microservices, and in some cases the use of information, are processed asynchronously. I was wondering what part of the message queue (for example, Rabbit, Active MQ Artemis, Kafka, etc.) will play to improve the technological stack in this environment, and if I understand the scripts, if I do not use it.

Answer 1

I would start by sharing a messaging infrastructure like RabbitMQ, with streaming / storage / event handling like Kafka. These are two different things made for two (or more) different purposes.

As for the source of events, you should have a place where you should store events. This store should only be append and support fast reading of unstructured data based on identity. One example of this persistence is EventStore .

Event identification comes with CQRS, which means that you must project your changes (events) into another store that you can request. This is done by projecting events into this repository, where events are processed to change the state of a domain object. It is important to understand that using message infrastructure for forecasts is usually a bad idea. This is due to the nature of messaging and two-phase commit.

If you look at how events are saved, you can see that they are stored in the repository as a single transaction. If you need to post events, this will be a different transaction. Since you share two different infrastructures, everything can break.

The problem with messages as such is that messages generally guarantee delivery "at least once," and the order of messages is usually not guaranteed. Also, when your message user fails and the NACK sends the message, it will be resent, but usually a bit later, breaking the sequence again.

Streamlining and duplication concerns someone who is not applicable to event streaming servers such as Kafka. In addition, EventStore guarantees that only event delivery will be possible only if you are using a catch-up subscription.

In my experience, messages are used to send commands and to implement an event-driven architecture to connect independent services in a reactive way. Event repositories, on the other hand, are used to save events, and only the events that get there are then projected into the request repository, and also published on the message bus.

Answer 2

Make sure you clearly understand the difference between send (command) and publish (event). Udi Dahan addresses this topic in his essay on buses and brokers .

In most cases, when you are a source of events, you do not want to restore the state from published events. If you need a condition, then request a technical authority / record book for the story and restore the state from the story.

On the other hand, the activity associated with the event from the message queue should be excellent. When one event (plus the status of the subscriber) has everything you need, then the bus shutdown is fine.

In some cases, you can do both. For example, if you were updating cached views, you subscribed to various BobChanged events to find out when your cache data was obsolete; To restore an obsolete view, you reload the history view and convert it to an updated view.

Answer 3

In the world of event management applications, message queues typically allow you to implement a publish-subscribe style with a model of communication between producers and consumers. In addition, they usually help you with delivery guarantees: which messages were delivered to subscribers and which were not.

But they do not save all messages indefinitely. You must have an event store for any event source.

The question is not to stand or not stand in line, but it is more like:

• Can this thing store a huge amount of events indefinitely?
• Does he have publishing-subscribing features?
• Does it provide a delivery guarantee at least once?

So, you should use something like Kafka or EventStore to have it all out of the box. In addition, you can combine the event store with the message queue manually, but it will be more active.