Kafka several consumers for the section

Question

I have a producer who writes posts to a topic / section. To maintain order, I would like to go with one section, and I want 12 consumers to read all the messages from this separate section (not a single group of consumers, all messages should not go to all consumers). Is it possible? I read several forums that only one consumer can read per section.

Answer 1

You can use SimpleConsumer to achieve exactly what you ask for - there are no consumer groups, all consumers can read one section. However, this approach means that you have to deal with bias handling and broker error handling yourself.

Another option is to use a high-level consumer with different consumer groups (you can simply assign an arbitrary UUID to each consumer). Thus, you can consume one topic / section with all consumers and be able to make biases and handle the disconnection of the broker.

The rule "only one consumer can consume a topic / section" applies only to consumer groups, for example. only one IN GROUP consumer can use one topic / section at a time.

Answer 2

If you have multiple users in one section, this exceeds your initial order-keeping requirements. Although, you would have ordered storage, but consumption would not have been ordered. Make sure you are really fine with this. If so, you can simply treat each consumer as a different group of consumers.