How to get confirmation from Kafka

How exactly do I receive confirmation from Kafka when a message is consumed or processed. It might seem silly, but is there a way to find out the start and end offset of this message for which ack was received?

+6
source share
2 answers

What I have found so far is 0.8, they have introduced the following method of selecting from the offset for reading.

kafka.api.OffsetRequest.EarliestTime () finds the beginning of the data in the logs and starts streaming from there, kafka.api.OffsetRequest.LatestTime () will send only new messages.

sample code https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example

Still not sure about confirmation

+1
source

Kafka is not really structured for this. To understand why, review the design documentation here .

To provide a one-time confirmation, you need to create some external tracking system for your application, where you explicitly write confirmations and implement locks on the transaction ID to ensure that things are processed only once. Estimated implementation costs, such as a system, are extremely high and are one of the main reasons large transactional systems require relatively exotic equipment and may have lower scalability than systems like Kafka.

If you don't need strong longevity semantics, you can use the group APIs to keep track of when the last message was read. This ensures that each message is read at least once. Please note: because the group API does not provide you with the ability to explicitly track your own processing logic, your actual processing guarantees in this scenario are rather weak. Schemes that rely on idempotent processing are common in this environment.

Alternatively, you can use the poorly named SimpleConsumer API (which is pretty complicated to use), which allows you to explicitly track the timestamps in your application. This is the highest level of processing guarantee that can be achieved using the Kafka native API, since it allows you to track your applications with their own data processing, which are read from the queue.

+1
source

Source: https://habr.com/ru/post/950690/


All Articles