Kafka - manufacturer - "failed to send" pen

Question

I run 0.8 Kafka and create a manufacturer using the provided Java API.
The API functions for sending a message (or messages) return void.

Is there a way to get the status of a sent message? If it is sent or failed?

This is very important for us, because we read messages from a file, and we want to delete the file after sending all messages. But if there were errors and some messages were not sent, and I delete the file, this will lead to the loss of very important data.

Answer 1

You can configure your producer until he receives n kcks from the Kafka cluster (request.required.acks) so that you have some kind of guarantee that the data has been fixed correctly before deleting the original file.

If you really need to make sure the message is sent successfully, you can consider an alternative to making a synchronous producer process (producer .type = sync). This way, you can catch any exception caused by a lock call and act accordingly. The exception thrown by send () is kafka.common.FailedToSendMessageException.

Kafka Java API is not perfect, hope this helps you.