Does Apache Kafka provide an asynchronous subscription callback API?

Question

My project sees Apache Kafka as a potential replacement for an aging JMS-based messaging approach. To make this transition as smooth as possible, it would be ideal if the queue replacement system (Kafka) had an asynchronous subscription mechanism similar to our existing JMS project engine using MessageListener and MessageConsumer to subscribe to topics and receive asynchronous notifications. I don’t care if Kafka does not strictly comply with the JMS API, but, on the contrary, I would prefer not to reverse engineer our entire class of publish-subscribe classes if I do not need it.

I can find all kinds of polls, but so far I have not been able to find examples with the notification of the client about new messages through asynchronous notification.

Does anyone know if the current version of Kafka (0.10.2 at the time of publication) provides such an API, or am I trying to rewrite my old code with a poll?

Answer 1

Kafka clients only provide on-demand aggregation, but you can use spring-kafka . It provides a MessageListener interface and KafkaListener annotation, etc. See the documentation .

Answer 2

You might want to try Confluent Kafka JMS Client.

http://docs.confluent.io/3.2.0/clients/kafka-jms-client/docs/index.html

Kafka JMS Client is a JMS API wrapper on top of the Kafka Producer / Consumer API, so it has all the standard JMS 1.1 interfaces. This is an Enterprise feature (subscription), but if you download Confluent Enterprise, you can try it for 30 days for free.

https://www.confluent.io/download/

Answer 3

If you can take some time to wait after all your messages have been delivered, you can use a timer and call consumer.poll (0), which immediately returns with available messages. After you consume the message, you set the timer again with the same acceptable delay, say 100 ms.

When throughput is low, batches will be small, and this delay will be more likely to interfere. However, since the script is asynchronous, the delay is also not very important. You never know exactly when a message arrives anyway.

When throughput is very high, lots will be large. For the new Kafka consumer, the default value of fetch.max.bytes is 52428800. The additional delay will be relatively small compared to the time taken to process a large batch of messages.

You can wrap this with a small component to which you provide a function that matches your current MBean handler.