Using the mqtt protocol with kafka as a message broker

Question

How can we use the mqtt protocol with kafka as a message broker?

Clients (android / ios / desktop java apps, etc.) will create and consume messages using the mqtt phao client libraries, which are available in different languages, using kafka as a message broker.

Any tips?

Answer 1

You can use the Kafka source connector, which will transfer data from the MQTT broker, such as Mosquitto, to the Kafka cluster. See https://github.com/evokly/kafka-connect-mqtt

The easiest way to start the connector is in offline mode, where one instance will be launched on the Kafka cluster on one node. You can also run it in distributed mode (albeit with a much larger configuration), and this will distribute the connector in the cluster for more bandwidth. In distributed mode, you can develop a topology that allows horizontal scaling, parallel bandwidth and high availability. Implementing additional warranties requires additional load balancers, several MQTT brokers, and the latest will and testament scenarios to fix connector failures, but this is probably not the case.

Using the connectivity approach takes advantage of the Kafka cluster, ensuring that your connector is alive and restarting it if necessary. Distributed mode offers even more benefits.

Answer 2

AFAIK, there is no "official" MQTT connector for Kafka. But you can use the MQTT-Kafka bridge. For inspiration, see https://github.com/km4rcus/mqttKafkaBridge (note the error in this implementation: kafka themes cannot contain "/", so you probably want to replace them with "_" in messageArrived in the Bridge file .java)

Please note that this code is just a very simple solution, not scalable. It is a good idea, perhaps writing your own implementation according to your expectations is better. But you should keep it as simple as possible - this is one point of failure. While you receive your data in Kafka, you get some guarantees, but you do not have guarantees from the MQTT broker. When the bridge falls, you just lose your data ...

Answer 3

This is not a good idea. The MQTT client is usually very light and with limited resources. These devices or IoTs have small memory / processor power. Kafka's client is usually heavy. For example, a Kafka client must track the offset. It also requires interaction with Zookeepers. In short, Kafka is not suitable as an MQTT broker. You should choose a popular MQTT broker such as Mosquito.

Answer 4

As @Miroslav Prymek mentioned there is no official bridge. One of them was created by Jacklund. I upgraded the bridge to support Kafka v 0.8.1.1. Also contains a finished binary compatible with JRE 1.7. Here is a link to the MQTT Kafka Bridge .

Answer 5

You can use Mosca , the mqtt adapter written in javascript.

Mosca is a node.js mqtt broker that can be used stand-alone or embedded in another node.js application

Mosca supports various backends such as redis and mongodb, but also kafka. The Kafka MQTT Bridge app is included in the Mosca examples .

This article on links describes the solution in more detail.

Disclaimer: I am a member of the application of the example of Mosca Kafka.