With a Kafka cluster of 3 and a Zookeeper cluster from the same, I picked up one distributed node connector. This node successfully worked with one task. Then I picked up the second connector, which seemed to work, since a certain part of the code in the task was definitely executing. However, it seems that he did not survive (although without errors and did not survive, there was a lack of expected activity, while the first connector continued to function correctly). When I call the URL http://localhost:8083/connectors/mqtt/tasks
, on each node connector it tells me that the connector has one task. I expect this to be two tasks: one for each node / worker. (Currently, the working configuration says tasks.max = 1
, but I also tried setting it to 3.
When I try to connect the third connector, I get an error message:
"POST /connectors HTTP/1.1" 500 90 5 (org.apache.kafka.connect.runtime.rest.RestServer:60) ERROR IO error forwarding REST request: (org.apache.kafka.connect.runtime.rest.RestServer:241) java.net.ConnectException: Connection refused
Attempting to call the connector's POST method again from the shell returns an error:
{"error_code":500,"message":"IO Error trying to forward REST request: Connection refused"}
I also tried upgrading to Apache Kafka 0.10.1.1, which was released today. I still see the problems. Each of the connectors runs on isolated Docker containers defined by a single image. They must be identical.
The problem may be that I try to run a POST request for http://localhost:8083/connectors
for each worker, when I only need to run it once on one worker, and then the tasks for this connector will automatically spread to other workers . If so, how do I distribute tasks? Currently, I have the maximum number set to three, but only one of them works on the same desktop.
Update
In the end, I got things using essentially the same approach that Yuri suggested. I gave each employee a unique group identifier, and then gave each connector job the same name. This allowed the three connectors and their single tasks to share one offset, so that in the case of the receiver connectors, the messages they consumed from Kafka were not duplicated. They mainly work as standalone connectors, since workers have different group identifiers and thus will not interact with each other.
If the working connectors have the same group identifier, you cannot add multiple connectors with the same name. If you give the connectors different names, they will have different offsets and will use duplicate messages. If you have three employees in one group, one connector and three tasks, theoretically you will have an ideal situation when tasks distribute the offset, and employees should make sure that the tasks always work and are well distributed (each task requires a unique set of sections). In practice, the connector structure does not create more than one task, even if task.max is set to 3 and when the consumed theme tasks have 25 sections.
If anyone knows why I see this behavior, please let me know.