The second and third Kafka Connector distributed workers are not working correctly

With a Kafka cluster of 3 and a Zookeeper cluster from the same, I picked up one distributed node connector. This node successfully worked with one task. Then I picked up the second connector, which seemed to work, since a certain part of the code in the task was definitely executing. However, it seems that he did not survive (although without errors and did not survive, there was a lack of expected activity, while the first connector continued to function correctly). When I call the URL http://localhost:8083/connectors/mqtt/tasks , on each node connector it tells me that the connector has one task. I expect this to be two tasks: one for each node / worker. (Currently, the working configuration says tasks.max = 1 , but I also tried setting it to 3.

When I try to connect the third connector, I get an error message:

 "POST /connectors HTTP/1.1" 500 90 5 (org.apache.kafka.connect.runtime.rest.RestServer:60) ERROR IO error forwarding REST request: (org.apache.kafka.connect.runtime.rest.RestServer:241) java.net.ConnectException: Connection refused 

Attempting to call the connector's POST method again from the shell returns an error:

  {"error_code":500,"message":"IO Error trying to forward REST request: Connection refused"} 

I also tried upgrading to Apache Kafka 0.10.1.1, which was released today. I still see the problems. Each of the connectors runs on isolated Docker containers defined by a single image. They must be identical.

The problem may be that I try to run a POST request for http://localhost:8083/connectors for each worker, when I only need to run it once on one worker, and then the tasks for this connector will automatically spread to other workers . If so, how do I distribute tasks? Currently, I have the maximum number set to three, but only one of them works on the same desktop.

Update

In the end, I got things using essentially the same approach that Yuri suggested. I gave each employee a unique group identifier, and then gave each connector job the same name. This allowed the three connectors and their single tasks to share one offset, so that in the case of the receiver connectors, the messages they consumed from Kafka were not duplicated. They mainly work as standalone connectors, since workers have different group identifiers and thus will not interact with each other.

If the working connectors have the same group identifier, you cannot add multiple connectors with the same name. If you give the connectors different names, they will have different offsets and will use duplicate messages. If you have three employees in one group, one connector and three tasks, theoretically you will have an ideal situation when tasks distribute the offset, and employees should make sure that the tasks always work and are well distributed (each task requires a unique set of sections). In practice, the connector structure does not create more than one task, even if task.max is set to 3 and when the consumed theme tasks have 25 sections.

If anyone knows why I see this behavior, please let me know.

+6
source share
1 answer

I ran into a similar problem in the same situation as yours.

  • Task.max is configured for the theme, and distributed workers automatically determine which nodes process the theme. So, if you have 3 employees in the cluster and your theme configuration says task.max = 2, then only 2 out of 3 workers will handle the theme. Theoretically, if one of the workers fails, the third should take the workload. But..
  • The distributed connector turned out to be very unreliable: after adding / removing some nodes, the cluster broke down and all the employees did nothing, but tried to choose a leader and could not. The only way to fix it is to restart the entire cluster, and preferably all workers at the same time.

I chose another way: I used a standalone worker, and it works like a charm for me, because load balancing is done at the Kafka client level, and as soon as some worker falls, the cluster automatically balances and clients connect to unoccupied topics.

PS. Perhaps this will be useful to you. A conflicting connector does not allow an invalid payload that does not match the theme layout. As soon as the connector receives some kind of invalid message, it will die quietly. The only way to find out is to analyze the metrics.

+3
source

Source: https://habr.com/ru/post/1013369/


All Articles