Kafka manufacturers / brokers not using the specified IP address

So, I have a cluster of 6 machines. These computers have 2 IP addresses (open and closed). Private IPs use a high-speed network (1 Gbit / s), and the public uses a slower network (120 Mbit / s).

When setting up my kafka / zookeeper cluster, I set up a private IP as a broker ip. Now my configuration is fine, as I see the output saying that all my brokers are connected to the correct IP

 INFO Connecting to zookeeper on 10.11.10.2:2181,10.11.10.3:2181,10.11.10.4:2181,10.11.10.5:2181,10.11.10.6:2181,10.11.10.7:2181 (kafka.server.KafkaServer)

Now that I tested the manufacturer’s performance, I could get a bandwidth of more than 120 Mbps.

bin/kafka-producer-perf-test.sh --topic highthr2 --num-records 10000 --record-size 100000 --throughput 10000 --producer-props acks=-1 bootstrap.servers=10.11.10.2:9092,10.11.10.3:9092,10.11.10.4:9092,10.11.10.5:9092,10.11.10.6:9092,10.11.10.7:9092

When I used nload to check how much data goes through each interface, I found that data is coming in and out through a public IP interface (slower network interface). I double-checked the zookeeper and kafka logs, but they both say that my brokers are on a private IP.

Any ideas on what is the issue or what do I need to change?

+4
source share

Source: https://habr.com/ru/post/1665431/


All Articles