After rebooting Linux, Kafka throws "no brokers found while trying to rebalance",

Question

I followed the excellent step-by-step guide on installing Kafka on Linux . Everything worked fine for me until I restarted Linux. After rebooting, I get the following error when I try to use the queue with kafka-console-consumer.sh .

 $ ~/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic TutorialTopic --from-beginning 

[2016-02-04 03: 16: 54,944] WARN [console-consumer-6966_bob-kafka-storm-1454577414492-8728ae43], brokers were not found when trying to rebalance . (Kafka.consumer.ZookeeperConsumerConnector)

Before I ran the kafka-console-consumer.sh script, I moved the data to the Kafka queue using the kafka-console-producer.sh script. These steps worked without problems before rebooting Linux.

I can fix the error by manually starting Kafka; but I would prefer Kafka to start automatically.

What can cause Kafka to start incorrectly after a restart?

Answer 1

So today I had a similar problem in our hortonworks. It seems that zookeeper did not start correctly. I first tried the kakfa-console manufacturer and got the exception below:

 kafka-console-producer.sh --broker-list=localhost:9093 --topic=some_topic < /tmp/sometextfile.txt``` kafka.common.KafkaException: fetching topic metadata for topics``` 

The solution for me was to restart the server on which they stopped responding. yours may be different, but play with the console producer and see what mistakes you get.

Answer 2

By default, kafka saves all the information in the /tmp folder. The tmp folder is deleted each time it is restarted. You can change the location of these files by changing the config/server.properties property in the config/server.properties property log.dirs .

Answer 3

Removing everything related to zookeeper and kafka from the / tmp folder worked for me. But my system is not production, so caution should be exercised for other people.

Answer 4

For me, when a user launches a message from a cluster of brokers. But the Kafka server was closed. Thus, the message "no brokers were found while trying to rebalance" was returned. When I restart the kafka server, the error disappeared.

Answer 5

I have the same problem today when starting a user:

WARN [console-consumer-74006_my-machine.com-1519117041191-ea485191], no brokers found when trying to rebalance. (kafka.consumer.ZookeeperConsumerConnector)

This turned out to be a problem on the disk on the machine. When space was freed, the problem was resolved.

Answer 6

In order for the consumer-producer to work, you must run a broker. A broker is that intermediate messages are transmitted from producer to consumer.

sh kafka/bin/kafka-server-start.sh kafka/config/server.properties

Broker defaults to port 9092