Topic cannot be found when creating posts: UNKNOWN_TOPIC_OR_PARTITION

Question

I have a kafka cluster with two nodes (EC2 instances) where each node is used as a separate broker. When I run the manufacturer in the leader instance with the following command:

kafka-console-producer.sh --broker-list localhost:9092 --topic test 

I get the following errors.

test message [2017-01-09 13: 22: 39,483] WARN Error retrieving metadata with correlation identifier 0: {test = UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2017-01-09 13: 22: 39,562] WARN Error retrieving metadata with correlation identifier 1: {test = UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2017-01-09 13: 22: 39,663] WARN Error retrieving metadata with correlation identifier 2: {test = UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) ...

A topic list with kaka-topics.sh shows that the topic exists.

Description of the topic using:

 kafka-topics.sh --zookeeper localhost:2181 --describe --topic test 

returns

  Topic:test PartitionCount:8 ReplicationFactor:1 Configs: Topic: test Partition: 0 Leader: 1 Replicas: 1 Isr: 1 Topic: test Partition: 1 Leader: 2 Replicas: 2 Isr: 2 Topic: test Partition: 2 Leader: 1 Replicas: 1 Isr: 1 Topic: test Partition: 3 Leader: 2 Replicas: 2 Isr: 2 Topic: test Partition: 4 Leader: 1 Replicas: 1 Isr: 1 Topic: test Partition: 5 Leader: 2 Replicas: 2 Isr: 2 Topic: test Partition: 6 Leader: 1 Replicas: 1 Isr: 1 Topic: test Partition: 7 Leader: 2 Replicas: 2 Isr: 2 

I am using kafka 0.10.1.1.

server.propertes file contains:

 listeners=PLAINTEXT://0.0.0.0:9092 advertised.listeners=PLAINTEXT://0.0.0.0:9092 port=9092 host.name=kafka-node1(kafka-node1 for the second host) advertised.host.name=kafka-node1(kafka-node2 for the second host) advertised.port=9092 

When I try to create messages from the second host, I get the following message:

WARN The received error triggers a response with correlation identifier 1 on topic-partition test-4, retrying (2 attempts left). Error: NOT_LEADER_FOR_PARTITION (Org.apache.kafka.clients.producer.internals.Sender) ....

Can anybody help? Thanks.

Answer 1

Make a replication factor of 2 for the topic test and you will no longer receive this exception. Since you have a 2-broker cluster, run the following command to create the messages:

 kafka-console-producer.sh --broker-list localhost:9092,localhost:9093 --topic test 

This will send messages to both brokers in the clusters.