Why does my kafka have a message in one section?

Question

I have two kafka brokers with two partitions on the local machine and use the following tool to write one local file in the kafka test2 theme.

# create topic ./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 2 --topic test2 Created topic "test2". # write 15MB file to kafka, very fast!! kafka-console-producer.sh --broker-list localhost:9093,localhost:9094 --topic test2 < data.txt # read data from kafka ./kafka-console-consumer.sh --zookeeper localhost:2181 --topic test2 --from-beginning 

Then I find that all messages are in one section, how to debug this?

 $ kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9093,localhost:9094 --topic test2 --time -1 SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. test2:0:68263 test2:1:0 

Section Status:

 $ kafka-topics.sh --describe --zookeeper localhost:2181 --topic test2 Topic:test2 PartitionCount:2 ReplicationFactor:2 Configs: Topic: test2 Partition: 0 Leader: 1 Replicas: 1,2 Isr: 1,2 Topic: test2 Partition: 1 Leader: 2 Replicas: 2,1 Isr: 2,1 

Answer 1

If I understand you correctly, you are wondering why the data is not replicated to another section. I think you may not understand how Kafka performs replication.

According to the Kafka Documentation, the entry goes only to one section in the subject:

Manufacturers publish data on topics of their choice. The manufacturer is responsible for choosing which record to assign to which section in the topic.

Thus, replication does not reflect data from one partition to another, but rather saves a copy of each partition to another server.

Kafka replicates the log for each topic section on a custom number of servers (this replication rate can be set for each section). This allows you to automatically refuse these replicas when a server crashes in the cluster, so messages remain available if there are failures.

So, at the end, all the data is written in one section, but on both of your servers there is a copy of this section.