Counting the number of posts stored in the Kafka topic

Question

I am using the 0.9.0.0 version of Kafka and want to count the number of posts in the topic without using the kafka-console-consumer.sh administrative script.

I tried all the commands in the Java answer , How to get the number of messages in a topic in apache kafka, but none of them give a result. Can anyone help me here?

Answer 1

You can try the following command:

bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092,localhost:9093,localhost:9094 --topic test-topic --time -1 

Then we summarize all the calculations for each section.

Answer 2

You can summarize all bills using this:

 .../bin/kafka-run-class kafka.tools.GetOffsetShell --broker-list <<broker_1>>:9092,<<broker_2:9092>>... --topic <<your_topic_name>> --time -1 | while IFS=: read topic_name partition_id number; do echo "$number"; done | paste -sd+ - | bc 

Answer 3

From a technical point of view, you can simply use all the messages from the topic and read them:

Example:

 kafka-run-class.sh kafka.tools.SimpleConsumerShell --broker-list localhost:9092 --topic XYZ --partition 0* 

However, the kafka.tools.GetOffsetShell approach will give you offsets, not the actual number of posts in the topic. This means that if the topic is condensed, you will get two different numbers if you count messages, consuming them or reading offsets.

Thematic Compaction: https://kafka.apache.org/documentation.html#design_compactionbasics

Answer 4

You can also do this using awk and a simple loop

 for i in 'kafka-run-class kafka.tools.GetOffsetShell --broker-list broker:9092 --time -1 --topic topic_name| awk -F : '{print $3}''; do sum=$(($sum+$i)); done