All geek questions in one place

Kafka Consumer for Multiple Themes

I have a list of topics (currently 10), the size of which may increase in the future. I know that we can generate several threads (for each topic) to consume from each topic, but in my case, if the number of threads increases, then the number of threads consuming threads increases, which I donโ€™t want, since the threads are not receive data too often, so the threads will sit perfectly.

Is there a way for one consumer to consume all topics? If so, how can we achieve this? Also, how will Kafka compensation be supported? Please offer answers.

+4
source share
2 answers

We can subscribe to several topics using the following API: consumer.subscribe (Arrays.asList (topic1, topic2), ConsumerRebalanceListener obj)

The consumer has information about the topic, and we can use it with consumer.commitAsync or consumer.commitSync () by creating an OffsetAndMetadata object as follows.

ConsumerRecords<String, String> records = consumer.poll(long value);
for (TopicPartition partition : records.partitions()) {
    List<ConsumerRecord<String, String>> partitionRecords = records.records(partition);
    for (ConsumerRecord<String, String> record : partitionRecords) {
        System.out.println(record.offset() + ": " + record.value());
    }
    long lastOffset = partitionRecords.get(partitionRecords.size() - 1).offset();
    consumer.commitSync(Collections.singletonMap(partition, new OffsetAndMetadata(lastOffset + 1)));
}
+5
source

, , . zookeeper, kafka- . , , zookeeper, . , kafka .

+1

Source: https://habr.com/ru/post/1655041/

More articles:


All Articles