How to create a separate Kafka listener for each topic dynamically in springboot?

Question

I am new to Spring and Kafka. I am working on a use case [using SpringBoot-kafka] where users are allowed to create kafka themes at runtime. The Spring application is expected to subscribe to these topics programmatically at runtime. What I know so far is that Kafka listeners are development time, and therefore topics should be specified before starting. Is there a way to dynamically subscribe to kafka themes in SpringBoot-Kafka integration?

https://github.com/spring-projects/spring-kafka/issues/132

The current approach that I plan to implement does not use Spring-Kafka integration, and Kafka consumer himself uses [using java code], as mentioned here spring boot kafka consumer - how to properly consume kafka messages from Spring boot

Answer 1

Kafka listeners are just “development time” if you want to specify them using annotations. Spring -kafka allows you to create them dynamically, see KafkaMessageListenerContainer .

The simplest example of a Kafka listener created on the fly would be:

Map<String, Object> consumerConfig = ImmutableMap.of(
    BOOTSTRAP_SERVERS_CONFIG, "brokerAddress",
    GROUP_ID_CONFIG, "groupId"
);

DefaultKafkaConsumerFactory<String, String> kafkaConsumerFactory =
        new DefaultKafkaConsumerFactory<>(
                consumerConfig,
                new StringDeserializer(),
                new StringDeserializer());

ContainerProperties containerProperties = new ContainerProperties("topicName");
containerProperties.setMessageListener((MessageListener<String, String>) record -> {
     //do something with received record
} 

ConcurrentMessageListenerContainer container =
        new ConcurrentMessageListenerContainer<>(
                kafkaConsumerFactory,
                containerProperties);

container.start();

. : http://www.douevencode.com/articles/2017-12/spring-kafka-without-annotations/