According to the documentation on kafka javadocs , if I:
- Subscribe to Template
- Create a theme that matches the pattern.
There must be a rebalancing that forces the consumer to read this new topic. But this does not happen.
If I stop and start with the consumer, he will bring up a new topic. Therefore, I know that the new theme follows the pattern. There is a possible duplicate of this question at https://stackoverflow.com/questions/37120537/whitelist-filter-in-kafka-doesnt-pick-up-new-topics , but this question has not been received anywhere.
I see kafka logs and there are no errors, it just does not cause rebalancing. Rebalancing starts when consumers join or die, but not when creating new topics (even when sections are added to existing topics, but this is a different object).
I use kafka 0.10.0.0 and the official Java client for the "New Consumer API", which means the GroupCoordinator broker instead of the thick client + zookeeper.
This is the code for the consumer fetch:
public class SampleConsumer {
public static void main(String[] args) throws IOException {
KafkaConsumer<String, String> consumer;
try (InputStream props = Resources.getResource("consumer.props").openStream()) {
Properties properties = new Properties();
properties.load(props);
properties.setProperty("group.id", "my-group");
System.out.println(properties.get("group.id"));
consumer = new KafkaConsumer<>(properties);
}
Pattern pattern = Pattern.compile("mytopic.+");
consumer.subscribe(pattern, new SampleRebalanceListener());
while (true) {
ConsumerRecords<String, String> records = consumer.poll(1000);
for (ConsumerRecord<String, String> record : records) {
System.out.printf("%s %s\n", record.topic(), record.value());
}
}
}
}
In the producer, I post to topics named mytopic1, mytopic2, etc.
Templates are pretty much useless if rebalancing doesn't start.
Do you know why rebalancing does not occur?