How to create a theme in apache kafka?

Question

What is the best way to create themes in kafka?

• How many replicas / partitions will be determined when creating topics?

In the new manufacturer API, when I try to post a message to a non-existing section, it first crashes and successfully posts.

• I would like to know what is the relationship between the replica, partitions and the number of cluster nodes.
• Do I need to create a topic before posting?

Answer 1

When starting the Kafka broker, you can define a set of properties in the conf/server.properties . This file is simply a key value property file. One of the properties is auto.create.topics.enable , if it is set to true (the default), Kafka will automatically create topics when sending messages to topics that do not exist.

All configuration options you can find are defined here . IMHO, a simple rule to create the following: the number of replicas should be no less than the number of nodes that you have. The number of topics should be a multiplier of the number of nodes in your cluster, for example:

• You have 9 cluster nodes
• Your topic should have 9 sections and 9 replicas or 18 sections and 9 replicas or 36 sections and 9 replicas and so on ...

Answer 2

The section number determines the parallelism of the topic, since one section can be consumed by only one consumer in the consumer group. For example, if you have only 10 sections for a topic and 20 consumers in a consumer group, 10 users are idle without receiving any messages. The number really depends on your application, but 1-1000s are all reasonable.

The replica number is determined by your durability. For a topic with a replication rate of N, Kafka can tolerate the failure of the N-1 server without losing messages sent to the log. 3 replicas are a common configuration. Of course, the replica number must be less than or equal to your broker number.

auto.create.topics.enable property when Kafka allows you to automatically create a theme on the server. If this parameter is set to true when applications try to produce, consume or retrieve metadata for a topic that does not exist, Kafka will automatically create a theme with a default replication rate and number of partitions. I would recommend disabling it in production and creating themes in advance.

Answer 3

set the property auto.create.topics.enable=true in the server.properties file, if you have several brokers, do the same for the entire * .properties server file and restart your kafka server. But make sure that you set the partitions for the corresponding number on the * .properties num.partitions=int server, otherwise there will be a performance problem if the partitions are expanded later.

Answer 4

The basic level of parallelism in Kafka is a section. Both on the side of the manufacturer and on the side of the broker, entries in different sections can be performed completely in parallel.

What you need to keep in mind

• More sections Requires more open files with files
• More sections may increase inaccessibility
• More sections may increase final delay

As a rule, it is probably a good idea to limit the number of sections per broker to 100 xbxr , where b is the number of brokers and r is the replication coefficient.

For example: If you have 9 brokers / nodes in your cluster, your theme may have

• 1800 partitions with 3 replicas or
• 900 sections and 2 replicas

---

EDIT: See the article How to choose the number of topics / sections in a Kafka cluster? for further details (the answer was made from there)

Answer 5

I would like to share my recent experience, which I described on my blog, “The side effect of getting Kafka’s topic metadata,” as well as provide answers to some of the questions raised here.

1) What is the best way to create themes in kafka? Do I need to create topics before posting?

I think that if we know that we are going to use a topic with a fixed name Kafka in advance, we would be better off creating a topic before writing or reading messages from it. Usually this can be done in the script after starting with bin / kafka-topics.sh, see, for example, the official documentation . Or we can use KafkaAdminClient, which was introduced in Kafka 0.11.0.0.

On the other hand, I see some cases where we would need to generate a topic name on the fly. In these cases, we cannot find out the fixed topic name, and we can rely on the property "auto.create.topics.enable". When it is turned on, the theme will be created automatically. And that raises the second question:

2) What actions will cause creation when auto.create.topics.enable is true

In fact, as @Lan already pointed out

If this parameter is set to true when applications try to create, use, or retrieve metadata for a topic that does not exist, Kafka will automatically create a theme with a default replication rate and number of partitions.

I would like to put it even simpler:

If automatic theme creation is enabled for Kafka brokers, then whenever a Kafka broker sees a specific topic name, this theme will be created if it does not already exist.

And also the fact that the selection of metadata will automatically create a topic is often ignored by people, including me. A specific example of this is the use of the API customer.partitionFor (topic) API, this method creates the given topic if it does not exist.

For those interested in the more detailed information that I mentioned above, you can take a look at my own blog post on the same topic . A side effect of getting Kafka theme metadata .