How does Kafka deal with a consumer who is slower than other consumers?

Question

Say I have 20 sections and five workers. Each section is assigned a worker. However, one worker is slower than other machines. It still processes (i.e., not the slow consumer described here ), but at 60% the speed of other machines. This may be due to the fact that the worker is running on a slower virtual machine on AWS EC2, a broken disk or processor, or much more. Does Kafka help balance correctly to give a slow worker fewer partitions?

Answer 1

Kafka doesn't really care about how fast messages are used. This is not even related to how many consumers there are or how many times each message is read. Kafka simply captures messages in sections and puts them at a given time.

The responsibility of a consumer group is to ensure that messages are read evenly and on time. In your case, you have two problems: reading one set of sections is delayed, and then processing messages from these sections lags behind.

API Kafka , , , . , , .

, - - Storm. , , .