I have one instance of kafka installed on a VM with 8 cores and 32 GB of RAM.
I write to him (producing) from 10 different machines and consume from one machine, all of which are on the same network.
The size of the data I create is ~ 35 Mbps.
For some reason, most of the time I can’t consume more than ~ 10 Mbit / s (for a limited period of time I manage to use all the data produced), although the kafka and consumer servers are mostly inactive (so I don’t think this is a storage problem )
Can kafka ignore some of the data received?
Some parameter values that may be useful for analysis:
num.network.threads=32
num.io.threads=16
message.max.bytes=2147483647
num.partitions=10
log.retention.ms=120000 (2 minutes)