Store images in Apache Kafka?

Do any large Kafka users (like LinkedIn) have uploaded images in Kafka? I like the architectural simplicity of putting everything in a magazine, but I am concerned that in practice this may not be possible.

+4
source share
1 answer

Theoretically, there are no restrictions, since you can easily store images as binary in the Kafka queue. But there may be other problems. I would try to think about it

There is a parameter in the user configuration message.max.bytes, and the default value is 1000000. This is actually done so that the brokers do not have enough memory, because the consumer does not have the ability to stream messages and must allocate memory to read the message. A workaround may be to compress messages in Kafka to save spaces.

The most suitable condition that I can find is to write my own implementation of the client, passing the preliminary distribution of the send and receive buffers, ensuring the transfer of files with a zero copy through Kafka, as shown in this discussion.

You need to check whether such efforts can be made to achieve the same.

+1
source

Source: https://habr.com/ru/post/1529838/


All Articles