Problems with Redis?

Question

I tried to put a lot of stress on my Redis for testing purposes and find out any upper limits. First, I downloaded it with 50,000 and 100,000 keys of 32 characters in size with values ​​of about 32 characters. For both sizes of keys it took no more than 8-15 seconds. Now I'm trying to put 4kb of data as a value for each key. The first 10,000 keys take 800 milliseconds. But from that moment on, it slowly slows down and installs as many as 50,000 keys, which takes 40 minutes. I am loading a database using NodeJs with node_redis (Mranney) . Is there some kind of error I'm making, or is it Redis only slow with large 4K sizes?

Another thing I found now is when I start another client parallel to the current one and update the keys, this second client finishes downloading the 50,000 keys with 4K values ​​in 8 seconds, while the first client still does its thing forever . Is this a bug in node or in the redis library? This is alarming and unacceptable for production.

Answer 1

You will need to get some kind of back pressure to perform bulk write operations from node to Redis. By default, node queues all records and does not apply an upper bound on the size of the outgoing queue.

node_redis has a leak event that you can listen to implement some basic backpressure.

Answer 2

The default redis configuration is not optimized for this use. I suspect that you are changing it to a disk with a page size of 32 bytes, which means that each key added must find 128 adjacent free pages and may end up using the system virtual machine or having to expand the swap file many times.

When you update the key, the space is already allocated, so you do not see performance problems.

Answer 3

Since I typed (key value) many times in NodeJ, which runs asynchronously, many socket connections open simultaneously. The NodeJs socket write buffer may be overloaded, and the GC may appear and play with the node process.

PS: I changed the redis memory configuration as Tom suggested, but it still did the same.