I doubt that maximizing the use of the Redis processor will benefit your backend design. The right question is whether Redis is efficient enough to maintain your bandwidth for a given delay. Redis is a single-threaded server: at 80% CPU usage, latency is likely to be very poor.
I suggest you measure the latency while the redis-benchmark works to make sure that it is suitable for your needs before trying to increase the consumption of the Redis processor. To do this, you can use the -latency parameter for redis-cli:
- launch redis server
- try redis-cli -latency, pay attention to the avg value, stop it
- in another window, run the test and make sure it works for a while
- try redis-cli -latency, pay attention to the avg value, stop it
- stop test
- compare two avg values
Now, if you really want to increase the consumption of the Redis processor, you need either an effective client program (for example, redis-benchmark) that can handle several connections at the same time, or several instances of your client program.
Lua is a quickly interpreted language, but it is still an interpreted language. It will be one or two orders of magnitude slower than C. Redis code works much faster when parsing / generating its protocol than lua-redis, so you can not saturate Redis with a unique Lua client (unless you use O (n ) Redis - see below).
webdis is implemented in C with an effective client library, but must parse the http / json protocols, which, as it turns out, are more detailed and complex than the Redis protocol. It probably consumes more processor than Redis itself for most operations. This way you will not saturate Redis with a single instance of webdis.
Here are a few examples to saturate Redis with several Lua clients.
If this has not already been done, I suggest that you first take a look at the Redis test page .
If you run your test in the same field as Redis:
The key point is to highlight the Redis kernel and run client programs on other kernels. On Linux, you can use the taskset command to do this.
# Start Redis on core 0 taskset -c 0 redis-server redis.conf
Lua should use pipelining to maximize throughput and reduce system activity.
local redis = require 'redis' local client = redis.connect('127.0.0.1', 6379) for i=1,1000000 do local replies = client:pipeline(function(p) for j=1,1000 do local key = 'counter:'..tostring(j) p:incrby(key,1) end end) end
On my system, the Lua program takes up more than 4x the Redis processor, so you need more than 4 cores to saturate Redis with this method (the 6-core box should be fine).
If you run your test in a field other than Redis:
Unless you are running on processor-based starvation computers, the bottleneck is likely to be the network in this case. I don’t think you can saturate Redis with anything but a 1 GbE link.
Be sure to convey your requests as much as possible (see the previous Lua program) in order to avoid network bottlenecks and reduce the cost of network interruptions on the CPU (filling in Ethernet packets). Try running Redis on a kernel that is not connected to a network card (and handles network interrupts). You can use tools like htop to check this last point.
Try running your Lua clients on other computers on the network, if you can. Again, you will need a large number of Lua clients to saturate Redis (6-10 should be fine).
In some cases, a single Lua process is sufficient:
Now you can redirect Redis with one Lua client if each request is expensive enough. Here is an example:
local redis = require 'redis' local client = redis.connect('127.0.0.1', 6379) for i=1,1000 do local replies = client:pipeline(function(p) for j=1,1000 do p:rpush("toto",i*1000+j) end end) end N = 500000 for i=1,100000 do local replies = client:pipeline(function(p) for j=1,10 do p:lrange("toto",N, N+10) end end) end
This program fills the list with 1M elements, and then uses the lrange command to extract 10 elements from the middle of the list (worst case for Redis). Thus, every time a request is executed, 500K elements are checked by the server. Since only 10 elements are returned, they are quickly parsed by lua-redis, which will not consume the processor. In this situation, all CPU consumption will be on the server side.
Final words
Probably faster Redis clients than redis-lua:
You can try them.