Python Redis Connections

Question

I am using a Redis server with python.

My application is multithreaded (I use 20 to 32 threads for each process), and I also run the application on different machines.

I noticed that sometimes using Redis cpu is 100% and the Redis server stops responding / slows down.

I would like to use for each application 1 connection pool of only 4 connections. So, for example, if I run my application on 20 machines at most, there should be 20 * 4 = 80 connections to the redis server.

POOL = redis.ConnectionPool(max_connections=4, host='192.168.1.1', db=1, port=6379) R_SERVER = redis.Redis(connection_pool=POOL) class Worker(Thread): def __init__(self): self.start() def run(self): while True: key = R_SERVER.randomkey() if not key: break value = R_SERVER.get(key) def _do_something(self, value): # do something with value pass if __name__ = '__main__': num_threads = 20 workers = [Worker() for _ in range(num_threads)] for w in workers: w.join() 

In the above code, 20 threads must be started that receive the connection from the max size 4 connection pool when the command is executed.

When is the connection released?

According to this code ( https://github.com/andymccurdy/redis-py/blob/master/redis/client.py ):

  #### COMMAND EXECUTION AND PROTOCOL PARSING #### def execute_command(self, *args, **options): "Execute a command and return a parsed response" pool = self.connection_pool command_name = args[0] connection = pool.get_connection(command_name, **options) try: connection.send_command(*args) return self.parse_response(connection, command_name, **options) except ConnectionError: connection.disconnect() connection.send_command(*args) return self.parse_response(connection, command_name, **options) finally: pool.release(connection) 

After each command, the connection is released and returned to the pool.

Can someone verify that I understood the idea correctly and the above sample code will work as described?

Because when I see redis connections, there are always more than 4.

EDIT: I just noticed in the code that the function has a return statement before ending. What is the goal finally?

Answer 1

As Matthew Scrugg mentioned, the finally clause is executed at the end of the test. In this particular case, it serves to free the connection back to the pool when it is completed, instead of leaving it open.

Regarding insensitivity, see what your server does. What is the memory limit of your Redis instance? How often do you save a disk? Are you running a Xen-based virtual machine, such as an AWS instance? Are you replicating, and if so, how many slaves are they in good condition or do they often require complete re-synchronization of data? Any of your save commands?

You can answer some of these questions using the command line interface. For example, redis-cli info persistence will tell you information about the process of saving to disk, redis-cli info memory will tell you about the consumption of your memory.

When you receive the save information that you want to see specifically, rdb_last_bgsave_status and rdb_last_bgsave_time_sec . They will tell you if the last salvation was successful and how much time has passed. The longer this takes place, the higher the likelihood that you will encounter resource problems, and the higher the likelihood that you will encounter a slowdown that may appear as immunity.

Answer 2

The final block will always work, although there is a return statement in front of it. Perhaps you look at redis-py / connection.py, pool.release (connection) only established a connection to the pool of available connections, so the connection is still alive. About redis server cpu, your application will always send a request and has no interruptions or sleep, so it just uses more and more cpus, but not memory. and using cpu is not related to open file numbers.