I recently had a problem with celery and redis: I used too many connections to my redis cloud account.
I seem to have fixed the issue so far with better settings and a big redis plan.
However, this led me to some experiments on how redis and celery work together, and I don’t understand something: the number of redis ConnectionPool that are created by the redis Python module.
I set up celery to use redis as a broker and result.
Here is my celery configuration:
CELERY_TIMEZONE = 'Europe/Paris' CELERY_BROKER_URL = REDIS_URL CELERY_RESULT_BACKEND = REDIS_URL CELERY_TASK_SERIALIZER = 'pickle' CELERY_SEND_EVENTS = False CELERY_IMPORTS = ('task1', 'task2') CELERY_ACCEPT_CONTENT = ['pickle'] CELERY_REDIS_MAX_CONNECTIONS = 2 BROKER_POOL_LIMIT = 1 BROKER_HEARTBEAT = None BROKER_TRANSPORT_OPTIONS = { 'max_connections': 30 }
I start celery with the following command:
celery worker --without-gossip --without-mingle --without-heartbeat --concurrency=1 --app=backZest.celery
Immediately after the celery boots, I already have 10 connections to my redis instance: is this normal?
I checked how many redis ConnectionPool was created: for this, I just changed the python redis module connection.py file in the __init__ method of the ConnectionPool class:
import sys print "ConnectionPool instance %s with %s max_connections" % (id(self), self.max_connections) sys.stdout.flush()
As you can see, there are many (8) pools, all of which have the same max_connections, 30, which is my setting in BROKER_TRANSPORT_OPTIONS :
ConnectionPool instance 4412957392 with 30 max_connections ConnectionPool instance 4412959888 with 30 max_connections ConnectionPool instance 4420369616 with 30 max_connections ConnectionPool instance 4420370320 with 30 max_connections ConnectionPool instance 4420367056 with 30 max_connections ConnectionPool instance 4420491792 with 30 max_connections ConnectionPool instance 4422318224 with 30 max_connections ConnectionPool instance 4422319504 with 30 max_connections
I don’t understand why there are so many of them. I want to control the number of connections to my redis. I can control the number of max_connections per ConnectionPool , but is it not useless if I do not control the number of ConnectionPool created?
Many thanks for your help!