Trying to better understand how Celery / Kombu and Redis interact under the hood to better scale the project and the cost of equipment (especially in development environments where I would like to have the smallest Redis setup and, therefore, the least number of connections).
I was unable to find the specific documentation that I am looking for on this topic in the Celery or Kombu user guide.
All of the following assumptions and questions are based on tests that I ran using the Celery application without any tasks (just app = Celery("DatabaseWorker", broker=redis_uri)) that then just sit there and do nothing, and using the redis-cli tool (using the monitor option for real-time update ) I deploy one worker of this type and get 8 open connections.
Looking for answers or corrections to my misguided assumptions:
Why do all 8 connections remain open? Is this because every connection comes from the pool and thus never closes?
If this is due to the pool, then why the settings in celeryconfig.py, such as the following, still lead to 8 connections (none of them work)
BROKER_TRANSPORT_OPTIONS = {
max_connections': 5,
}
BROKER_POOL_LIMIT = 5
, (2) celeryev ( ). ( PUBLISH) (PSUBSCRIBE). , --without-hearbeat ( CELERY_SEND_EVENTS config var), . ? , , , .
(4) , //, . . ? :
1449649220.026758 [0 [::1]:57605] "INFO"
1449649220.027633 [0 [::1]:57605] "MULTI"
1449649220.027655 [0 [::1]:57605] "LLEN" "celery@xxxxxx-MacBook-Pro.local.celery.pidbox"
1449649220.027665 [0 [::1]:57605] "LLEN" "celery@xxxxxx-MacBook-Pro.local.celery.pidbox\x06\x163"
1449649220.027674 [0 [::1]:57605] "LLEN" "celery@xxxxxx-MacBook-Pro.local.celery.pidbox\x06\x166"
1449649220.027681 [0 [::1]:57605] "LLEN" "celery@xxxxxx-MacBook-Pro.local.celery.pidbox\x06\x169"
1449649220.027691 [0 [::1]:57605] "EXEC"
1449649220.027983 [0 [::1]:57605] "SADD" "_kombu.binding.celery.pidbox" "\x06\x16\x06\x16celery@xxxxxx-MacBook-Pro.local.celery.pidbox"
, (1) . , pidbox, . , , ?
1449697220.549016 [0 [::1]:62992] "PUBLISH" "celery.pidbox" "{\"body\": \"eyJyZXBseV90byI6IHsicm91dGluZ19rZXkiOiAiMTUwYWZhYzEtZThmNy0zNDI2LWEwM2ItNWRhNGYzMzg3M2JhIiwgImV4Y2hhbmdlIjogInJlcGx5LmNlbGVyeS5waWRib3gifSwgInRpY2tldCI6ICJjNGUyNTVjMS05YzZjLTQxNzktOGM4Yi05NzRmOGVjYmE5ZDQiLCAiZGVzdGluYXRpb24iOiBudWxsLCAibWV0aG9kIjogImhlbGxvIiwgImFyZ3VtZW50cyI6IHsicmV2b2tlZCI6IHt9LCAiZnJvbV9ub2RlIjogImNlbGVyeUBFc3RldmFucy1NYWNCb29rLVByby5sb2NhbCJ9fQ==\", \"headers\": {\"expires\": 1449697221.548759, \"clock\": 1}, \"content-type\": \"application/json\", \"properties\": {\"body_encoding\": \"base64\", \"delivery_info\": {\"priority\": 0, \"routing_key\": \"\", \"exchange\": \"celery.pidbox\"}, \"delivery_mode\": 2, \"delivery_tag\": \"e8e4ad76-bb0a-4c83-8cca-d01e25f3633b\"}, \"content-encoding\": \"utf-8\"}"
, redis Kombu, , , . , / , , _kombu.binding.celery.pidbox, , ? ( , _kombu.binding.celery - , _kombu.binding.celeryev - )
1449649219.005095 [0 [::1]:57599] "SADD" "_kombu.binding.reply.celery.pidbox" "bc8319b5-c8d3-38b9-8848-da686bd088b7\x06\x16\x06\x16bc8319b5-c8d3-38b9-8848-da686bd088b7.reply.celery.pidbox"
1449649220.020213 [0 [::1]:57604] "SADD" "_kombu.binding.celeryev" "worker.#\x06\x16\x06\x16celeryev.4834be60-b102-4fd5-9fdc-617bb945c079"
1449649220.024899 [0 [::1]:57603] "SADD" "_kombu.binding.celery" "celery\x06\x16\x06\x16celery"
1449649220.027983 [0 [::1]:57605] "SADD" "_kombu.binding.celery.pidbox" "\x06\x16\x06\x16celery@xxxxxx-MacBook-Pro.local.celery.pidbox"
(1) , , , , . ? , , SADD. 3 ?
1449649224.677975 [0 [::1]:57601] "BRPOP" "celery" "celery\x06\x163" "celery\x06\x166" "celery\x06\x169" "1"
, , 8-, , .