All geek questions in one place

Huge Redis memory usage when using celery as a broker due to pidboxes answers

We work with celery 3.1.8 with Redis as a brokerage broker. Our configuration is quite simple, and we do not need results, for this we install

CELERY_IGNORE_RESULT = True

However, we noticed that our copy of the redis browser takes up a huge amount of memory (~ 3 GB), while the actual number of queues and messages in them is quite small at any time. After examining the RDB file, we noticed that there are several huge lists:

 database,type,key,size_in_bytes,encoding,num_elements,len_largest_element 0,list,"ed2d26b5-b8fb-3478-ace7-6714e8a7b4ed.reply.celery.pidbox",70733024,linkedlist,4,17696745 0,list,"e3eda502-27eb-348d-a86e-1d78fc31b165.reply.celery.pidbox",35350876,linkedlist,2,17686857 0,list,"6421b8c1-bbc0-3a59-a7ee-f26450552a60.reply.celery.pidbox",35443327,linkedlist,2,17762611 0,list,"8439dcd0-921e-3922-8504-9057b6c9834a.reply.celery.pidbox",106088780,linkedlist,6,17696745 0,list,"85c667c8-63b6-338f-b00a-e1f2cd4da143.reply.celery.pidbox",17762845,linkedlist,1,17762611 0,list,"05d0c0f2-9530-37f3-a9cb-189fc237303c.reply.celery.pidbox",106088769,linkedlist,6,17696743 0,list,"d3f200fd-c81e-3d6d-acf3-d0e9021e7e5c.reply.celery.pidbox",35431615,linkedlist,2,17762611 0,list,"7b4291c7-b916-3806-910b-c250c9a7fece.reply.celery.pidbox",88401866,linkedlist,5,17696745 0,list,"0c8b64c1-7efe-3070-b2e3-980f395b84e8.reply.celery.pidbox",123752294,linkedlist,7,17696743 0,list,"e5cf288b-8ced-3f6c-891e-34e2d302c89c.reply.celery.pidbox",70711492,linkedlist,4,17691717 0,list,"a9cafe29-204d-3d97-9b7b-322a847d0789.reply.celery.pidbox",53121613,linkedlist,3,17762611 0,list,"1c1f90ca-1fe1-35e4-a144-3a97177a674b.reply.celery.pidbox",35431683,linkedlist,2,17762611

These lists contain several json elements, each of which contains a huge body.

Any indication of why these response pidboxes are present, although we found that celery ignored the result? Do they need celery if we have no interest in the results of the tasks? We would like to get rid of them, since they are turning an instance that must accept ~ 10 mb of memory to receive ~ 3 gb of memory.

Any help would be great, let me know if you need more configuration information.

+6
source share
1 answer

I think the setting you are looking for should get rid of these pidbox messages CELERY_SEND_EVENTS . It is enabled by default for tools like celerymonmon. However, for me, setting CELERY_SEND_EVENTS = False does not change anything. You could give him a chance, maybe this will work for you.

http://celery.readthedocs.org/en/latest/configuration.html#celery-send-events

0
source

Source: https://habr.com/ru/post/974033/

More articles:


All Articles