Celery + RabbitMQ + Socket Error

Question

I use celery in Django with RabbitMQ as a broker on Heroku. My service is RabbitMQ CloudAMQP Tough on Heroku. If appropriate, we had some frequent memory leaks that I tried to plug in, but usually the service does not get worse when this happens.

When the site is heavily loaded (for example, today), I start to receive random errors, for example:

Couldn't log in: a socket error occurred 

The task is completely thrown out and is not registered anywhere. This is obviously a business critical issue. Celery settings below:

 BROKER_URL = os.getenv('CLOUDAMQP_URL', DEFAULT_AMQP) CELERY_TASK_SERIALIZER = 'pickle' CELERY_RESULT_SERIALIZER = 'json' CELERY_ACCEPT_CONTENT = ['pickle', 'json'] CELERY_ENABLE_UTC = True # CELERY_RESULT_BACKEND = 'djcelery.backends.database:DatabaseBackend'] CELERY_STORE_ERRORS_EVEN_IF_IGNORED = True CELERY_SEND_TASK_ERROR_EMAILS = True CELERY_RESULT_BACKEND = False CELERY_IMPORTS = ('business.admin', 'mainsite.views', 'utils.crons', 'mainsite.forms', ) BROKER_POOL_LIMIT = 5 # trying to clean up this memory leak CELERYD_MAX_TASKS_PER_CHILD = 5 CELERYD_TASK_TIME_LIMIT = 60*60 

I'm a little new to celery, so I gladly provide to keep track of any logs / etc, but I'm not even sure what you should provide at this point. Is there something obvious in my settings or environment, it looks like this could cause this problem in heavy trading?

Answer 1

A socket error may be due to the fact that RabbitMQ or Heroku kill the Linux Out-of-Memory Killer. When the server runs out of memory due to unused memory allocation for some processes, the Linux kernel tries to find the cause and kills the associated process. Using too much memory with RabbitMQ can lead to death. You can find out if Linux OOM killed a specific process using grep -i kill /var/log/messages*

Use the following links for more information and check out the Linux OOM configuration:

How to set up Linux Out-of-Memory Killer

Are you using supervisord?

Supervisord is a great daemon for starting and monitoring processes. Using this one, you can make sure that all lengthy processes, such as RabbitMQ, will always be up and running, even if the process is killed.

There are two possible causes of memory leaks:

• If the .DEBUG parameters are correct, this may lead to a memory leak. Make sure settings.DEBUG if your working configuration is set to False.

• You must use the results of the task if you plan to save them. If you do not consume them, you will run into memory leak problems. To solve the problem, you can change your settings using the following lines:

   # Just ignore the results, in case you're not consuming results. CELERY_IGNORE_RESULT = True CELERY_STORE_ERRORS_EVEN_IF_IGNORED = False