Celery tasks lost after sending rabbitmq to broker

Question

I had a strange problem when some of my tasks get lost after sending to brokers. This happens in about 1 out of 10 tasks. I checked that there is no old celery worker consuming this task.

I used a database with backend and flower support to track missing tasks, but task_id returned after applying apply_async in the database or in the flower. His status always comes on hold.

Then I used celery signals to find out what was going on. I found that for missing tasks, only before_task_publish and after_task_publish signals are fired. Publish that there is no trace for this task.

These are my signals

@before_task_publish.connect def before_task_publish_handler(sender=None, headers=None, body=None, **kwargs): # information about task are located in headers for task messages # using the task protocol version 2. logger.info("BEFORE TASK SENT id:"+body['id']) @after_task_publish.connect def after_task_publish_handler(sender=None, headers=None, body=None, exchange=None, routing_key=None, **kwargs): # information about task are located in headers for task messages # using the task protocol version 2. logger.info("AFTER TASK SENT id:"+body['id']) @task_prerun.connect def task_prerun_handler(sender=None, task_id=None, task=None, **kwargs): logger.info("TASK PRERUN with TASK_ID:"+str(task_id)) 

This is what I found in the magazines.

 $ cat gunicorn-access.log | grep -i 103de274-00dc-4765-844f-d319e9e199c2 BEFORE TASK SENT id: '103de274-00dc-4765-844f-d319e9e199c2' AFTER TASK SENT id: '103de274-00dc-4765-844f-d319e9e199c2' 

I am not sure whether the task was ignored by the rabbit or for some reason they were silent.

Answer 1

From time to time, celery loses tasks before it is completed. If you do not want to lose them, you must set enable task_acks_late ( CELERY_ACKS_LATE in older versions).

In your celery settings, set

 task_acks_late = True 

This ensures that task messages are acknowledged after the task is completed .