Fork processes for every task in celery

Question

I am currently using the C extension library for Python, but it seems to have memory leaks. The tasks that are done on my celeryd do something using this C extension library, and celeryd eats a lot of memory in about an hour. I cannot fix this C extension library for many reasons, but instead I want to develop processes for each task in Celery. Are there any options for celery?

Answer 1

You can use the CELERYD_MAX_TASKS_PER_CHILD option or --maxtasksperchild celeryd.

To restart workflows after each task:

CELERYD_MAX_TASKS_PER_CHILD=1

https://celery.readthedocs.org/en/latest/userguide/workers.html#max-tasks-per-child-setting