Can celery problems survive?

Question

I need to create a system that processes two types of tasks. One type can create more tasks of itself or another type. There will be very few workers (2-3) and only one owner. The most important requirement is that the system must correctly handle the reboots: for example, when restarting, tasks that were supposed to be executed should start from scratch, and workers should perform tasks that were queued before the reboot. If you look at celery, it is suitable for this use case. However, I have a few questions:

1) Does celery have the ability to handle reboots of the entire system as described? (Celery + workers + broker, everything restarts, complete power failure).

2) What is the best broker to use in this scenario? I would prefer SQLAlchemy + SQLite, since it is more "self-preserving", but if RabbitMQ is the right choice, given the requirement to properly handle full reloads, I can go with it.

Answer 1

1) Does celery have the ability to handle reboots of the entire system as described? (Celery + workers + broker, everything restarts, complete power failure).

Yes, when you create your own queue on RabbitMQ ( Persistent Queues ), even after rebooting your server, the task will resume and will be carried out by celery.

2) What is the best broker to use in this scenario? I would prefer SQLAlchemy + SQLite, since it is more "self-preserving", but if RabbitMQ is the right choice, given the requirement to properly handle full reloads, I can go with it.

Use RabbitMQ. We use RabbitMQ + Celery in production. So I would suggest you use RabbitMQ as a broker.

Best Practice Celery + RabbitMQ

Message Reliability

Notes:

• Use more queues (i.e. not just the default)
• Use priority workers
• Use Celery error handling mechanisms.