All geek questions in one place

Using Celery to Create Microservices

I am going to break the project into small microservices.

All microservices are based on cron. I mean celery as a distribution of tasks, as well as a mechanism for performing periodic tasks (celerybeat).

I do not want to create several celery applications on the microservice, as this will increase the overhead of the fact that several brokers and several flower systems will be used for monitoring.

I tried with one application on multiple servers, but failed. My needs with celery:

  • I need to have independent servers for each microservice
  • A task related to a specific microservice should be performed only on its servers; lack of task sharing among other servers.
  • If the microservice does not work, I do not want celerybeat to hammer the broker with thousands of unresolved tasks, which caused the service to stop working in other microservices.
  • There is no need to communicate between microservices.

I tried dividing the queues into one employee, which seems impossible. I tried one worker on the server, but I need more than one worker on microservices.

+6
source share
1 answer

For your use case, simple routing based on the queue from one broker is enough.

Keep only one broker working on any server or on a separate server.

Now, when completing tasks, add them to separate queues.

From microservice 1:

In [2]: add.apply_async(args=(12, 1), queue='queue1')
Out[2]: <AsyncResult: 2fa5ca61-47bc-4c2c-be04-e44cbce7680a>

,

celery worker -A tasks -l info -Q queue1

2:

In [2]: sub.apply_async(args=(12, 1), queue='queue2')
Out[3]: <AsyncResult: 4d42861c-737e-4b73-bfa8-6d1e86241d57>

,

celery worker -A tasks -l info -Q queue2

, .

+4

Source: https://habr.com/ru/post/1016402/

More articles:


All Articles