All geek questions in one place

The number of processes associated with celery depends on the parameter - concurrency?

We launch Celery behind the Supervisor and start with

celeryd --events --loglevel=INFO --concurrency=2

This, however, creates a process graph that consists of three layers and contains up to 7 celeryd processes (Supervisor spawns one celeryd, which spawns several others that spawn processes again). Our machine has two processor cores.

Do all these processes work on tasks? Perhaps some of them are just worker pools? How is concurrency related to the number of processes actually spawned?

+6
source share
1 answer

You should not have 7 processes if --concurrency is 2.

Actual running processes:

  • Main consumer process

    Delegates work with a work pool

  • Worker pool (this is the number that --concurrency decides)

So these are 3 processes with a concurrency of two.

In addition, a very easy process has begun, which is used to clear semaphores if force_execv is enabled (which by default I use a different transport than redis or rabbitmq).

NOTE that in some cases process lists also include threads. a worker can start several threads if it uses transports other than rabbitmq / redis, including one broker thread, which always starts if CELERY_DISABLE_RATE_LIMITS not enabled.

+4
source

Source: https://habr.com/ru/post/921828/

More articles:


All Articles