Why is celery.control.inspect reporting fewer queued tasks than rabbitmqctl?

Question

rabbitmqctl correctly reports thousands of tasks in the queue:

 $ sudo rabbitmqctl -q list_queues name messages messages_ready messages_unacknowledged default 13142 13126 16 

However, celery reports:

 >>> len(app.control.inspect().active()[' celery@default ']) 4 >>> len(app.control.inspect().scheduled()[' celery@default ']) 1 >>> len(app.control.inspect().reserved()[' celery@default ']) 16 >>> len(app.control.inspect().revoked()[' celery@default ']) 0 

The correct number (thousands) of tasks seems to display in app.control.inspect().stats()[' celery@default ']['total'] , but I really want to know the correct number of outstanding tasks in the queue from python, and active() et al only seem to report up to 16 or so - maybe there is a limit?

With the exception of using privileged subprocess calls on rabbitmqctl , how can I get the full score of the task from python, preferably through celery (by the way, this server is using Celery 3.1.8 now)

Answer 1

Celery app.control.inspect will check tasks that are processed only by running workers .

Even if you have thousands of tasks in line, your employee will only complete a few tasks at any given time. This is an active task.

In addition to this, the employee can pre-select some tasks that will be reserved for this employee. They will be shown in reserved tasks.

If you installed ETA for your tasks or there were periodic tasks, they will fall under scheduled tasks.

It looks like you started a worker with concurrency of 4 (or a worker with default settings on a 4-core computer). Thus, there are 4 active tasks. Each workflow predefined 4 tasks, as a result of which 16 reserved tasks were completed.

AFAIK, there is no way to get the total number of tasks in the queue with celery.

However, there are several python solutions to get the total number of messages in a queue. You can check my other answer here for other ways to do this.

Update:

pika is a python client for interacting with rabbitmq. You can use it to consume messages. The following is a simple example to consume each message. You can check more usage examples at pika docs.