Celery task time measurement

Question

I reworked the standalone batch job to use celery to send work in progress. I am using RabbitMQ. Everything runs on the same machine, and no other processes use an instance of RabbitMQ. My script just creates a bunch of tasks that are handled by workers.

Is there an easy way to measure the time from the start of my script until all tasks are complete? I know that this is a bit complicated with the design when using message queues. But I do not want to do this in production, just for testing and evaluating performance.

Answer 1

You can use chord by adding a fake task at the end, which will be passed the time at which the tasks were sent, and this will return the difference between the current time and the time that will be executed during execution.

import celery import datetime from celery import chord @celery.task def dummy_task(res=None, start_time=None): print datetime.datetime.now() - start_time def send_my_task(): chord(my_task.s(), dummy_task.s(start_time=datetime.datetime.now()).delay() 

send_my_task sends the task you want to profile, along with dummy_task , which will print how long it took (more or less). If you need more accurate numbers, I suggest passing start_time directly to your tasks and using signals .

Answer 2

You can use celery signals , registered functions will be called before and after the task is completed, measure elapsed time trimerically:

 from time import time from celery.signals import task_prerun, task_postrun d = {} @task_prerun.connect def task_prerun_handler(signal, sender, task_id, task, args, kwargs): d[task_id] = time() @task_postrun.connect def task_postrun_handler(signal, sender, task_id, task, args, kwargs, retval, state): try: cost = time() - d.pop(task_id) except KeyError: cost = -1 print task.__name__, cost