Run chord callback even if the main tasks are not performed

Question

Is it possible to execute a chord callback even if the main tasks failed?

I created a chord in which I added a bunch of tasks and registered a callback. My problem is that if one of the tasks fails, the callback does not start, but I would like the callback to start anyway.

I tried registering a callback with si () ( immutability )

callback = tasks.run_delete_rule.si([timestamp]) header = [tasks.run_update_rule.s(i, timestamp) for i in item_ids] result = chord(header)(callback) 

I also tried adding the ignore_result=True parameter to both task decorators, but was unsuccessful.

Answer 1

From github problem # 1881 , if the callback has a link_error parameter that accepts a list of task names, then when the chord task is not running, link_error tasks will be executed.

 @task(name='super_task.good') def good(): return True @task(name='super_task.raise_exception') def raise_exception(): raise ValueError('error') @task(name='super_task.callback') def callback(*args, **kwargs): logger.info('callback') logger.info(args) logger.info(kwargs) return 'finished' @task(name='super_task.error_callback') def error_callback(*args, **kwargs): logger.info('error_callback') logger.info(args) logger.info(kwargs) return 'error' >>> c = chord( [raise_exception.s(), good.s(), raise_exception.s()], callback.s().set(link_error=['super_task.error_callback']) ) >>> result = c() 

This will lead to a chord and in your celery log, you will see that the raise_exception task will fail, and the error_callback that will be received in it will call task_id from callback .

At this point, the result value will contain an AsyncResult callback instance, but because in the chord the errors propagate to the callback, making result.get() will result in the exception of tasks and result.traceback gives you a trace.

If you want to have one callback, just pass the chord callback name to link_error

 callback.s().set(link_error='super_task.callback') 

NOTE

Another option is to set CELERY_CHORD_PROPAGATES = False , which will return to atrial 3.1 behavior and will always call back.

But this is not recommended because, as you can find in github issue # 1349

Celery 3.1 determines how chordal errors are handled, previous behavior has never been documented and more of an accident, since it was never intended to work that way.

We could not change the behavior in the patch release, so instead we had to set a parameter, but there was never an intention for someone to consciously disable the new behavior.

New behavior exists to protect against this kind of problem, and remote compatibility can be removed. I suggest you find another way to handle errors here (and I would not mind the suggestion if you can come up with a good api for it)

Answer 2

You just need to change the link_error way. Instead of referencing a string, pass the signature with the required arguments.

In the above example, you can pass an argument as follows

 c = chord( [raise_exception.s(), good.s(), raise_exception.s()], callback.s().set(link_error=[error_callback.s(<arguments_here>)]) )