How to repeat celery task for a hard timeout?

Question

I have a celery task:

@celery.task(name='tasks.ht_run', default_retry_delay=30, max_retries=15, time_limit=1) def ht_run(str_command): try: f = os.popen(str_command) output = f.read() if output == '': raise Exception except Exception as exc: raise ht_run.retry(exc=exc) return output.split('\n') 

And it's called like that: appserver.ht_run.delay (string)

While I expect it to try again when there is a timeout, instead, if it just fails. In the Celery window, I get the following error:

 [2014-12-10 11:50:22,128: ERROR/MainProcess] Task tasks.ht_run[6a83793a-8bd6-47fc-bf74-0b673bf961f2] raised unexpected: TimeLimitExceeded(1,) Traceback (most recent call last): File "/Users/andy.terhune/cgenv/lib/python2.6/site-packages/billiard/pool.py", line 639, in on_hard_timeout raise TimeLimitExceeded(job._timeout) TimeLimitExceeded: TimeLimitExceeded(1,) [2014-12-10 11:50:22,128: ERROR/MainProcess] Hard time limit (1s) exceeded for tasks.ht_run[6a83793a-8bd6-47fc-bf74-0b673bf961f2] [2014-12-10 11:50:23,644: ERROR/MainProcess] Process 'Worker-28' pid:2081 exited with 'signal 9 (SIGKILL)' 

How can I get this for a timeout and try again when this happens?

Answer 1

Thanks user2097159,

tried a soft timeout and worked like a charm,

 @celery.task( name='tasks.ht_run', default_retry_delay=30, max_retries=15, soft_time_limit=1) def ht_run(str_command): try: f = os.popen(str_command) output = f.read() if output == '': raise Exception except Exception as exc: raise ht_run.retry(exc=exc) return output.split('\n') 

...

 [2014-12-10 12:16:55,580: WARNING/MainProcess] Soft time limit (1s) exceeded for tasks.ht_run[f31bbd15-e755-440c-9261-cd0864ceb3a9] [2014-12-10 12:16:55,603: INFO/MainProcess] Received task: tasks.ht_run[f31bbd15-e755-440c-9261-cd0864ceb3a9] eta:[2014-12-10 18:17:25.593613+00:00] [2014-12-10 12:16:55,603: DEBUG/MainProcess] basic.qos: prefetch_count->121 [2014-12-10 12:16:56,897: INFO/MainProcess] Task tasks.ht_run[f31bbd15-e755-440c-9261-cd0864ceb3a9] retry: Retry in 30s: SoftTimeLimitExceeded() 

Answer 2

Now in version 4.0 you can specify autoretry for specific exceptions in the decorator. Here is an example from the documentation:

 from twitter.exceptions import FailWhaleError @app.task(autoretry_for=(FailWhaleError,)) def refresh_timeline(user): return twitter.refresh_timeline(user) 

One of these exceptions may be SoftTimeLimitExceeded to suit your case.