Challenge the task of celery hanging in Heroku's environment

Question

I have a Django application on Heroku that calls a task using the Selery delay method, which should pass additional processing to an employee. But when I make an HTTP request for the appropriate view, the Heroku web dynamometer freezes and ultimately causes a timeout request. Here's the test task (the application is called waittimes):

@task def test_tasks(message, name='waittimes.tasks.test_tasks'): print message 

And the test view:

 class TaskTest(View): def get(self, request): print "about to call the task" test_tasks.delay("the task was successful!") return HttpResponse("view was successful") 

If I make an http request for this view, I expect that "the task was successful" to get output to the console, and the answer, which says that "the view was successful." This happens successfully when I make a request to the development server on my computer. It also works if I run the django shell in my Heroku application environment and use the django test client to execute the request.

 app[celeryd.1]: [2013-06-26 23:57:48,018: INFO/MainProcess] Got task from broker: waittimes.tasks.test_tasks[67036069-b49e-45ba-aef4-3c64d7161a67] app[celeryd.1]: [2013-06-26 23:57:48,133: WARNING/PoolWorker-3] the task was successful! app[celeryd.1]: [2013-06-26 23:57:48,200: INFO/MainProcess] Task waittimes.tasks.test_tasks[67036069-b49e-45ba-aef4-3c64d7161a67] succeeded in 0.09690284729s: None 

But when I make a request directly to the Heroku URL, the request freezes and I end up with a terrible H12 timeout error from Heroku.

 heroku[router]: at=error code=H12 desc="Request timeout" method=GET path=/task/test/ dyno=web.1 connect=2ms service=30000ms status=503 bytes=0 

I know that calling a task causes a problem because the "calling a task call" is printed on the console. Thus, the problem is that the system has problems solving the "delay" method (and apply_async). It just freezes and does not return an asynchronous object. And this only happens when the code runs in the web dyno process.

So far, these are my findings:

1) The task is properly registered and my Redis browser works, because everything works when I call the view using the test client from the shell (however, this is done in a separate shell mode on Heroku and not in web dyno, which usually receives requests)

2) The system correctly routes and sends a handler for the request, as it is "about to invoke a task" printed. This does not seem to be a problem with the Heroku router.

3) The problem is not related to a specific view, because even a disabled test case like this does not work

In addition to a direct solution, any tips for further debugging are also welcome.

Answer 1

Okay, this may not be a direct answer, but given the age of this question and how long it has been ignored, I will continue and provide my understanding to everyone who is unlucky to face this problem, like me.

This particular problem seems poorly documented, and it is quite difficult to find, and I just ran into it when I set up a third-party project on Heroku.

Heroku seems to be endemic when some calls to Python functions behave differently on the platform, rather than locally (or with any normal Python deployment).

In my case, the problem was that my time.sleep() task was to call the Python function time.sleep() .

As a test case, I used time.sleep(1) to simply demonstrate in the logs that the task was actually running asynchronously. I have successfully completed this test on a normal infrastructure (including virtual machines) many times.

When I transferred this test to Heroku, I encountered the same problem as Gentro. The logs clearly indicated that Celery and my broker initialized normally and knew about my application, however, when I switched to the call through the Django view, my web dynamo mysteriously interrupted the timeout from H12 as the only log message.

When I commented out the sleep call, everything worked perfectly fine.

TL DR - check your call stack leading to your celery task, make sure that you don't go into any functions that may cause Heroku dyno to crash, like sleep()

I'm not saying that this is exactly what caused Asker’s initial problem, but if you see this behavior, this is absolutely one of the potential causes.