Long-term tasks with an asynchronous server

Question

I think everyone knows what to do with long-standing challenges in django: use celery and relax. But what if I want to get the benefits of websockets with aiohttp (or tornado)?

Let's say I have a very difficult task with a processor, which can take from several seconds to several (5-10) minutes. It seems pretty good to handle this task in the websocket loop and notify the user of progress. No ajax requests, very quick answer for short tasks.

async def websocket_handler(request): ws = web.WebSocketResponse() await ws.prepare(request) async for msg in ws: if msg.tp == aiohttp.MsgType.text: answer_to_the_ultimate_question_of_life_the_universe_and_everything =\ long_running_task(msg.data, NotificationHelper(ws)) ws.send_str(json.dumps({ 'action': 'got-answer', 'data': answer_to_the_ultimate_question_of_life_the_universe_and_everything, })) return ws 

But, on the other hand, the task associated with the CPU serviced in this way blocks the entire thread, as I understand it. If I have 10 employees and 11 clients who want to use the application, the 11th client will not be served until the first client task is completed.

Perhaps I should run tasks that look big in celery and tasks that look small in the main loop?

So my question is: is there a good design template for serving long-running tasks with an asynchronous server?

Thanks!

Answer 1

Just start your long-term task with a processor- loop.run_in_executor() and send notifications about the work done loop.call_soon_threadsafe() .

If your work is not CPU related, but IO (for example, sends emails), you can create a new task by calling loop.create_task() . This is like spawning a new topic.

If you cannot use the fire-and-forget approach, you need to use a persistent message broker such as RabbitMQ (there is a https://github.com/benjamin-hodgson/asynqp library for communicating with the rabbit in asynchronous mode).