Asynchronous subprocess using flask-aiohttp

Question

I use the Flask web server to provide an interface for time-consuming computing. To improve performance, I want

• start computing as a new subprocess to be able to use multiple processor cores for multiple parallel computing.
• let the calculations be done asynchronously using asyncio

To call asyncio coroutines from Flask, I started using flask-aiohttp , which works great for simple delay tasks, as shown in the examples, however, I cannot call the asynchronous subprocess from inside Flask:

 #!/usr/bin/env python3 # coding: utf-8 from flask import Flask from flask.ext.aiohttp import AioHTTP, async import asyncio from asyncio.subprocess import PIPE CALC_SCRIPT = './calc' app = Flask(__name__) aio = AioHTTP(app) @app.route('/calc/<int:n>') @async def calc(n): print('calc({}) called'.format(n)) create = asyncio.create_subprocess_exec(CALC_SCRIPT, str(n), stdout=PIPE, stderr=PIPE) print('create... ', end='') process = yield from create print('process created. {!r}, type={}'.format(process, type(process))) yield from process.wait() print('process finished.') # yields (stdout, stderr) result = '\n'.join(ch.decode().rstrip() for ch in (yield from process.communicate()) if ch) return result if __name__ == '__main__': aio.run(app, debug=True) 

A process is created but never returned:

 GET http://127.0.0.1:5000/calc/5 calc(5) called creating... process created. <Process 5647>, type=<class 'asyncio.subprocess.Process'> 

What am I doing wrong?

Answer 1

Reason: asyncio subprocesses from subflows have limitations, see asyncio docs Subprocess and threads .

Details: using debug=True flag-aiohttp processing requests in the subflow started by Werkzeug run_with_reloader . Turn off debugging and your code is working correctly.

Alternatively, as per the docs above, Flask-aiohttp should add a call to asyncio.get_child_watcher() just before the run_with_reloader call. With this call, your code even works with debug=True .