What should I decorate with @ asyncio.coroutine for asynchronous operations?

Question

I want to run my code asynchronously. What should I decorate with @asyncio.coroutine and what should I call with yield from for asynchronous operations?

In my case, I have sample code without a decorator. (Simple view of IRC bot browser)

 import asyncio class ChatBot: def __init__(self, loop): conn = asyncio.open_connection(HOST, PORT, loop=loop) self.reader, self.writer = yield from conn def send(self, msg): self.writer.write(msg) def read(self): msg = yield from self.reader.readline() return msg def run(self): while True: msg = self.read() self.parse(msg) def parse(self, msg): if msg.startswith('PING'): self.some_work(msg) elif msg.startswith('ERROR'): self.some_error() else: self.server_log(msg) def some_work(self, msg): # some work. It can call asynchronous function like I/O or long operation. It can use self.send(). def some_error(self, msg): # some work. It can call asynchronous function like I/O or long operation. It can use self.send(). def server_log(self, msg): # some work. It can call asynchronous function like I/O or long operation. It can use self.send(). loop = asyncio.get_event_loop() bot = ChatBot(loop) loop.run_until_complete(???) loop.close() 

I think ??? bot.run() and ChatBot.run should be decorated with @asyncio.coroutine . Then, what about other methods? I can’t understand when the @asyncio.coroutine decorator and call method are used with yield from or asyncio.async . (I already read PEP-3156 for understanding asnycio. But I cannot fully understand.)

Answer 1

When to use the @asyncio.coroutine decorator

If you have a function that should use yield from to call a coroutine, you should decorate it with asyncio.coroutine . Also note that coroutines are often (not always) viral. As soon as you add the yield from function to the function, it will become a coroutine, and in addition, any function that calls this coroutine usually (though not always) must also have a coroutine.

When to use asyncio.async

Why coroutines are not always viral? Because you do not always need to use yield from to call a coroutine. You need to use yield from if you want to call coroutine coroutine and wait for it to complete. If you just want to run the coroutine in the background, you can simply do this:

 asyncio.async(coroutine()) 

This will start coroutine to start as soon as the control returns to the event loop; he will not wait for the coroutine to finish until the next line. A regular function can use this to schedule the execution of a coroutine, without also requiring it to become a coroutine itself.

You can also use this approach to run multiple coroutines at the same time. So, imagine that you have these two coroutines:

 @asyncio.coroutine def coro1(): yield from asyncio.sleep(1) print("coro1") @asyncio.coroutine def coro2(): yield from asyncio.sleep(2) print("coro2") 

If you have this:

 @asyncio.coroutine def main(): yield from coro1() yield from coro2() yield from asyncio.sleep(5) asyncio.get_event_loop().run_until_complete(main()) 

After 1 second, "coro1" will be printed. Then, after two seconds (so only three seconds), "coro2" will be printed, and after five seconds the program will exit, executing 8 seconds of the total execution time. Alternatively, if you used asyncio.async :

 @asyncio.coroutine def main(): asyncio.async(coro1()) asyncio.async(coro2()) yield from asyncio.sleep(5) asyncio.get_event_loop().run_until_complete(main()) 

This will print "coro1" after one second, "coro2" after one second, and the program will exit after 3 seconds, for a total of 5 seconds of execution time.

How does this affect your code?

So, following these rules, your code should look like this:

 import asyncio class ChatBot: def __init__(self, reader, writer): # __init__ shouldn't be a coroutine, otherwise you won't be able # to instantiate ChatBot properly. So I've removed the code that # used yield from, and moved it outside of __init__. #conn = asyncio.open_connection(HOST, PORT, loop=loop) #self.reader, self.writer = yield from conn self.reader, self.writer = reader, writer def send(self, msg): # writer.write is not a coroutine, so you # don't use 'yield from', and send itself doesn't # need to be a coroutine. self.writer.write(msg) @asyncio.coroutine def read(self): msg = yield from self.reader.readline() return msg @asyncio.coroutine def run(self): while True: msg = yield from self.read() yield from self.parse(msg) @asyncio.coroutine def parse(self, msg): if msg.startswith('PING'): yield from self.some_work(msg) elif msg.startswith('ERROR'): yield from self.some_error() else: yield from self.server_log(msg) @asyncio.coroutine def some_work(self, msg): # some work. It can call asynchronous function like I/O or long operation. It can use self.send(). @asyncio.coroutine def some_error(self, msg): # some work. It can call asynchronous function like I/O or long operation. It can use self.send(). @asyncio.coroutine def server_log(self, msg): # some work. It can call asynchronous function like I/O or long operation. It can use self.send() @asyncio.coroutine def main(host, port): reader, writer = yield from asyncio.open_connection(HOST, PORT, loop=loop) bot = ChatBot(reader, writer) yield from bot.run() loop = asyncio.get_event_loop() loop.run_until_complete(main()) loop.close() 

Another thing to keep in mind is that adding yield from in front of a function does not magically make this call non-blocking. Does not add the @asyncio.coroutine decorator. Functions are only non-blocking if they directly or indirectly call their own asyncio coroutines that use non-blocking I / O and integrate with the asyncio event asyncio . For example, you mentioned REST API calls. To prevent these REST API calls from blocking the event loop, you need to use the aiohttp library or asyncio.open_connection . Using something like requests or urllib blocks the loop because they are not integrated with `asyncio.

Answer 2

You have to decorate everything that yield from uses, because the asyncio.coroutine decorator will take your function as a generator and do all the callback / async work during the damage.

In your case, run needs to be rewritten as follows:

 @asyncio.coroutine def run(self): while True: msg = yield from self.read() yield from self.parse(msg) 

Then read and parse must also be co-programmed. You should read about asynchronous operation before using it, this will help you a lot.

Answer 3

 @asyncio.coroutine def read(self): msg = yield from self.reader.readline() return msg @asyncio.coroutine def run(loop): while True: msg = yield from read() yield from parse(msg) loop = asyncio.get_event_loop() loop.run_until_complete(run(loop)) loop.close()