Gathering results from python coroutines before loop completion

Question

I have a python bot spot built using discord.py, which means the whole program runs inside an event loop.

The feature I'm working on includes creating several hundred HTTP requests and adding the results to the final list. This takes about two minutes, so I use aiohttp to make them asynchronous. The related parts of my code are identical to the quickstart example in aiohttp docs, but it throws a RuntimeError: Session closes. The methodology was taken from an example in https://pawelmhm.imtqy.com/asyncio/python/aiohttp/2016/04/22/asyncio-aiohttp.html in the section "Get multiple URLs".

async def searchPostList(postUrls, searchString) futures = [] async with aiohttp.ClientSession() as session: for url in postUrls: task = asyncio.ensure_future(searchPost(url,searchString,session)) futures.append(task) return await asyncio.gather(*futures) async def searchPost(url,searchString,session)): async with session.get(url) as response: page = await response.text() #Assorted verification and parsing Return data 

I do not know why this error occurs because my code is so similar to two supposedly functional examples. The event loop itself is working fine. It works forever, as it is a bot application.

Answer 1

In the example you linked, the collection of results was in the async with block. If you do this outside, there is no guarantee that the session will not be closed before even requests are made!

Moving the return statement inside the block should work:

 async with aiohttp.ClientSession() as session: for url in postUrls: task = asyncio.ensure_future(searchPost(url,searchString,session)) futures.append(task) return await asyncio.gather(*futures)