Improving asyncio's “Sending 100,000 Requests” Speed ​​Using Multiprocessor and Multithreaded

Question

First of all, I want to send several requests as quickly as possible using 1 connection. The code below works fine and fast, but I want it to go beyond asynchronous. Back to my question, is it possible to run this in parallel using multi-threaded or multi-processor processing. I heard that you can use ThreadPoolExecutor or ProcessPoolExecutor.

import random
import asyncio
from aiohttp import ClientSession
import time
from concurrent.futures import ProcessPoolExecutor

async def fetch(sem,url, session):
    async with sem:
        async with session.get(url) as response:
            return await response.read()
async def run(r):
    url = "http://www.example.com/"
    tasks = []
    sem = asyncio.Semaphore(1000)
    async with ClientSession() as session:
        for i in range(r):
            task = asyncio.ensure_future(fetch(sem, url.format(i), session)) #return a task
            tasks.append(task)
        responses = asyncio.gather(*tasks)
        await responses
if __name__ == "__main__":
    number = 10000
    loop = asyncio.get_event_loop()
    start = time.time()
    loop.run_until_complete(run(number))
    end = time.time() - start
    print (end)

from testing, he managed to send about 10 thousand requests in 49 seconds. I need it to be faster, any suggestion? (thread, process)

Answer 1

ProcessPoolExecutor - . , . , 4 ProcessPoolExecutor (max_workers = 4)

asyncio , - :

def main(n):
    loop = asyncio.get_event_loop()
    loop.run_until_complete(run(n))


with concurrent.futures.ProcessPoolExecutor(max_workers=4) as exc:
    exc.submit(main, 2500)
    exc.submit(main, 2500)
    exc.submit(main, 2500)
    exc.submit(main, 2500)

run: ensure_future Tasks, async def , asyncio.gather

async def run(r):
    url = "http://www.example.com/"
    sem = asyncio.Semaphore(1000)
    async with ClientSession() as session:
        coros = [fetch(sem, url.format(i), session) for i in range(r)]
        await asyncio.gather(*coros)