Urllib and concurrency - Python

I am serving a python script through WSGI . The script accesses the web resource through urllib , computes the resource, and then returns the value.

The problem is that urllib does not seem to handle many concurrent requests for the exact URL .

As soon as requests will be processed to 30 simultaneous requests, requests will be slow before scanning .: (


Help would be greatly appreciated !: D

+4
source share
1 answer

Yes, urllib does not do much concurrency. Each time you urlopen , it should establish a connection, send an HTTP request and get the status code and headers from the response (and possibly handle the redirect from there). Therefore, although you can read the response text at your own pace, most of the request time-out will already be reached.

If you need more concurrency, you probably have to pick up some kind of asynchronous I / O tool on the network (for example, Eventlet seems to have a suitable example on the first page) or just run each urlopen in its own stream.

+3
source

Source: https://habr.com/ru/post/1307762/


All Articles