Yes, urllib does not do much concurrency. Each time you urlopen , it should establish a connection, send an HTTP request and get the status code and headers from the response (and possibly handle the redirect from there). Therefore, although you can read the response text at your own pace, most of the request time-out will already be reached.
If you need more concurrency, you probably have to pick up some kind of asynchronous I / O tool on the network (for example, Eventlet seems to have a suitable example on the first page) or just run each urlopen in its own stream.
source share