Python MultiThreading with Urllib2 Problem

Question

I can download several files quite quickly with many streams at once, but the problem is that after a few minutes it gradually slows down to almost a complete stop, I have no idea why. There is nothing wrong with my code that I can see, and my RAM / CPU is fine. The only thing I can think of is that urllib2 does not handle a huge number of connections correctly. If this helps, I use proxies, but I had a problem without them. Does anyone have any suggestions or ideas on this? Thanks!

Answer 1

Can you confirm that doing the same number of concurrent downloads without python continues to load quickly? Perhaps the problem is not with your code, but with the fact that your connection receives throttling or a site serving files.

If this is not a problem, you can try the pyprocessing library to implement a multi-processor version instead of a multi-threaded version. If you are using python 2.6, pyprocessing is included in the distribution as multiprocessing . It is very easy to convert multi-threaded code to multi-process code, so it’s worth a try, if only to confirm the problem using streaming.

Answer 2

, , . , .

, .

, - , , , , , .

, , httplib2 urllib2.