Currently, I use Mechanize and the get () method to get each site, and I check with the content () method every main page. I have a very fast connection to my computer + 10 Mbps, and yet it took 9 hours to check 11K sites, which is unacceptable, the problem is that the speed of the get () function, which obviously should get the page, is there any way to do it's faster, perhaps disabling something, since I only need to check the html main page.
Thank,
source
share