Python timeout detection urllib2 urlopen

Question

I'm still relatively new to Python, so if this is an obvious question, I'm sorry.

My question is about urllib2 library and urlopen function. Currently, I use this to load a large number of pages from another server (they are all on the same remote host), but the script is killed by a timeout error from time to time (I assume this is from large requests).

Is there a way to keep the script running after a timeout? I would like to get all the pages, so I want the script to keep trying until it gets the page, and then goes over.

On the other hand, will the connection be open to server help?

Answer 1

. . , urllib2.HTTPError. , try...except. :

import urllib2
import time

for url in urls:
    while True:
        try:
            sock=urllib2.urlopen(url)
        except (urllib2.HTTPError, urllib2.URLError) as err:
            # You may want to count how many times you reach here and
            # do something smarter if you fail too many times.
            # If a site is down, pestering it every 10 seconds may not
            # be very fruitful or polite.
            time.sleep(10)
        else:              
            # Success  
            contents=sock.read()
            # process contents
            break                # break out of the while loop

Answer 2

urllib2