What is the best way to upload a file using urllib3

Question

I want to upload a file via HTTP using urllib3 . I managed to do this using the following code:

  url = 'http://url_to_a_file' connection_pool = urllib3.PoolManager() resp = connection_pool.request('GET',url ) f = open(filename, 'wb') f.write(resp.data) f.close() resp.release_conn() 

But I was wondering what the right way to do this is. For example, it will work well for large files, and if not, what to do to make this code more error resistant and scalable.

Note. It is important for me to use the urllib3 not urllib2 , for example, because I want my code to be thread safe.

Answer 1

Your code snippet is close. Two things to note:

• If you use resp.data , it will use the whole response and return the connection (you do not need resp.release_conn() manually). This is normal if you are cool at holding data in memory.

• You can use resp.read(amt) , which will transmit the response, but the connection should be returned via resp.release_conn() .

It would look like ...

 import urllib3 http = urllib3.PoolManager() r = http.request('GET', url, preload_content=False) with open(path, 'wb') as out: while True: data = r.read(chunk_size) if not data: break out.write(data) r.release_conn() 

In this case, the documentation may be slightly missing. If anyone is interested in creating a pull-request to improve urllib3 documentation , that would be greatly appreciated. :)

Answer 2

The most correct way to do this is probably to get a file-like object representing the HTTP response and copy it to the real file using shutil.copyfileobj, as shown below:

 url = 'http://url_to_a_file' c = urllib3.PoolManager() with c.request('GET',url, preload_content=False) as resp, open(filename, 'wb') as out_file: shutil.copyfileobj(resp, out_file) resp.release_conn() # not 100% sure this is required though 

Answer 3

add the named variable preload_content else, you will end up loading the full content

 http.request('GET', url, preload_content=False)