Closing a Python Requests connection from another thread

Question

To close the application as soon as possible, can I interrupt the request.post call from another thread and immediately disconnect it?

I played with adapters, but so far no luck:

for ad in self.client.session.adapters.values(): ad.close() 

Answer 1

The right way to do this is to use messaging in another thread. We can make a version for the poor using a common global variable. As an example, you can try running this script:

 #!/usr/bin/env python # A test script to verify that you can abort streaming downloads of large # files. import threading import time import requests stop_download = False def download(url): r = requests.get(url, stream=True) data = '' content_gen = r.iter_content() while (stop_download == False): try: data = r.iter_content(1024) except StopIteration: break if (stop_download == True): print 'Killed from other thread!' r.close() if __name__ == '__main__': t = threading.Thread(target=download, args=('http://ftp.freebsd.org/pub/FreeBSD/ISO-IMAGES-amd64/9.1/FreeBSD-9.1-RELEASE-amd64-dvd1.iso',) ).start() time.sleep(5) stop_download = True time.sleep(5) # Just to make sure you believe that the message actually stopped the other thread. 

When you do this during production, especially if you do not have GIL protection, you will need to be very careful about the messaging state in order to avoid uncomfortable multithreading errors. I leave this to the developer.

Answer 2

I found a way, here's how to disconnect

 def close(): time.sleep(5) r.raw._fp.close() t = threading.Thread(target=close).start() print "getting" s = requests.Session() r = s.get("http://download.thinkbroadband.com/1GB.zip", stream = True) for line in r.iter_content(1024): log.debug("got it: %s", len(line)) print "done" 

However, this is a hack, and I do not like it, private members may change in the future, I return to urllib2

Answer 3

So, if you do the following from the interactive shell, you will see that closing the adapters is not what you are looking for.

 import requests s = requests.session() s.close() s.get('http://httpbin.org/get') <Response [200]> for _, adapter in s.adapters.items(): adapter.close() s.get('http://httpbin.org/get') <Response [200]> s.get('https://httpbin.org/get') <Response [200]> 

It seems that this may be a mistake in the requests, but in general, closing the adapter should prevent you from completing additional requests, but I'm not quite sure that it will interrupt the current requests.

Looking at the HTTPAdapter (which supports both standard and standard adapters 'http://' and 'https://' ), the call that calls it will call clear in the urrllib3 base pool manager. From the documentation of this urllib3 method, you see that:

 This will not affect in-flight connections, but they will not be re-used after completion. 

In fact, you see that you cannot affect a connection that is not yet completed.