Is this a httplib2 error

Question

My python program uses httplib2.Http to make an http request. As soon as I need to generate a request, I create an httplib2.Http object, so my program will often create / destroy httplib2.Http objects.

I found that my program crashed easily due to reaching the maximum number of open files. When checking / proc // fd, there were too many open fds. The problem made me look at the httplib2 source code.

Then I found that the httplib2.Http._conn_request method had this code:

else: content = "" if method == "HEAD": conn.close() else: content = response.read() response = Response(response) if method != "HEAD": content = _decompressContent(response, content) break 

This shows that the socket only closes when the http method is HEAD. Maybe httplib2 wanted to reuse the socket somehow. But the Http class does not have a close () method. This means that when I make an Http request, the socket will not close until my process completes.

Then I changed the code:

  else: content = "" if method == "HEAD": conn.close() else: content = response.read() response = Response(response) if method != "HEAD": content = _decompressContent(response, content) conn.close() # I ADD THIS CLOSE break 

After that, my program worked well.

But I'm still curious if this is really an httplib2 error, given that httplib2 is a very old and general library.

Answer 1

No, the fact that the connections are not closed is not an error; you must reuse the connection for the next request.

The httplib2 method is httplib2 , it expects the server to close the connection or a timeout for it, it will not actively close them (execept on errors). If the connection was closed, httplib2 will simply try to restore it the next time it is used.

Closing the connection (as your patch to the source does) will make this impossible, each request will have to use its own connection.

Destroying the httplib2.Http object should automatically close open connections, since it only mentions the connections dictionary, so if you do not store links to the Http objects you created, it should work. Are you sure that you have no links to them?

Alternatively, you can pass the Connnection: close header to your request:

 http.request("http://example.org", headers={'Connection': 'close'}) 

This should force the server to close the connection after each request. Or you can try to manually close all connections in the connections dict object.