Register all requests from python-requests module

Question

I am using python Requests . I need to debug some OAuth actions, and for this I would like it to log all requests being executed. I could get this information using ngrep , but unfortunately it is not possible to grep https connections (which are necessary for OAuth )

How do I enable logging of all URLs (+ parameters) that Requests accesses?

Answer 1

The urllib3 core library registers all new connections and URLs using the logging module , but not the POST body. For GET requests, this should be enough:

 import logging logging.basicConfig(level=logging.DEBUG) 

which gives you the most detailed logging option; see the logging HOWTO for more details on how to configure logging levels and destinations.

Short demo:

 >>> import requests >>> import logging >>> logging.basicConfig(level=logging.DEBUG) >>> r = requests.get('http://httpbin.org/get?foo=bar&baz=python') INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): httpbin.org DEBUG:requests.packages.urllib3.connectionpool:"GET /get?foo=bar&baz=python HTTP/1.1" 200 353 

The following messages are recorded:

• INFO : New Connections (HTTP or HTTPS)
• INFO : remote connections
• INFO : Redirects
• WARN : Connection pool is empty (if this often increases the size of the connection pool)
• WARN : Retrying Connection
• DEBUG : Connection information: method, path, HTTP version, status code and response length

Answer 2

You need to enable debugging at httplib level ( requests urllib3 httplib ).

Here are some functions for switching ( ..._on() and ..._off() ) or temporarily turning on:

 import logging import contextlib try: from http.client import HTTPConnection # py3 except ImportError: from httplib import HTTPConnection # py2 def debug_requests_on(): '''Switches on logging of the requests module.''' HTTPConnection.debuglevel = 1 logging.basicConfig() logging.getLogger().setLevel(logging.DEBUG) requests_log = logging.getLogger("requests.packages.urllib3") requests_log.setLevel(logging.DEBUG) requests_log.propagate = True def debug_requests_off(): '''Switches off logging of the requests module, might be some side-effects''' HTTPConnection.debuglevel = 0 root_logger = logging.getLogger() root_logger.setLevel(logging.WARNING) root_logger.handlers = [] requests_log = logging.getLogger("requests.packages.urllib3") requests_log.setLevel(logging.WARNING) requests_log.propagate = False @contextlib.contextmanager def debug_requests(): '''Use with 'with'!''' debug_requests_on() yield debug_requests_off() 

Using demo:

 >>> requests.get('http://httpbin.org/') <Response [200]> >>> debug_requests_on() >>> requests.get('http://httpbin.org/') INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): httpbin.org DEBUG:requests.packages.urllib3.connectionpool:"GET / HTTP/1.1" 200 12150 send: 'GET / HTTP/1.1\r\nHost: httpbin.org\r\nConnection: keep-alive\r\nAccept- Encoding: gzip, deflate\r\nAccept: */*\r\nUser-Agent: python-requests/2.11.1\r\n\r\n' reply: 'HTTP/1.1 200 OK\r\n' header: Server: nginx ... <Response [200]> >>> debug_requests_off() >>> requests.get('http://httpbin.org/') <Response [200]> >>> with debug_requests(): ... requests.get('http://httpbin.org/') INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): httpbin.org ... <Response [200]> 

You will see the REQUEST, including HEADERS and DATA, and RESPONSE with HEADERS, but without DATA. The only thing missing is response.body, which is not registered.

Source

Answer 3

For those using python 3+

 import requests import logging import http.client http.client.HTTPConnection.debuglevel = 1 logging.basicConfig() logging.getLogger().setLevel(logging.DEBUG) requests_log = logging.getLogger("requests.packages.urllib3") requests_log.setLevel(logging.DEBUG) requests_log.propagate = True