Why is Python 3 http.client much faster than python requests?

Question

I tested different Python HTTP libraries today, and I realized that the library http.clientis much faster than requests.

To test it, you can run the following two code examples.

import http.client

conn = http.client.HTTPConnection("localhost", port=8000)
for i in range(1000):
    conn.request("GET", "/")
    r1 = conn.getresponse()
    body = r1.read()
    print(r1.status)

conn.close()

and here is the code that does the same with python requests:

import requests

with requests.Session() as session:
    for i in range(1000):
        r = session.get("http://localhost:8000")
        print(r.status_code)

If I run SimpleHTTPServer:

> python -m http.server

and execute on code samples (I am using Python 3.5.2). I get the following results:

http.Client:

0.35user 0.10system 0:00.71elapsed 64%CPU 

python requests:

1.76user 0.10system 0:02.17elapsed 85%CPU 

Are my measurements and tests correct? Can you reproduce them? If so, does anyone know what is going on inside http.client, which makes it much faster? Why is there a big difference in processing time?

Answer 1

, , requests DNS , http.client .

# http.client
ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     1974    0.541    0.000    0.541    0.000 {method 'recv_into' of '_socket.socket' objects}
     1000    0.020    0.000    0.045    0.000 feedparser.py:470(_parse_headers)
    13000    0.015    0.000    0.563    0.000 {method 'readline' of '_io.BufferedReader' objects}
...

# requests
ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     1481    0.827    0.001    0.827    0.001 {method 'recv_into' of '_socket.socket' objects}
     1000    0.377    0.000    0.382    0.000 {built-in method _socket.gethostbyname}
     1000    0.123    0.000    0.123    0.000 {built-in method _scproxy._get_proxy_settings}
     1000    0.111    0.000    0.111    0.000 {built-in method _scproxy._get_proxies}
    92000    0.068    0.000    0.284    0.000 _collections_abc.py:675(__iter__)
...

http.client.HTTPConnection() , , gethostbyname . requests.Session , , , -, .

EDIT: . , -, gethostbyname .

Answer 2

@Lukasa :

, , httplib. httplib : . - , , , . . , httplib.

, cProfile : , httplib. : , .

, . , , " , httplib", : , " , ". , : .