Gevent urllib slow

Question

I created a set of TCP server demos, however my gentent examples are noticeably slower. I really do not test the performance, but at the moment something makes the gevent version 5 times slower.

I'm sure it should be how I compiled gevent, but can't solve the problem. I am using OSX leopard using fink compiled python 2.6 and 2.7. I tried both stable gevent and gevent 1.0b1, and it acts the same. The echo takes 5 seconds to answer where other examples take <1s. If I remove the urllib call, the problem goes away.

I put all the code in https://github.com/djay/geventechodemo

To run the examples, I use zc.buildout to build

$ python2.7 bootstrap.py $ bin/buildout 

To run the gevent example:

 $ bin/py geventecho3.py & [1] 80790 waiting for connection... $ telnet localhost 8080 Trying 127.0.0.1... ...connected from: ('127.0.0.1', 56588) Connected to localhost. Escape character is '^]'. hello echo: avast 

It will take 3-4 seconds to answer my system.

However threaded example

 $ bin/py threadecho2.py 

or twisted example

 $ bin/py twistedecho2.py 

Less than 1 s Any idea what I'm doing wrong?

Answer 1

The answer was to change the default DNS resolver described in this conversation.

https://groups.google.com/forum/#!topic/gevent/5uNfkdgzWVc

setting GEVENT_RESOLVER = thread started operation as expected speed

Answer 2

Just tried this on Windows XP. Does not respond immediately, but much faster than 3 seconds. Will mock up the client to measure the exact time.

PS Creating libevent on Windows is not perfect! I had to play with the included ones and fix one error in the actual code. I will stick with Linux for libevent / Gevent in the future;)

Answer 3

urllib does not support reusing the http 1.1 connection. Each time you get a page, it creates a new TCP connection and a new TCP handshake appears. urllib is and will always be slow with or without gentent.