Creating a huge amount of HTTP requests in Python

Question

I am trying to test a web application, part of which includes creating ~ 10K requests, taking a few <1K that return 200 OKand view their data. Webapp is faulty, and there are false positives, so every 200 OK should be checked at least three times.

Working in Python I tried to do this with threading and urllib, but on linux I get thread errors after ~ 920 threads. (My theory is this /proc/sys/kernel/threads-maxdivided by thirty, which is terribly accurate, but this contradicts the fact that each thread is registered as 30 threads with os). In any case, I am looking for a good solution to this problem. I looked at Twisted, but it looks like I will still be thread bound.

Any ideas?

Answer 1

I tested the whit apache ab TORNADO web server and could not get through more than 1000 connections per second on my dual-core Athlon @ 2Ghz. 30% of the resources took the test tool ab and stayed for the server. I am pretty convinced that most resources are spent at the OS and IP-eth level.

http://amix.dk/blog/post/19581
non-blocking servers have better performance than blocking servers because they do not generate a tread for each connection. Theoretically, they can work in one pass.

Answer 2

You can try using asynchronous HTTP requests (there is sample code at the bottom of the article).

Answer 3

FunkLoad -.

Answer 4

I have used Python bindings for libcurl ( pycurl ) in the past for this. Use the multi client function, which does this asynchronously in C. This is pretty fast.