After a load test, every new mio connection is immediately suspended

I wrote a multi-threaded asynchronous HTTP server in Rust using mio. When I run a load test (using a siege), the server works fine in the first load test, but when the load test is running, all server start requests are interrupted.

Using some logging, I noticed that every new connection I receive with accept() receives a hangup event immediately. The connection to the server itself does not cause any errors or freezes.

I am running Rust 1.12.0 with mio 0.6 on OS X 10.11 El Capitan

Here is the main event loop of my server:

 pub fn run(self) { let poll = Poll::new().unwrap(); let server = TcpListener::bind(&SocketAddr::from_str(&self.host).unwrap()).unwrap(); poll.register(&server, SERVER, Ready::readable(), PollOpt::edge()).unwrap(); let mut events = Events::with_capacity(1024); let mut next_conn: usize = 1; let mut workers = Vec::new(); // Create worker threads. for _ in 0..self.num_workers { let (tx, rx) = channel(); let worker_handler = self.event_handler.duplicate(); thread::spawn(move || { Self::process_events(rx, worker_handler); }); workers.push(tx); } loop { println!("Polling..."); match poll.poll(&mut events, None) { Err(e) => panic!("Error during poll(): {}", e), Ok(_) => {} } for event in events.iter() { match event.token() { SERVER => { println!("Accepting.."); match server.accept() { Ok((stream, _)) => { println!("Registering new connection..."); match poll.register(&stream, Token(next_conn), Ready::readable(), PollOpt::edge()) { Err(e) => panic!("Error during register(): {}", e), Ok(_) => { println!("New connection on worker {} ", next_conn % self.num_workers); workers[next_conn % self.num_workers] .send(Msg::NewConn(next_conn, stream)) .unwrap(); next_conn += 1; } } } Err(e) => panic!("Error during accept() : {}", e), } } Token(id) => { println!("Sending event on conn {} to worker {}", id, id % self.num_workers); workers[id % self.num_workers] .send(Msg::ConnEvent(id, event.kind())) .unwrap(); } } } } } fn process_events(channel: Receiver<Msg>, mut event_handler: Box<EventHandler>) { loop { let msg = channel.recv().unwrap(); match msg { Msg::NewConn(id, conn) => { event_handler.new_conn(id, conn); } Msg::ConnEvent(id, event) => { event_handler.conn_event(id, event); } } } } 

The full code with the webapp example that I use is available on GitHub .

Load Testing Team:

 siege -b -c10 -d10 -t20S http://localhost:8080 
+5
source share
1 answer

I do not know why download testing applications do not document this better. A few months ago I ran into the same problem. It looks like you have reached the "Ephemeral Port Limit" . Here are some citations from an article that summarizes ideas:

Whenever a connection occurs between a client and a server, the system associates this connection with an ephemeral port โ€” the set of ports indicated at the upper end of the allowable port range.

The total number of ephemeral ports available in OS X is 16,383.

Please note that this restriction does not affect real-world requests to a live server, since each TCP connection is determined by a tuple of source IP address, source port, destination IP address and destination port - therefore, the ephemeral port limit applies to only one client / server .

In other words, this is due to the fact that you are performing a load check from localhost to localhost and terminating the ephemeral ports after about 16383 connections.

There are several things you can do to check if this is a problem:

  • Ask your load tester to report the number of connections. If it's about 16,000, then this is a likely criminal.

  • Increase the ephemeral port limit and run your load tests again. If you get more connections, then this is probably the problem. But remember, if this is a problem, it will not be a problem in the wild.

You can see your ephemeral port range using the following command:

$ sysctl net.inet.ip.portrange.first net.inet.ip.portrange.last

And you can increase it with this command:

$ sysctl -w net.inet.ip.portrange.first=32768

After running the tests, you should probably set the port range back to what it was before, since this increase is a non-standard range.

+5
source

Source: https://habr.com/ru/post/1258268/


All Articles