Manage many TCP connections efficiently and effectively

Question

I am curious if anyone can give me any ideas or suggestions on how to efficiently manage multiple TCP connections. I say about 1000 tcp connections (maybe more). The application managing all of these connections needs to periodically retrieve information from the client (the other end of the connection). For example, maybe every 30 seconds. I would use .NET 4.0 for this. Is there anything built-in to help with this or a special way to structure everything in order to manage all these connections without causing this application to clutter that it is not useful?

Answer 1

I can give you one quick look:

Don't even think about creating a thread for each connection. Instead, manage connections using the async / task built-in frameworks. I.e; let the thread pool do this.

Also keep a close eye on what your threads are doing; Make sure you can never read and write a connection stream from multiple streams at the same time. But at the same time, be careful how you use blocking mechanisms to do this, so you won’t get a process with 1000 active connections and a thread pool full of threads waiting for locks that they will never receive.

Answer 2

Working with thousands of connections is not a difficult task if you use only asynchronous APIs. The concept of “one thread per connection” is not only not scaled, but (in the general case of the protocol) is erroneous. Note that asynchronous APIs are different from using ThreadPool .

Asynchronous programming takes a little time to catch, but it is not so difficult. You will have to manually track more states than with synchronous programming.

Make sure protocol classes can handle partial accepts of any size (which requires extra state) and that you have one protocol instance for each connection.

Any socket errors should result in closing the connection and clearing the state.

And a magazine. Everything. TraceSource is your friend; Learn how to use .NET tracing, which you can enable (even during production) by editing app.config.

Answer 3

I cannot recall the built-in support for this particular scenario. But basically this is how I would do it and generally known as good practice:

1) You have a thread pool (small size - maybe around 5), responsible for the connections. This will be read from the job queue (open or close) and control the opening and closing of connections.

2) You have a thread pool (medium size) for managing messages and operations. I think a pool of 50 should be fine.

3) You will have one thread responsible for organizing the entire operation, but this should not be missed with any operation, but only with queue operations for other threads.

Answer 4

Just add the “minor” thing to the already great answers here.

An often forgotten area is the capabilities of the network cards themselves. Verify that the network adapter on your server supports TCP offloading. For that matter, pay for the good one and make sure the drivers are the latest / greatest.

The TCP offload mechanism built into the adapter works at a lower level than your code and will handle all the work of shuffling data to / from clients. This can have tremendous performance and scalability for your software.

Answer 5

Using an Asynchronous Server Socket

Answer 6

First, use async methods as others have said. 1000 connections should not be a problem with smart equipment and smart design.

I will have a collection of these connections protected by a single reading unit of several writers. I probably have a connection timer to deal with the periodic nature of the survey. I would fire the request to the set-top box when the timer fires, and then allow asynchronous reading on this connection to accumulate the response, and then process it. Ideally, I would try to avoid access to the collection of connections and store the data associated with the connection so that they can be accessed without blocking the read.

A blog on supporting lots of parallel connections is here: http://www.serverframework.com/asynchronousevents/2010/10/how-to-support-10000-or-more-concurrent-tcp-connections---part-2- --perf-tests-from-day-0.html the important thing, IMHO, is to make sure you test for this kind of download from the very beginning so you can quickly notice when you add bad design decisions that not scaled.