Threading and Scaling Model for TCP Server with epoll

Question

I read the C10K document, as well as many related articles on extending a socket server. All roads indicate the following:

• Avoid the classic "thread to join" error.

• Prefer epoll over select.

• Similarly, the obsolete async io mechanism in unix can be difficult to use.

My simple TCP server just listens for client connections in the listening slot on a dedicated port. Having received a new connection, it analyzes the request and sends a response. Then gracefully closes the socket.

I think I have a good handle on how to scale this on a single thread using epoll. Only one loop that calls epoll_wait for the listening socket, as well as for existing client connections. Upon return, the code will process new creations of new client connections, as well as manage the state of existing connections, depending on which socket has just received the signal. And perhaps some logic for managing connection timeouts, graceful socket closures, and efficient allocation of resources for each connection. Seems simple enough.

But what if I want to scale it to take advantage of multiple threads and multiple processor cores? The basic idea that comes to mind is this:

One dedicated thread to listen for incoming connections in the TCP listening socket. Then a set of N threads (or a pool of threads) to handle all active concurrent client connections. Then come up with some thread-safe way in which the listener thread will “send” a new connection (socket) to one of the available worker threads. (ala IOCP on Windows). The workflow will use the epoll loop for all connections it processes to do what a single-threaded approach would do.

Am I on the right track? Or is there a standard design pattern for running a TCP server with epoll for multiple threads?

Suggestions on how the listening thread sends a new connection to the thread pool?

Answer 1

• First, note that this is C * 10K *. Do not worry if you are under 100 (using the regular system). Even then, it depends on what your sockets are doing.
• Yes, but keep in mind that epoll manipulations require system calls, and their cost may or may not be more expensive than the cost of managing multiple fd_set yourself. The same goes for poll . At low cost, it is cheaper to do user-space processing at each iteration.
• Asynchronous IO is very painful if you are not limited to a few sockets, which you can juggle as needed. Most people handle using event loops, but these fragments invert the flow of your program. It also usually requires the use of large, bulky frameworks for this purpose, since a reliable and fast event loop is not easy to obtain.

First question: do you need this? If you can easily cope with existing traffic by creating threads to handle each incoming request, continue to do so this way. The code will be easier for him, and all your libraries will play well.

As I mentioned above, juggling concurrent requests can be tricky. If you want to do this in one cycle, you will also need guarantees of processor hunger when generating your answers.

The shipping model you proposed is a typical first step solution if your answers are expensive to generate. You can either fork or use threads. The cost of branching or generating a stream should not be taken into account when choosing a merging mechanism: rather, you should use this mechanism to limit or order the load placed in the system.

Dosing sockets on multiple epoll cycles are excessive. Use several processes if it is despair. Note that it is possible to accept on a socket from multiple threads and processes.

Answer 2

I would suggest that you are on the right track. But I also think that the details depend on the specific situation (bandwidh, query patterns, processing indifidual queries, etc.). I think you should try and study carefully.