Recording a multithreaded TCP server on Linux

Question

At work, I was tasked with implementing a TCP server as part of a Modbus slave. I read a lot here and on the net as a whole (including the excellent http://beej.us/guide/bgnet/ ), but I'm struggling with a design problem. Thus, my device can only accept 2 connections, and on each connection incoming Modbus requests will be received, which I must process in my main controller loop and then respond with success or failure status. I have the following ideas on how to implement this.

• Have a listener thread that creates, binds, listens and receives connections, and then creates a new pthread to listen for the connection for incoming data and close the connection after a period of waiting for downtime. If the number of active threads is currently 2, new connections instantly close to allow only 2.

• Do not create new streams from the listener stream, use select () instead to detect incoming connection requests, as well as incoming Modbus connections in active connections (similar to the approach in the Beejs manual).

• Create two listener threads, each of which creates a socket (the same IP number and port number) that can block accept () calls, then close the fd socket and handle the connection. Here I am (possibly naive), suggesting that this will only allow max 2 connections, which I can handle using blocking reads.

I have been using C ++ for a long time, but I'm pretty new to Linux development. I would really welcome any suggestions as to which of the above approaches is the best (if any), and if my inexperience in Linux means that any of these are really very bad ideas. I try to avoid fork () and stick to pthreads, as incoming Modbus requests will be queued and periodically read from the main controller loop. Thanks in advance for any advice.

Answer 1

The third option will not work, you can only bind to a local address.

I would probably use the second alternative if you don't need to process a lot, in which case a combination of the first with alternatives may be useful.

The combination of the first two alternatives that I think of is to have a main thread (the one you always have when starting the program) creates two worker threads and then blocks the accept call to wait for a new connection. When a new connection arrives, tell one of the threads to start working on a new connection and return to the block on accept . When the second connection is accepted, you will tell the other thread to work on this connection. If both connections are already open, either do not accept until one connection is closed or wait for new connections, but close them immediately.

Answer 2

Since you are dealing with only two connections, the thread for each connection is ideal for this type of application. Object-oriented approaches using non-blocking or asynchronous I / O would be better if you had to scale to thousands of connections. 2 listener threads makes sense, you don't need to close accept fd. Just come back to accept it when the connection is complete. In fact, the option is to have three threads block the execution of the adoption. If two threads are actively processing connections, the third discards the newly created connection (or returns a busy response, which is suitable for your device).

In order for all three threads to block at reception, you need the main thread to create and bind your socket before starting three threads to process their acceptance / processing.

The page for pthreads on Linux indicates that accept is thread safe. (The section in thread safe functions lists functions that are not thread safe, skip to the picture.)

Answer 3

All the proposed design options are not very object oriented, and they are all oriented more to C than to C ++. If your work allows you to use boost, then the Boost.Asio library is fantastic for creating simple (and complex) socket servers. You can take almost any of your examples and trivially expand it to allow only 2 active connections, closing all others as soon as they are open.

At the top of my head, their simple HTTP server can be changed to do this by storing the static counter in the connection class (including the constructor, dec in the destructor), and when a new one is created, check to count and decide whether to close the connection. The connection class can also get boost :: asio :: deadline_timer to track timeouts.

This is most similar to your first design choice, boost can do it in 1 thread, and in the background something like select() (usually epoll() ). But this is the “C ++ path”, and in my opinion, using select() and raw pthread is way C.