Kernel-based data relay (Linux) between two TCP sockets

Question

I wrote a TCP relay server that acts as a peer router (excess).

The simplest case is two open sockets and a data relay between them:

clientA <---> server <---> clientB

However, the server should serve about 2000 of these AB pairs, i.e. 4000 connectors ...

Userland implements two well-known data stream relay implementations (based on socketA.recv () → socketB.send () and socketB.recv () → socketA.send ()):

• use of select / poll functions (non-blocking method)
• using threads / forks (blocking method)

I used threads, so in the worst case, the server creates 2 * 2000 threads! I had to limit the size of the stack and it works, but is this the right solution?

The core of my question is:

Is there a way to avoid actively transferring data between two sockets in userland?

There seems to be a passive path. For example, I can create a file descriptor from each socket, create two channels and use dup2 () - the same method as stdin / out redirection. Then two threads are useless for data relays and can be finished / closed. The question is, should the server ever close sockets and channels and how to know when the pipe is broken to register a fact?

I also found "pairs of sockets", but I'm not sure about this for my purpose.

What solution would you recommend to disable user feed and limit the number of threads?

Additional explanations:

• The server defined a static routing table (for example, ID_A with ID_B - conjugate identifiers). Client A connects to the server and sends ID_A. Then the server waits for client B. When A and B are paired (both sockets are open), the server starts the data relay.
• Clients are simple devices behind symmetric NAT, so N2N protocols or NAT bypass methods are too complex for them.

Thanks Gerhard Rieger I have a hint:

I am aware of two kernel methods to avoid reading / writing, recv / send in user space:

• Sendfile
• splicing

Both have restrictions regarding the type of file descriptor.

dup2 will not help to do something in the kernel, AFAIK.

Pages: splice (2) splice (2) vmsplice (2) sendfile (2) tee (2)

Related links:

• Overview of sendfile () and splice ()
• http://blog.superpat.com/2010/06/01/zero-copy-in-linux-with-sendfile-and-splice/
• http://yarchive.net/comp/linux/splice.html (Linus)
• Difference C, sendfile () and send ()
• bridge between two file descriptors
• Send and receive a file in socket programming on Linux using C / C ++ (GCC / g ++)
• http://ogris.de/howtos/splice.html

Answer 1

BSD implements SO_SPLICE:

• relayd asiabsdcon2013 slides / paper
• http://www.manualpages.de/OpenBSD/OpenBSD-5.0/man2/setsockopt.2.html
• http://metacpan.org/pod/BSD::Socket::Splice .

Does Linux support something like this, or only its own kernel module - is this a solution?

• TCPSP
• SP-MOD described here
• TCP-Splicer described here
• L4 / L7
• HAProxy

Answer 2

Even for loads like 2000 simultaneous connections, I will never use threads. They have the highest stack and switching overhead, simply because it is always more expensive that you can be interrupted anywhere than when you can only be interrupted in certain places. Just use epoll () and splicing (if your sockets are TCP, which looks like this) and everything will be fine. You can even do work with epoll in event trigger mode, where you only log once.

If you absolutely want to use threads, use one thread per processor core to distribute the load, but if you need to do this, it means that you play at a speed where the proximity, the location of RAM on each processor socket, etc. plays a significant role, which does not seem to have a place in your question. Therefore, I assume that one thread in your case is more than enough.