How do ServicePointManager.ReusePort and SO_REUSE_UNICASTPORT ease the depletion of the ephemeral port?

Question

Windows 10 and Windows Server 2016 introduced the SO_REUSE_UNICASTPORT socket SO_REUSE_UNICASTPORT . It has been available for use in .NET since version 4.6 using the static property ServicePointManager.ReusePort . I am experiencing excessive port exhaustion in my .NET application during very high loads (many outgoing requests through HttpClient at the same time), and I am considering using this option to solve this problem. I know of other ways to solve the problem (for example, editing the Windows registry to change the maximum number of ephemeral ports or reducing TIME_WAIT), but I would also like to fully use this solution.

The documentation for ServicePointManager.ReusePort very minimal:

Setting this property to true causes all outgoing TCP connections from HttpWebRequest to use the socket parameter SO_REUSE_UNICASTPORT on the socket. This leads to sharing of the main outgoing ports. This is useful for scenarios when a large number of outgoing connections occur in a short time, and the application runs the risk of terminating from ports.

Looking at the documentation for SO_REUSE_UNICASTPORT , it does not contain any additional information:

During installation, enable reuse of the ephemeral ports for Winsock API connection functions that require explicit binding, such as ConnectEx. Note that connection functions with implicit binding (for example, a connection without an explicit binding) have this option set by default. Use this parameter instead of SO_PORT_SCALABILITY on platforms where both are available.

I could not find any explanation on the Internet about how exactly this "ephemeral port reuse" is achieved, how it works at a technical level and how much it reduces the risk of ephemeral port depletion. How much can I expect improvement? Using this function, how can I calculate a new limit for my application? Are there any disadvantages to including this?

This is all shrouded in mystery, and I would love it if someone could explain this new mechanism and its consequences.

Answer 1

A TCP connection is uniquely identified (local IP address, local port, remote IP address, remote port). This means that it is quite possible to use the same pair (local IP, local port) for several sockets connecting to different remote endpoints. Suppose you want to make an http request for "site1.com" and "site2.com". You are using sockets with the following code:

 using (var socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp)) { socket.Bind(new IPEndPoint(IPAddress.Parse("some local ip"), 45455)); socket.Connect(server, port); socket.Send(someBytes); // ... } 

So, you bind the socket to a specific local endpoint with port 45455. If you now try to do this simultaneously to make requests for "site1.com" and "site2.com", you will get an exception "address already in use".

But if you add the ReuseAddress option (note that this is not a variant of your question) before binding:

 socket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, true); 

You can bind the same local (ip, port) to sockets, and you will see two ESTABLISHED connections in netstat.

All of the above should show that, in theory, nothing allows even the ephemeral port to be reused in order to make several connections to various remote endpoints. But when you bind to the ephemeral port ( 0 ), it is not yet known which remote endpoint you are going to connect to. Suppose all the ephemeral ports are in use and you bind to 0. The OS provides you with some port for reuse during the bind phase, there is one socket that uses this port, already connected to "site1.com". You are also trying to connect to "site1.com" and this fails (because all 4 values ​​that identify the tcp connection are the same for both sockets).

What SO_REUSE_UNICASTPORT does is that it delays the choice of the ephemeral port when binding to 0 until the actual connection phase (for example, Connect() ). At this stage (unlike bind), you already know the local ip, remote ip, the remote port you are going to connect to, and you need to select the ephemeral port. Suppose all ports are in use. Now, instead of picking a random port to reuse (and possibly not connecting later), you can choose a port that is connected to another remote endpoint (different from the one the current socket is trying to connect to).

You can confirm this, for example, in the MS support article :

SO_REUSE_UNICASTPORT

For the connection scenario that must be implemented, the socket parameter must be before binding the socket. This parameter tells the system to defer port allocation until the connection time, when a 4-tuple (four) to connect.

Note that SO_REUSE_UNICASTPORT only affects explicit bindings (as indicated in the question quotation, but still worth repeating). If you bind implicitly (for example, when you just Connect() without binding) - this option is already installed by default (where supported, of course).

What effect does this apply to your particular application. First of all, it should be clear that if your application submits a ton of requests to the same remote endpoint (for example, to the same HTTP server), this parameter will have no effect. However, if you make many requests to different endpoints, this should help prevent port exhaustion. Still the effect of ServicePointManager.ReusePort depends on how the HttpClient works with sockets inside, I think. If this is just Connect() without an explicit binding, this option should be enabled (by supported systems) by default, so setting ServicePointManager.ReusePort to true will not have an additional effect, otherwise it will happen. Since you do not know (and should not rely) on your internal implementation, its value allows ServicePointManager.ReusePort in your specific scenario.

You can also run tests with this option on \ off, limiting the range of ephemeral ports (with a command like netsh int ipv4 set dynamicport tcp ) to a small amount and see how this happens.