Unlocking Socket Connections: Beyond the Port Limit

Motivation

I was discussing with one of my coworkers that we need multiple ingress hosts in Kubernetes because there is a limit on number of socket connections that can be opened from a single host. I thought, lets explore to expand this limit on a Linux host.

TCP Multiplexing

Before jumping into actual topic, lets first discuss some details about how TCP connections are established from client to server. We are going to keep everything in IPv4 namespace in this blog.

TCP connections are identified by 4 parameters — source_ip , source_port, dest_ip, dest_port. Lets fix dest_ip and dest_port as these are deterministic in nature based on the server application we want to reach to and based on the address resolution by DNS. For instance, imagine you need to connect to example.com (93.184.215.14) on port 80.

Since we are going to be discussing about socket connections from client, lets shift our focus on source_ip and source_port. A combination of source_ip and source_port identifies a connection uniquely on the client side.

In normal scenarios, when a TCP connection is established, operating system fills in source_ip and source_port for us. The source_ip is typically the IP address of our Network Interface which can be seen by running ifconfig command on the machine.

For instance, here it is 192.168.51.63

As for source_port, this is also chosen by the operating system from one of the dynamic ports available at the time of connection.

Generally speaking, 2¹⁶ ports are divided in following 3 categories

• Ports 0 to 1023 are well known ports or system ports and are used by system processes to provide widely used network services like HTTPs(443), HTTP(80), DNS (53) etc.
• Ports 1024 to 49151 are registered ports by IANA for specific services like Oracle Database (1521) etc. but these can be assigned if not used.
• Ports in the range 49152–65535 are dynamic or private ports that cannot be registered by IANA and can be ephemerally allocated for TCP connections.

So, pratically speaking, if we fix the network interface’s IP address, we can only create (65535 — 49152) = 16383 socket connections from a machine as only those many unique combinations of (source_ip, source_port) would exist on a machine.

For what its worth, the dynamic ports space can be larger based on system values. For instance, in my system, the port space is little different and wider: 60999–32768 = 28231

Nonetheless, the number of ports are limited and hence there is limit on number of socket connections — which we are going to cross in this blog next.

Verifying Limits

Lets verify the theory we developed earlier regarding the number of socket connections which can be opened based on the system’s limit of dynamic ports. In my case which should be close to 28231. It will be little less than that since some of the ports would be used by open applications like browser at the time of writing this blog to open TCP connections to open websites.

Before we can do so, I have observed that there is another limit on number of files which can be opened by a process on some *nix systems. Since opening a socket connection in Linux is analogous to opening a file, we need to bump this limit by running ulimit command:

 ulimit -n 1000000

Lets create 65535 sockets:

for (int i = 0; i < 65535; i++) {
  int sockfd, portno;
  /* AF_INET is for selecting IPv4 address space. */
  /* SOCK_STREAM is to open TCP socket. */

  if ((sockfd = socket(AF_INET, SOCK_STREAM, 0)) < 0) {
      perror("cannot create socket");
      return -1;
  }

  struct sockaddr_in myaddr;
  socklen_t addrlen = sizeof(myaddr);

  /* INADDR_ANY is the IP address and 0 is the socket */
  /* AF_INET is for Ipv4 */
  /* htonl converts a long integer (e.g. address) to a network representation */
  /* htons converts a short integer (e.g. port) to a network representation */

  memset((char*)&myaddr, 0, sizeof(myaddr));
  myaddr.sin_family = AF_INET;
  myaddr.sin_addr.s_addr = htonl(INADDR_ANY);
  /* Let the OS choose the port. */
  myaddr.sin_port = htons(0);

  if (bind(sockfd, (struct sockaddr*)&myaddr, sizeof(myaddr)) < 0) {
      perror("bind failed");
      j += 1;
      return -2;
  }

  /* To get assigned port bind.*/
  if (getsockname(sockfd, (struct sockaddr*)&myaddr, &addrlen) < 0) {
      perror("ERROR on getsockname");
      return -3;
  }

  portno = ntohs(myaddr.sin_port);
  printf("conn = %d | port: %d\n", i, portno);
}

I have added comments in the code to explain some of the MACROS but if anything is unclear or you want to learn more, you can go through man pages for socket , bind , getsockname system calls.

Running the above code, yields following output. I was able to open 28210 (pretty close to 28231 theoretical limit we established earlier) sockets after which the OS was not able to find free ports to bind to.

vasusharma@eagle-nest:~/tcp-multiplex$ gcc main.c -o client ; ./client
....
....

conn = 28206 | port: 57244
conn = 28207 | port: 57246
conn = 28208 | port: 57906
conn = 28209 | port: 57908
conn = 28210 | port: 57918
bind failed: Address already in use

vasusharma@eagle-nest:~/tcp-multiplex$ echo $?
254

Pushing the boundaries

We cannot increase the number of ports on the system, so lets find another way — lets change thesource_ip which is used by operating system to create sockets. This will create more unique combinations of (source_ip, source_port) and we can create more sockets !

As mentioned before, this is chosen by Operating system as the IP address assigned to NIC by network DHCP server. Lets change this trend.

As you might have guessed, OS will not allow us to do this nasty thing by freely binding to any IP address of our will. We need to bypass this check by specifying a socket option called IP_FREEBIND allowing us to do so.Please read man pages of setsockopt to know more about the system call.

So, the testing code becomes the following where we change the IP address when the bind system call fails. We begin from 192.168.51.63 which is the IP address of my network interface and thereon I increment the suffix 63 to increase my combinations of (source_ip, source_port) and create more sockets when bind fails ( refer this article for more information on adding arbitrary IP addresses to network interfaces ):

int suffix = 63, free_bound = 0, opt = 1;
char ip[16];
sprintf(ip, "192.168.51.%d",suffix);
for (int i = 0; i < 65535; i++) {
    int sockfd, portno;
    if ((sockfd = socket(AF_INET, SOCK_STREAM, 0)) < 0) {
        perror("cannot create socket");
        return -1;
    }
    if (setsockopt(sockfd, SOL_IP, IP_FREEBIND, &opt, sizeof(opt)) != 0) {
        perror("cannot set socket options");
        return -4;
    } else{
        free_bound = 1;
    }
    struct sockaddr_in myaddr;
    socklen_t addrlen = sizeof(myaddr);

    /* INADDR_ANY is the IP address and 0 is the socket */
    /* htonl converts a long integer (e.g. address) to a network representation */
    /* htons converts a short integer (e.g. port) to a network representation */

    memset((char*)&myaddr, 0, sizeof(myaddr));
    myaddr.sin_family = AF_INET;
    // myaddr.sin_addr.s_addr = htonl(INADDR_ANY);
    inet_pton(AF_INET, ip, &myaddr.sin_addr);

    myaddr.sin_port = htons(0);

    if (bind(sockfd, (struct sockaddr*)&myaddr, sizeof(myaddr)) < 0) {
        perror("bind failed");
        sprintf(ip, "192.168.51.%d",++suffix);
        if (!free_bound) return -2;
    }

    if (getsockname(sockfd, (struct sockaddr*)&myaddr, &addrlen) < 0) {
        perror("ERROR on getsockname");
        return -3;
    }

    portno = ntohs(myaddr.sin_port);
    printf("conn = %d | IP: %s | port: %d\n", i, ip, portno);
}

This yields following successful output to create 65535 connections (yayy!):

vasusharma@eagle-nest:~/tcp-multiplex$ gcc main.c -o client ; ./client
...
conn = 15255 | IP: 192.168.51.63 | port: 46647
conn = 15256 | IP: 192.168.51.63 | port: 48571
conn = 15257 | IP: 192.168.51.63 | port: 49847
conn = 15258 | IP: 192.168.51.63 | port: 52465
conn = 15259 | IP: 192.168.51.63 | port: 49671
conn = 15260 | IP: 192.168.51.63 | port: 33265

....
....

conn = 54474 | IP: 192.168.51.64 | port: 35287
conn = 54475 | IP: 192.168.51.64 | port: 48659
conn = 54476 | IP: 192.168.51.64 | port: 40613
conn = 54477 | IP: 192.168.51.64 | port: 51207

....
....

conn = 65530 | IP: 192.168.51.65 | port: 47295
conn = 65531 | IP: 192.168.51.65 | port: 60043
conn = 65532 | IP: 192.168.51.65 | port: 42887
conn = 65533 | IP: 192.168.51.65 | port: 35519
conn = 65534 | IP: 192.168.51.65 | port: 39005

vasusharma@eagle-nest:~/tcp-multiplex$ echo $?
0

Interestingly enough, while the script is running, if you try to go to a website on browser, it will fail since there are close to NO open ports to use (i.e. browser uses default IP address for connections). We are deliberately not shutting down the sockets since we don’t want the ports to be re-used when new sockets are created. But when the script terminates, all open files are closed i.e. all socket connections are closed and all bound ports are released.

Testing sockets

I wrote a sample Server in C which just listens to connection requests from clients on port 8080 and writes the bytes received on the established connection to the STDOUT .

Now that sockets are created using souce IP addresses which are not bound to any network interface and chosen by us , can they really make connections to servers ? Will the OS allow connections from such sockets ?

Turns out NO. Operating system validates that the IP addresses belong to a turned up network interace before allowing such connections. So, if you try to create a connection using the socket with IP address other than 192.168.51.63 which the IP assigned by DHCP to my WLAN network card, there will be Network is Unreachable Errors!

int connection_test(int fd) {
    struct sockaddr_in servaddr;  /* server address */
    /* fill in the server's address and data */
    memset((char*)&servaddr, 0, sizeof(servaddr));
    servaddr.sin_family = AF_INET;
    servaddr.sin_port = htons(8080);
    /* Server is running  on local address i.e. 192.168.51.63 */
    inet_pton(AF_INET, "192.168.51.63", &servaddr.sin_addr);
    /* connect to server */
    if (connect(fd, (struct sockaddr *)&servaddr, sizeof(servaddr)) < 0) {
        perror("connect failed");
        return 0;
    }
    write(fd,"Hello Server",12);
}

vasusharma@eagle-nest:~/tcp-multiplex$ gcc main.c -o client
conn = 0 | IP: 192.168.51.64 | port: 42177
connect failed: Network is unreachable

We just need to add these new IP addresses to our Network Cards assigned IP addresses and we should be good to go (Yes, it is possible). This can be simply done by using ip addr Linux system commands.

Here, the wlp1s0 is the device name of my WLAN Network Interface Card which can be obtained by running ifconfig command.

vasusharma@eagle-nest:~/tcp-multiplex$ sudo ip addr add 192.168.51.64/24 dev wlp1s0
[sudo] password for vasusharma: 
vasusharma@eagle-nest:~/tcp-multiplex$ sudo ip addr add 192.168.51.65/24 dev wlp1s0

Running ip addr command shows that the addresses 192.168.51.64/24 and 192.168.51.64/24 are added as secondary addresses for wlp1s0NIC.

Connections to Server will now be successful from 192.168.51.64 as well:

vasusharma@eagle-nest:~/tcp-multiplex$ ./server 
Server listening on port 8080
Client connected
Client sent: Hello Server

Thanks for reading along! The source code for snippets above is present at https://github.com/vasusharma7/tcp-multiplex

References

1. For more information around CIDR notations and assigning more than one IP address to a network interface, refer: https://ostechnix.com/how-to-assign-multiple-ip-addresses-to-single-network-card-in-linux/
2. Socket Programming: https://people.cs.rutgers.edu/~pxk/rutgers/notes/sockets/
3. IP man page: https://man7.org/linux/man-pages/man8/ip.8.html
4. Socket man page : https://man7.org/linux/man-pages/man2/socket.2.html