How to keep more than 65535 simultaneous TCP connections?

Question

Computer networks

I would like to keep about 150k simultaneous WebSocket connections with one server.

Inspired by a post about 600k concurrent connections on Amazon EC2, and a post about WebSocket load testing by Tsung.

I read that the theoretical limit is 65535 - this is from one IP to one IP, port. Here will be connected from different IP.

I'm trying to test on DigitalOcean. Raised the server, launched the simplest WS server on Swoole , installed Tsung for load testing.

I raise 3 more VPS as "workers" - they will bomb the server with connections. Several - so that from several different IPs.

But in all tests, the highest number of concurrents was only 65526 ​​- this is when all the workers were in the same DigitalOcean region. The second time I tried to run each worker in a separate region - it only got worse: about 56k. Otherwise, they fell off with an EADDRINUSE error.

Where is the bottleneck? Does the DO network somehow limit the number of connections to one virtual machine?

Answer 1

the number of connections per port is limited only by the capabilities of the system.
the limit of 65,536 is about the total number of ports per system, it has nothing to do with the number of connections .
to find out the conditions for providing you with resources, you need to read the terms of the contract or the terms of the tariff on the basis of which you are given computer power.

Answer 2

In the case of incoming connections, there is no obvious limit of 65535 connections, most likely you will run into a limit on file descriptors (sockets), for each connection you need a socket. In this case, the limits can be at the level of user limits (and it must be taken into account that for services launched through systemd, there are separate limits, and not those specified in limits.conf). Usually, by default, descriptor limits are in the region of 1024-4096, significantly lower than 65535. For a very large number of sockets, it will also be necessary to change sysctl to the maximum number of open files, and possibly rebuild the kernel to bypass the upper limits.
In the case of outgoing connections, in addition to sockets, you will run into a lack of ephemeral ports, each outgoing connection requires a separate ephemeral port. By default, they are around 16k and can be expanded to 63k through the appropriate sysctls. Above this value, you can resolve the lack of ports by adding additional IP + depending on the system, you may need to set the flags SO_REUSEADDR / SO_REUSEPORT / SO_PORT_SCALABILITY). How they work, and what combinations should be used depends not only on the system, but also on the kernel version, there is an attempt to parse it here:
https://stackoverflow.com/questions/14388706/socke...
But if outgoing traffic is through NAT (in the case of virtualization, this is almost always the case), it is necessary to solve a similar problem at the NAT level.
For a very large number of connections, you will also hit other limits - sysbufs, memory, and others.

Answer 3

I raise a few more VPSs as "workers"

A few is how much?
There is such a sysctl parameter as net.ipv4.ip_local_port_range - the range of ports for client connections (ports are assigned from this pool when a new connection is opened, in which the machine acts as a client). By default, it looks something like this: 32768 60999 (i.e. a little more than 28k available ports). Those. two workers will not make 150k connections without setting (even with the setting, because the limit of outgoing connections to one IP is 65535).
There are also several parameters that affect the processing queue for outgoing and incoming connections. You can read about them and their settings here (eng) and here (rus).
Well, in the articles to which you yourself give links, there are recommendations for setting up some parameters (for example, limiting the number of file descriptors).

Answer 4

4 ip to balance and keep yourself.

Answer 5

There are no restrictions on the number of connections. This Number of ports 1-65535 restrictions can be in the router, the provider, and in the operating system, for example, the processor and RAM of the router do not always cope with 5000 connections

Answer 6

You are talking about the 64k port limit. It manifests itself, for example, in the fact that you cannot run (on one IP) more than 64k network services on the server. (ssh listens on port 22, apache listens on 80, mysql listens on 3306) etc. Each listening service is identified by a socket ( IP + port), you have one IP, 64k ports, which means 64k listening sockets.
But for established TCP connections:
socket
An address which specifically includes a port identifier, that
is, the concatenation of an Internet Address with a TCP port.
connection
A logical communication path identified by a pair of sockets.
https://tools.ietf.org/html/rfc793
That is, the connection is identified by server IP, server port, client IP, client port.
Yes, and you yourself on any more or less active web server see many connections through lsof, and they are all established with one of your sockets (IP:80 or IP:443), but they have a different second socket. If a user, for example, downloads a file in two streams, there will be one connection: server:80 - client:4444 and another connection: server:80 - client:4445. These are different TCP connections.